Tag: infrastructure

Is Your Storage Too Slow for DevOps?
To have a successful DevOps transformation, teams on both the development and the operations side need fast storage infrastructure and high-performing tools to maximize its performance and the people that work with it.

Over the last decade, a major operational problem in on-premises enterprise data centers have emerged, as the fundamental mismatch between the infrastructure and the needs of increasingly virtualized applications has become glaringly apparent. There is a major contrast between what organizations want their staff to focus on—strategic projects, new application deployment, new customer acquisition and releasing new products—and the things they end up spending too much time on. This includes infrastructure “plumbing” and reconfiguring, re-architecting or redeploying across all stacks.

The widespread adoption of virtualization has introduced increasing complexities to IT infrastructure. It is also spurring businesses to accelerate development efforts and deliver applications and services through a DevOps model. The arrival of cloud infrastructure and cloud-native workloads has made addressing the mismatch an even more pressing issue. As they seek to overcome the shortcomings of outdated traditional infrastructures, organizations are adopting a cloud strategy that includes the public cloud and, in many cases, their own private cloud built from all-flash storage and intelligent software.

This move to a hybrid cloud platform, a mix of public and private cloud seeks to blend the benefits of public cloud for workloads that need agility and scale, while retaining control of workloads and data that would be better served on-premises because they are too valuable to entrust to other providers.

An effective storage platform should combine the performance, control and management of an internal data center with the agility and scale of public cloud. This provides organizations with the ability to build and run agile environments for cloud-native and mission-critical applications in their own data centers. It also helps to solve the fundamental mismatch between infrastructure and virtual applications and helps prepare organizations to adopt DevOps practices, which cannot be fully supported by traditional infrastructure.

Choosing the Right Storage Platform for DevOps

The growing emphasis on DevOps is placing an extra burden on infrastructure teams, so choosing the right platform to underpin the IT infrastructure is highly significant. This will ensure the DevOps model works for the organization and makes life easier for the IT infrastructure team.

Traditional storage systems often are unable to match the requirements for a simple, flexible and automated enterprise infrastructure. A number of issues must be taken into consideration for the enterprise cloud infrastructure to work with the DevOps model, including:
- Copy data management
- Data protection
- Disaster recovery (DR)
- Quality of service (QoS) and performance guarantees
- Monitoring and troubleshooting
- Automation
The problem for copy data management in DevOps is that refreshing and updating testing/development environments is time-consuming, and rapid test cycles require data synchronization with potentially hundreds of servers. A system is required that provides up-to-date virtual copies to the DevOps team, eradicates the need for physical data duplication, lifts the load on production storage and protects the performance of applications. Modern storage systems in conjunction with next-generation software can create data copies using snapshots, cloning and replication—and they can utilize the same technologies to protect and manage the development side of the environment in the event of failure or data corruption.

When seeking to adopt DevOps, quality of service (QoS) and guaranteed performance is a major factor. QoS is vital to control the performance allocation of a storage system to different workloads. It gives organizations the capability to set a maximum and minimum on the number of input/output operations per second (IOPS) or bandwidth consumed. Using the higher performance of all-flash and QoS, DevOps can be consolidated on the same platform as production, making it easier to access production data sets while reducing the storage footprint.

Organizations that want to provide data at a speed for DevOps need the ability to monitor the infrastructure and correct problems and misconfigurations rapidly. Continuous monitoring and end-to-end visibility across the IT stack, along with predictive analytics, integration with other monitoring tools and VM-level monitoring, are particularly valuable attributes when seeking to provide the basis for data-driven decisions in a DevOps environment.

It is also critical that the storage platform provides the capability to provision and manage IT infrastructure programmatically. With the increased use of automation, organizations can create and break down environments as necessary, incorporate snapshots and cloning as part of daily workflows and eliminate the potential for error from manual or interactive configuration.

The Case for Fast, Virtualization-Aware Storage

Implementing a storage system that is able to match the requirements of a simple, flexible and automated enterprise infrastructure delivers many benefits to the organization and to the DevOps environment. Virtualization-aware storage eradicates the need for logical unit numbers (LUNs) and volumes and enables enterprises to work at the VM and container level. With clean REST APIs, they can connect all-flash to compute, network and other elements of the cloud and see and share across the entire infrastructure.

Using virtualization-aware storage, application test and development can be accelerated from a matter of days or weeks to minutes. By retaining custom settings and automating with APIs, rather than having to rebuild the entire environment after every data refresh, enterprises can radically speed up the release cycle and integration testing.

For example, using this approach, one financial services company was able to reduce development update time from five hours to five minutes and slash latency by 82 percent. With the visibility into files, VMs and vDisks provided by virtualization-aware storage, it is possible to recover these in fewer than five clicks and response times can be reduced dramatically.

Data at DevOps Speed

As enterprises move inexorably toward a mixture of virtual and physical, on-premises and cloud-based IT, it is imperative that their storage platform can seamlessly work across physical and virtual environments. This will facilitate DevOps environments to meet its goals while eliminating concerns around performance, scalability, manageability, resilience and flexibility.

The storage system underpinning their IT infrastructure platform needs to support businesses in their desire to replicate the agility of public cloud within the data center while making it easier to manage enterprise and cloud-native applications. Adopting virtualization-aware storage can ensure their data is ready for the DevOps model. Enterprises wanting to deliver data at DevOps speed can’t afford to keep their storage in the slow lane.

— Chris Colotti
February 13, 2018
HashiCorp Adds Terraform Workflow Tools for IT Ops

HashiCorp this week moved to templatize automating IT operations by making it easier to share Terraform configuration files which make it simpler to programmatically invoke infrastructure that exposes application programming interfaces (APIs) without knowing how to code.

Available in an on-premises edition or as a software-as-a-service (SaaS) application, HashiCorp Terraform Enterprise is intended for use by IT operations teams. A HashiCorp Terraform Enterprise Module Registry function has been added to enable IT operations teams to implement a publish-and-subscribe mechanism for Terraform configuration files.

Company CTO Armon Dadgar said HashiCorp Terraform Enterprise is designed to enable IT operations teams to respond faster to requests from developers for infrastructure resources without requiring them to develop programming skills. It’s unlikely developers working for traditional enterprise IT organizations will be taking over control of IT infrastructure anytime soon, he said, and Hashicorp Terraform Enterprise allows IT organizations to inject a level of agility into their workflow processes without requiring IT operations teams to completely re-engineer existing processes.

The HashiCorp Terraform Enterprise Module Registry extends that capability further by making it easier for IT operations professionals who have developed expertise employing declarative Terraform configuration files to share those files with other members of the team, Dadgar said.

Related Content:

HashiCorp Terraform Enterprise Empowers IT Administrators

Operations Management Puts the ‘Ops’ in DevOps

At the same time, IT operations teams that want to build workflow applications on HashiCorp Terraform Enterprise can use a set of application programming interfaces (APIs) that HashiCorp has exposed. In addition, the company has revamped the user interface for Terraform—workspaces now can be created by combining Terraform files into modular components that can be assigned to specific IT teams. Permissions to access are provided via support for the secure access markup language (SAML) as well as the Sentinel policy management software developed by HashiCorp.

The HashiCorp approach to automating IT operations strikes at the core of the DevOps debate in the enterprise. While web-scale companies typically can afford to hire IT professionals with programming skills to manage IT infrastructure, the average enterprise IT organization still relies on administrators to manage servers, storage and networking. Not only are IT administrators in short supply, most of them would not be in IT operations if they knew how to program—they’d be writing and testing applications instead.

Rival approaches to automating the management of IT infrastructure, despite being around for years, have yet to gain mainstream acceptance, mainly because they require programming skills. Declarative approaches allow IT infrastructure to be managed as code still, with less resistance from IT operations teams that tend to be more comfortable employing declarative tools. In fact, there’s long been a chicken-and-egg debate over the degree to which the transition to modern DevOps practices requires organizations to acquire new tools first or vice versa. The HashiCorp approach strikes a middle ground by providing tools that enable IT operations teams to first become more efficient, which ultimately should enable them to implement new processes that often is much less disruptive to the existing IT culture.

— Mike Vizard

February 9, 2018
Cuba Embraces DevOps, Despite Infrastructure Challenges

A few weeks ago I had the pleasure of speaking at the second DevOpsDays event in Cuba alongside some great technologists from across the United States, Europe and Latin America. The conference was held in a beautiful hotel in Varadero overlooking a marina right by the beach. Varadero is approximately two hours by car from Havana.

The first thing I noticed is that Cuba is full of anachronisms. This became evident as soon as I walked out of Havana Airport and realized that the whole “they have some old cars” thing was real. It was fascinating to see that every single car was pre-1950s, some perfectly maintained with great paint jobs and others in worse condition yet still capable of doing their core job as a mode of transport.

Talking technology in Cuba was eye-opening, to say the least. Do not expect to have 3G or 4G data on your cellphone. Do not expect to have WiFi in most hotels (you can purchase prepaid internet credit, but the speed will be slower than what you are used to) and definitely do not expect to have WiFi in a private home. These are luxuries.

But here’s the surprising part: There are companies in Cuba with fully automated CI/CD pipelines achieving 10 to 12 deployments per day and solving real problems, such as the team working under Anesto del Toro at Datys. Cuba even has its very own Docker Captain, who gave a superb demonstration on the value of Docker in front of the DevOpsDays audience.

Now let me remind you, these guys are not working with stable internet connections and they manage to get the job done on a daily basis.

Why am I Rambling on About Cuba?

Engineering is about solving problems. No matter how big or small, it’s about looking at a problem and finding the solution. Now, this is part of Cuban life. If something is broken, they don’t throw it in the trash and buy a new one; they fix it. Their 80-year-old car breaks down two hours outside of town? They get out of the car to fix it and—take my word for this—their friends, family and neighbors will come outside to fix the problem, too. This creates a mindset of solving the problem at hand without fail. This is easily demonstrated when you talk to Cubans about software development: They won’t say they are a Python engineer, a Java engineer or any other choice of language—they are software engineers who understand how development works as a whole. They will use the language required to get the job done to the highest standard.

Like everywhere else in the world, there were huge differences in levels of DevOps adoption between the various organizations in the country. In contrast to the companies mentioned earlier, there also are organizations that have only just heard of these methodologies or are now considering setting up their first CI pipelines. Like many of us, they are still figuring out how to solve their technical debt and modernize their IT Infrastructure, but the key point is that the stakeholders, C-level executives, junior engineers and everyone else in between want to learn. They are open-minded to the principles of DevOps. Every person in the room at DevOpsDays Cuba was focused and listening to what we had to say. This excited me as a speaker and everyone else in attendance. We could see people who really did want to change and transform the cultures within their technology teams and overall businesses to start adopting these modern and proven practices.

The journey ahead is a long one. The infrastructure challenges remain in place and there is much to be learned, but it was great to see such positive energy and eagerness to adapt. I want to thank everyone who attended DevOpsDays Cuba for making it such an amazing experience. An even bigger thanks to the organizers Enrique Carbonell, Rudy Gevaert, Yadier Perdemo, Manuel Oliver, Mike Rosado, Manuel Morejon and anyone I’ve missed. The event couldn’t have been better and it was a genuine pleasure to visit Cuba, I hope to return soon and reconnect with as many of you as possible!

About the Author / Conor Delanbanque

Conor Delanbanque has been recruiting and leading recruitment teams in the DevOps space for the past few years. As well as being exclusively responsible for growing some of the most innovative DevOps & SRE teams in the US and Europe as an external consultant and recruiting partner, Conor also founded the Future of DevOps Thought Leaders Debate, sponsors DockerNYC Meetup on a regular basis and co-hosts ServerlessNYC. You can tweet him at @ConorDevOps, find him on LinkedIn or at events, expos and writing articles on devopsy.kinsta.cloud.

November 21, 2017

Scaling DevOps at Pearson

Pearson might be one of the more influential companies you have never heard of. Its footprint in the educational publishing marketplace is expansive, and its scope was just one challenge when it made the move to DevOps.

Sean D. Mack (@SeanDMackNYC) is formerly the VP of Operations and Application for Pearson. To give you an idea of the scale of the company’s operations when it began implementing DevOps, Sean oversaw 110 development teams and 3,000 different products across the globe. To further complicate things, Pearson is a mature company with both legacy and new products. This is a case study of how the company approached implementing DevOps with the goal of helping you learn how to do it. Sean presented it at the 2016 All Day DevOps conference.

As they say, you often learn your biggest lessons with your failures. Sean noted that, ultimately, DevOps was a failure at Pearson due to organizational change. But, he presented the case study with the hope you will learn from what Pearson did and take it forward. He continues to be a passionate evangelist for DevOps.

Pearson had its challenges. It was a siloed organization operating at a large scale, and had a toss-it-over-the-wall mentality in dealing with problems. Out of sight, out of mind.

In addition, all of Pearson’s teams were at different levels of maturity, from legacy teams running .Net on Windows 2000 servers while others doing CI/CD to AWS. Obviously, this creates challenges when implementing DevOps, but we will see later how the company tried to tackle that, specifically.

At the high level, the solution involved several aspects:

A DevOps practice area
A DevOps steering committee
Measuring progress based on global KPIs focused on measuring business impact
Measuring teams based on a DevOps scorecard
Push from the bottom – plenty of developers and operations who embraced and were eager for the change
Push from the top – support from key executives

Pearson’s DevOps Steering Committee set up several workstreams:

Tracking tools
Metrics
Deployments automation and monitoring practices
Test automation
App engineering resource contention
Operationalization
Communications

While these are specific to Pearson, many may be applicable to your organization. In any case, it is important to step back, look at your organization, and decide what your organization needs.

Pearson also looked at four core competencies to track and measure: automation; independence (how independent is the feature); operationalization (how easy it is to maintain); and continuous improvement. The below chart outlines each area in more detail.

Automation

Continuous Improvement

Independence

Operationalization

Unit tests
Critical Scenarios
Dependent Systems
Continuous Build
Continuous Deployment
Test Automation

KPI
SLA
Metric Tracking
Metric Reporting
Remediation

Isolated Configuration
Code Dependencies
Isolated Infrastructure

Monitoring and Alerting
Tier 1 Service Desk System Knowledge
Tier 2 Service Desk System Knowledge
Documentation

To help each team measure their competencies and improvement, Pearson built a self-reporting tool to empower teams but encourage visibility, and developed business metrics and then a competency model to measure teams against to see how they were progressing. As Sean noted, “We were not trying to get everyone to a 5. Some teams, yes, but we also had legacy products that were running for years, successfully, with little or no maintenance. The investment to bring the teams up wasn’t worth the effort. There had to be a business reason.“

In the end, Pearson raised DevOps maturity for all of the teams. As noted earlier, though, this was eventually phased out, but Sean presented this case study with the hope you found something to drive DevOps in your company.

In the end, if you have just a couple takeaways, they are this:

Begin by determining what your desirable outcomes are.
Don’t just do DevOps to be cool. Do it to drive business results.

You can watch Sean’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online on Oct. 24.

— Derek E. Weeks

October 6, 2017

Speaking for the Dead: Are ‘Agile’ and ‘Monolithic’ Actually Good?
What if Agile is the product of getting bored at work, microservices are the result of us needing a challenge and scrums are the result of an obsessive project manager? Do we all think waterfall development is a dinosaur whose extinction is deserved because of what we have been told and not because of reality? Do we think Agile is better because it simply sounds desirable?

Michael DeHann (@laserllama), creator of Ansible, spoke at the 2016 All Day DevOps conference, which espouses the benefits of DevOps, and, by association, Agile and microservices. It essentially says, “Let’s step back and evaluate the past. Maybe waterfall development and monolithic architectures are actually a good thing?”

Michael has been a software engineer and created a key tool in the IT automation toolbox: Ansible. His arguments deserve a look, because re-evaluation can keep you from making mistakes and improve processes going forward.

Michael started his talk by making the point that in this day and age, it is waterfall versus Agile and monolithic architecture versus microservices. The assumption is that if you are still developing software with the waterfall method and/or developing monolithic applications, you might as well be riding a horse and buggy to work and dialing up coworkers on your rotary phone.

Throughout his talk, he asked questions to make his points, beginning with, “Is big design up front bad?” with the conclusion, “I don’t think so.” His argument is that design and requirements at the beginning of the process enable a better production. Marketing and engineers hammered out requirements based on needs, abilities and budgets. There was plenty of time to research and design solutions. Both sides worked together, raising questions and comments and documenting mutual agreement. This presented an accurate view of the strengths of various team members and dependencies between tasks and task orders. And, perhaps most important, engineering was engineering—the engineers had greater responsibility and less hand-holding.

Contrasting that with his view of Agile, big design up front is labeled as bad or counterproductive, engineers are viewed as interchangeable and scope is more unpredictable. Estimates are encouraged to not be time-based, which often results in a constant crunch time. Finally, meetings can consume a significant portion of team members’ time and often include “games reminiscent of Kindergarten, such as Fist of Five or Planning Poker.” Even in lightweight Agile systems, testing can suffer, requirements are more vague and there is no mutual sign-off.

Michael argues that there can, in fact, be a middle ground where teams:
- Do share design
- Have an accessible, effective product owner
- Know hard requirements versus soft requirements
- See test-driven development and test automation as super important
- Understand that Agile and iterative development can be closer than we think
Agile and DevOps are also often associated with microservices versus monolithic applications, but according to Michael, “What’s so bad about a monolith?”

Sample Monolithic Diagram

Sample Microservices Diagram

An example microservices shop has 150 services, many thousands of Jenkins builds per day, eight Amazon regions including dev, staging and production and thousands of EC2 instances. This can result in Jenkins configuration and security enforcement drifts, suboptimal code sharing and DevOps writing lots of custom code to manage it.

Michael contrasts that with monoliths, which have:
- Increased ability to develop on one laptop
- Interfaces that benefit from static typing versus design-by-contract
- Very small numbers of different server/image types in production
- Local function calls that are much faster than RPC
- Few instances sitting around with 5 percent utilization
- No need for “containers” to get around above utilization problem
- Greatly simplified debugging
- Easy code-sharing
- Very simple version upgrades
Rather than argue that all applications should be monolithic, he makes the point that many of the concerns that resulted in microservices could be solved with improved communications. He contends that microservices are a tech solution to a people problem.

Michael concludes that:
- Labels cloud true meanings,
- Monolithic architectures are more efficient,
- Agile can be less efficient,
- Microservices attempt to solve people problems but introduce new technical and people problems, and
- We should try and separate the labels from the true meaning of things.
Most importantly, though, is the fact that there is something to learn from everything. So, even if you don’t agree with Michael’s conclusion, you should listen to his talk online here. You might learn an unexpected lesson.

If you missed any of the other 30-minute long presentations from 2016 All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
September 13, 2017
Liberty Mutual: Taking an Organic Approach to DevOps
Some people are directors, managers, and administrators. Others are disrupters.

Eddie Webb (@edwardawebb) is an IT disrupter for Software Development Platforms at Liberty Mutual and was a presenter at the 2016 All Day DevOps conference. His talk, “Organically DevOps: Building Quality and Security into the Software Supply Chain at Liberty Mutual,” looked at Liberty Mutual’s transformation to continuous integration, continuous delivery and DevOps. For a large, heavily regulated industry, this task can be not only daunting, but also viewed by many as impossible.

Eddie’s talk looks at the origins of DevOps at Liberty Mutual, the challenges of centralizing DevOps, and how to make it easy for developers and operations to do the right thing. He noted that DevOps is not Liberty Mutual’s goal; continuous delivery is the goal and DevOps is the enabler.

It all started at Liberty Mutual in 2004 with a vision: “Black-box reusable services that benefit the business.” The company also had a concept that “developers own the quality of the code; we have no QA team.” Employees were empowered with this “concrete vision but malleable strategy,” as Eddie said. The company gave them time and space to explore solutions, because ownership increases adoption.

The team started with fixing what frustrated them, including CI/CD pipelines, artifact and code repositories, elastic cloud-native runtime and Agile planning and delivery.

But what about centralizing DevOps?

The reality is that tools can be centralized, but culture cannot.

Eddie laid out an example of a large organization’s IT structure, saying that this is how things work at Liberty Mutual. Eddie’s team is the Central IT, and their customers are the Market IT team, not Liberty Mutual’s end customers. As Eddie pointed out, the more layers you create, the more friction that enters the system. Often, organizations try to reduce the friction through microfixes, but Eddie’s team asked how to change the culture to reduce the friction.

This required distributing innovation and centralizing the tools that work. As an example, if you build a manufacturing plant, you wouldn’t build your own power plant. If you are in IT, why build your own cloud?

This is the Netflix way. At Netflix, teams are independent, but responsible. Engineers can choose the best tools for the job. No one single team is responsible for innovation. However, the Engineering Tools team helps direct other teams’ experimentation toward new products without getting in the way of innovation.

Eddie quoted the book, “Drive,” when looking at how to motivate innovation in complex work environments. The author says if you want to incentivize or motivate cognitive tasks, you need three things: autonomy, mastery and purpose.

When you centralize non-value added platforms, it enables teams to:
- Master their specific domains of business;
- See clear purpose in everything they build; and
- Maintain autonomy of choice of frameworks, patterns, and approach.
When you decentralize, it raises the question: What about security? Developers are humans, and humans inherently try to avoid tasks they see as unnecessary or overly burdensome, and this becomes easier when the compliance person isn’t there (think of how we handle speed limits). Traditional security measures erode profit and market share, and, from a stat Eddie quoted from Gartner, 77 percent of IT security professionals agree that security slows down IT.

Whether you are centralized or decentralized, security (and other compliance-oriented tasks) can increase adoption rates by making it easy to do the right thing. How do you do this? Eddie proposes governing with tools (think of speed governors in fleet cars).

You also need to have immutable environments, because, “We can’t trust humans to manipulate production.” So, again, you have to make the path you want them to take, and make this the easiest path. As Eddie notes, “You can’t put too much friction into the pipeline, because developers are good at bypassing friction.” You have to make your CI/CD pipelines the easiest way to get ideas to production by. You also have to abstract complexity and automate policies. You have to have an “opinionated platform, and observe your customer so you can address their pain.”

Eddie concluded with the following final points:
- Don’t mandate DevOps. Give employees the chance to master their discipline with examples to set and follow.
- Favor deep end-to-end accomplishments over broad but incremental steps forward. Focus on taking the right teams far before encouraging broad adoption.
- Centralize the platforms and tools that your teams shouldn’t be thinking about. Provide foundational services/commodities and let teams stay on purpose.
- Incorporate contributions from everyone; don’t stifle autonomy. Stay open to new ways of working.
- Challenge security policies, but respect intentions. Find new ways to enforce concerns without abandoning precaution.
You can listen to Eddie’s entire talk online here. You might learn an unexpected lesson.

If you missed any of the other 30-minute long presentations from 2016 All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 100 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
September 12, 2017
Securing Immutable Servers in a Serverless World
Snowflakes are beautiful, unique creations. But, let’s keep them in nature. They don’t belong in our server infrastructure. Snowflake servers, where every configuration is just a little different, can introduce unnecessary security vulnerabilities and complications. While common in IT infrastructure, in the DevOps realm they are gradually becoming ancient history.

At All Day DevOps 2016, Erlend Oftedal (@webtonull), with Blank and head of the OWASP Norway chapter, discussed the benefits of immutable infrastructure practices within serverless architectures. Erlend walked us through the positive effects on security as well as offered insight on potential problems.

Erlend outlined the progress from physical servers to immutable servers, discussing the benefits of each step and the concerns that lead to the next phase, ending with immutable servers—that is, they can’t be changed. Once set up, they are set. Automated processes coupled with immutability principles can eliminate snowflake servers.

Immutability isn’t a new concept. Some classic examples include live Linux CDs, internet cafes, schools, libraries and hotel business centers. To set up immutable servers, Erlend outlines these steps:
1. Find a suitable base image
2. Build a specific image for the application
  1. Dependencies
  2. Application
  3. Secrets
3. Deploy image to the server(s)
4. Repeat steps 2-3 for new versions
Of course, this immediately raises questions:
- What about data? He notes that because data inherently changes, data has to be stored externally.
- How do you deploy changes? Well, you don’t change the server. You take one out and put a new one in. What if that one fails? Roll back to the previous one.
- How do you manage secrets? You can build secrets in, leverage cloud key management, or use third-party services.
- How do we apply security patches? If truly immutable, you deploy a new server with the patch. However, you can have a semi-immutable server that allows for automatically installing patches but still doesn’t allow for logins or manual changes.
- Speaking of security, what are the security advantages? For starters, if an attacker is in your server, it will be thrown out when you change servers, although it may come back if the vulnerability isn’t patched. It also allows for very specialized images by removing all unnecessary packages. With these specialized images, you can more easily audit and monitor the system for unexpected file changes, logins and connections.
Erlend discussed containers because they are a step closer to a serverless world. Containers isolate processes and all run on the same OS. Of course, you can make them immutable, but regardless of whether they are, there are some key security best practices you’ll want to consider:
- Don’t run as root inside container
- Implement user namespaces so the root account within the container is separated from the root account outside the container
- Utilize the least privileges necessary for capabilities and resources
- Keep them up-to-date
- Automate scanning using tools such as Sonatype Nexus, Twistlock, Clair, or Aqua
- Separate untrusted and trusted containers
Erlend also dove into serverless architecture. While there is debate over what exactly qualifies as serverless, there are five principles that Erlend highlighted:
- Use a compute service to execute code on demand
- Write single-purpose, stateless functions
- Design push-based, event-driven pipelines
- Create thicker, more powerful front ends
- Embrace third-party services
Migrating to immutable, serverless infrastructure is a topic to be discussed well beyond this blog post and even Erlend’s session. Whatever you do in the future, though, take his parting words to heart: “The cloud can solve many of your problems, but in the end you cannot transfer your responsibility for security.”

Erlend’s session was packed with information. If this piqued your interest, you should watch his complete session from All Day DevOps 2016 (just 30 minutes). The other 56 presentations from the conference are also available online, free-of-charge here.

Also, this year’s All Day DevOps is Oct. 24. It’s online and free, so be sure to check out www.alldaydevopsy.kinsta.cloud to learn more and register.

— Derek E. Weeks
August 28, 2017
All Day DevOps: Taking Lambda to the Max
Lambda is the 11th letter of the Greek alphabet, and it is also the name of the Amazon Web Services (AWS) service that lets you run code without actually configuring a server. Amazon chose the name Lambda because—well, okay, I have no idea. Comment here if you know.

Thankfully, you don’t need to know that to understand how to utilize Lambda. Matt Williams, DevOps Evangelist at Datadog (@technovangelist), outlined how to set up AWS Lambda at the 2016 All Day DevOps conference in his talk titled, “Hold the Infrastructure: Getting the Most Out of Lambda.”

First, to understand what Lambda is, we must look at the evolution of servers.

We started with physical boxes—we had to buy and maintain these boxes, and no matter how much we used, we had to pay for all of it. We then evolved to multiple virtual machines on a physical server so that we could maximize the resources of physical boxes. Now, AWS and similar cloud services are common. We create a virtual machine that we need, and the service provider maintains the physical servers. We also have containers to keep environments identical across machines and, finally, we arrive at Lambda, which is service that runs code but doesn’t require you to set up the server. It takes care of all of that automatically, and you only pay for the time that the function is actually running. So, if your function takes 500ms to run, you pay for 500ms to run it once.

Lambda is trigger-based. When it receives a trigger, it executes a specified function. The trigger can be a variety of actions with AWS services, such as when a file shows up on S3 or an Alexa skill is requested.

Lambda is serverless, as far as you are concerned.

Lambda is stateless. This means you can’t rely on local states, so, if you need to retain information between function calls, you have to save that data somewhere else.

Lambda executes functions. Each function:
- Is code in a container on LXC on AMZN Linux
- Does not have restrictions on what it can do
- Is priced based on exec time, quantity and memory
Lambda links to various AWS services, such as:
- Files (S3)
- Database Updates (DynamoDB)
- Streams (Kinesis)
- Messages (SNS, SES)
- IoT (Echo, API Gateway)
Matt mentioned several good use cases, such as bootstrapping (the first 1M requests per month are free), cron jobs, file manipulation, API translation, social media stream processing, scaling infrastructure, mobile/IoT backends and Alexa skills, to name a few.

He also mentioned three specific examples to explore:
Matt then outlined how you can get started and noted that setup is simple as long as you think of application development as simple (write the code, set up on AWS, and debug). He walked through a specific example of automatically resizing images on S3 when they are added. You can watch his full presentation with screenshots here.

Matt also provided a list of some AWS Lambda Tools:
Matt rounded out his 30-minute presentation with a discussion regarding the need for monitoring and how to use Lambda for monitoring.

You can watch Matt’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
August 17, 2017
Automation: Critical Missing Pieces in the CD Puzzle

As enterprises accelerate the frequency with which they develop and release new code into production to meet the ever-mounting demands of their consumers, there’s growing pressure on networking and security operations teams to keep pace with their DevOps counterparts. And it’s not hard to see why. According to Forrester, nearly four out of five business leaders believe it is IT’s responsibility to ensure the network can support the company’s digital transformation plans.

And yet, 67 percent of businesses believe the network is the bottleneck—and with good reason. The current networks that NetOps and SecOps teams manage are highly complex, distributed entities that have historically been configured and provisioned manually—and often, in silos. Yet, as DevOps accelerates ahead at lightning pace, underpinned by the continuous integration, delivery and deployment (CI-CD) approach, NetOps and SecOps teams are scrambling for new ways to gain speed to more quickly launch, scale and secure applications and code updates in production.

The Need for Speed

Enterprises are increasingly leveraging DevOps principles to execute daily deployments into their multicloud test and production environments. For example, Etsy deploys 80 times per day, while Netflix and Amazon deploy up to thousands of times a day. In this new world order, collaboration, communication and synchronization across development, operations, network and security teams is essential.

This also means that testing of the network and application services infrastructure must happen concurrently with the application development and testing processes—not afterward, which can dramatically slow down the agile CI-CD process. Similarly, security cannot be retrofitted after the development and test phases are complete. A recent study by Puppet Labs found that high-performing teams spend 50 percent less time remediating security issues when security is addressed at every stage of the software development and delivery cycle.

New CI-CD Tools Deliver on the Promise of DevOps

New innovations in virtualization and programmability of network and application service infrastructure are making it possible to test policies and configurations prior to production—even in highly complex environments. Powered by validated templates and APIs, these network orchestration and application services tools help automate task execution for scale, preserve provisioning and configuration policies for consistency and bolster security, while dramatically reducing management time cycles and the risk of human error. They empower NetOps and SecOps teams to leverage DevOps principles and apply them to network and security during the development, deployment and delivery of multi-cloud apps and code updates.

Similarly to how software-defined components such as servers and storage are provisioned, network orchestration tools leverage a software-defined infrastructure (SDI) approach to rapidly provision network services across multiple network devices. These tools support flexible application provisioning across physical and virtual environments, help identify and address issues quickly and improve overall network performance. In the CI-CD model, network orchestration helps NetOps and SecOps teams ensure that all production-critical application and network elements—such as availability, core network services, security services and performance—are accounted for and are rolled out in proper sequence and at scale.

Application services automation focuses on application availability, performance, and security. While automation used to focus solely on simple load balancing, it now also includes a range of solutions that help network teams configure and automate application services during the test phase. These include web app and network firewalling, DDOS protection, application acceleration, WAN optimization, DNS services and TCP optimization, among others. In the dev and test phases of the CI-CD process, automation allows for faster configuration and deployment of network settings and policies. Similarly, in the production phase, automation significantly speeds up and simplifies areas such as policy adjustment and other time-intensive processes by eliminating the need to reconfigure underlying infrastructure. This means that ticket resolution and necessary changes can be completed in a matter of minutes versus weeks. At the time of rollout, application services automation toolsets also help to ensure the infrastructure is ready to perform at peak levels.

Network and application services automation innovations—the critical missing pieces in the continuous deployment puzzle—work together to ensure the entire DevOps team—including network and security—can deliver and deploy applications with the speed, reliability and security necessary to meet today’s business challenges.

About the Co-Author / Ranga Rao

Ranga Rao is Director of Product Management & Solutions Engineering at Cisco Systems, where he leads a team of product managers, architects/TMEs and software engineers to collaborate with partners to define use cases and build software integrations for leading Cisco solutions including Cisco ACI, Nexus 9000 and Tetration Analytics. Ranga earned an M.S. in Computer Science at Stanford after receiving his B.E. from the National Institute of Technology Karnataka. Ranga is also an author of a popular book, Programmability and Automation with Cisco Open NX-OS, published by Cisco.

— Lori MacVittie

August 9, 2017
All Day DevOps: Modern Infrastructure Automation
“We’re no longer an airline. We’re a software company with wings,” claims Veresh Sita, CIO, Alaska Airlines. The success of today’s businesses rests on software. It is an integral part of infrastructure, so we need to always ask, “How can it be better? More reliable? More secure?”

Nathen Harvey (@nathenharvey) is the VP of Community Development at Chef and the co-host of the Food Fight podcast. He is also an evangelist of automating software operations, which is what he presented on at the 2016 All Day DevOps conference.

Nathen started his talk by stating that “automation at scale is required for high velocity IT,” making the point that in today’s environment, you can’t wait months between deployments. Enterprises are moving to continuous integration/continuous deployment, and he advocates for continuous automation.

To implement automation at scale, you have to start with a dynamic infrastructure by:
- Provisioning and setting up environments
- Implementing dynamic scaling of compute resources
- Migrating legacy workloads to the cloud
- Deploy in multi-cloud and hybrid-cloud environments
- Support heterogeneous environments
Several speakers of All Day DevOps talked about automation, and all of them said a critical step is to treat infrastructure as code. According to Nathen, this means that to programmatically provision and configure components, such as servers, databases, firewalls, etc., you also have to treat it like any other code base. This includes version control and automated testing, but also:
- Automated, full-stack application policies
- Package and service installation
- Versionable, testable and repeatable workflow
- Scalable application policies
- Management of interdependencies across nodes
When you treat your infrastructure as code and deliver at scale, you can scale elegantly from one to tens of thousands of managed nodes across multiple complex environments.

For software to be effective, users have to trust that it will do what they expect it to do. This is true from handheld calculators to self-driving cars, and it is also true when we automate our development and operations. Nathen advocates that we have to build a pipeline to production that creates faith in us that our code is ready for production. This pipeline must:
1. Test the code/verify locally
2. Verify in pipeline with automated testing
3. Approve with another engineer (aka code review or Nathen’s “four eye rule”)
4. Build an artifact
5. Deploy into acceptance environment to automatically test the artifact
6. Ask someone else, “Do we want to ship this?”
7. Ship the code with a click of a button
8. Union (bring together dependencies and run tests)
9. Rehearsal
10. Delivered (ship to customers)
What does the future hold?

Infrastructure automation is NOT enough. Why? Auditors show up to audit your infrastructure and applications. The problem is, they use different tools—Excel spreadsheets, notebooks, PDFs, etc. This makes it very difficult to audit. Nathen advocates managing compliance documents as code, too. When we translate compliance into code, it can be executed as part of the pipeline so we can test for compliance throughout the entire life cycle.

Finally, Nathen stresses the need to have application automation, too. Modern applications look like infrastructure as code.

Developers can make applications that operations can deliver and run in production. This makes it easier to consume, run, and keep applications healty. Operable applications need to be:
- Isolated
- Immutable
- Configurable
- Built on a common interface for monitoring health
- Rebuildable from sources
- Reside in common packaging
- Maintain runtime independence
As enterprises implement infrastructure, application, and compliance automation, it creates a freedom for people to focus, and people are the key.

Nathen notes that, “DevOps is a cultural revolution and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.” He underscores that people are the future of automation. Automation places the priority on people, which enables great companies and great products.

You can watch Nathen’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online on Oct. 24.
— Derek E. Weeks
August 2, 2017