devopsy.kinsta.cloud

Tag: all day devops

Characterizing and Contrasting Container Orchestrators
Admiral Calcote, also known as Lee Calcote (@lcalcote) or the Ginger Geek to his friends, gave a presentation titled, “Characterizing and Contrasting Container Orchestrators,” at the 2016 All Day DevOps conference.

Okay, he isn’t really an admiral—nor does anyone call him that—but he used the title “admiral” to describe what container orchestrators do, relating it to an admiral directing a fleet of container ships. You could also say that they are like the conductor of an orchestra, directing the individuals to work together as a group toward a common goal while each musician is still able to play their own instrument.

Lee is the head of Technology Strategy at SolarWinds, and for his talk, he discussed four open-source container orchestrators: Nomad, Swarm, Kubernetes and Mesos-Marathon.

He emphasized the obvious: There is no one perfect solution. Each organization is different, so for each solution, he looked at:
- Genesis and purpose
- Support and momentum
- Host and service discovery
- Scheduling
- Modularity and extensibility
- Updates and maintenance
- Health monitoring
- Networking and load balancing
- Secrets management
- High availability and scale
Lee noted that while there are many core capabilities, any orchestrator must have cluster management and scheduling.

He then dove deeper into the four solutions. Below are summaries (the full talk is chock full of information and is online here):

Nomad
- Designed for both long-lived and short-lived batch processing workloads
- Cluster manager with declarative job specifications
- Ensures constraints are satisfied and resource utilization is optimized by efficient task packing
- Supports all major OSs and workloads
- Written in Go and with a Unix philosophy
- Host discovery: Gossip protocol – Serf is used; servers advertise full set of Nomad servers to clients; creating federated clusters is simple
- Service discovery: Integrates with Consul
- Scheduling: two distinct phases – feasibility checking and ranking; optimistically concurrent; three scheduler types when creating jobs
- Uses task drivers to execute a task and provide resource isolation, but it does not support pluggable task drivers
- Built for managing multiple clusters/cluster federation
Docker Swarm 1.12
- Simple and easy to set up
- Architecture is not as complex as Kubernetes and Mesos
- Written in Go – lightweight, modular and extensible
- Strong community support
- Host discovery: used in the formation of clusters by the Manager to discover Nodes (hosts); pull model – worker checks in with the Manager
- Service discovery: Embedded DNS and round-robin load balancing
- Scheduler is pluggable and is a combination of strategies and filters/constraints
- Ability to remove “batteries”
- Rolling updates are supported
- Managers may be deployed in a highly available configuration, but does not support multiple failure isolation regions or federation
Kubernetes
- An opinionated framework for building distributed systems
- Written in Go and is lightweight, modular and extensible
- Led by Google, Red Hat and others
- Young – about 2 years old
- Robust documentation and community
- Scheduling is handled by kube-scheduler
- Pluggable architecture and an extensible platform
- Choice of database for service discovery or network driver and container runtime
- Supports rolling back deployments, automating deployments and rolling updating applications
- Inherent load balancing
- Uses Pods, an atomic unit of scheduling. Each pod has its own IP address, no NAT required and intra-pod communication via localhost
Mesos-Marathon
- Mesos is a distributed systems kernel
- Mesos has been around the longest (since 2009)
- Mesos is written in C++
- Marathon is a framework that runs on top of Mesos
- Mesos is used by Twitter, AirBnB, eBay, Apple, Cisco, and Yodle
- Marathon is used by Verizon and Samsung
- Mesos-DNS generates an SRV record for each Mesos task
- Marathon ensures that all dynamically assigned ports are unique
Wrapping up, Lee provided the following overview comparing the different container orchestration solutions.

Lee packed a tremendous amount of information into his talk. If you use containers, his talk is worth your time and is available online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
September 20, 2017
Speaking for the Dead: Are ‘Agile’ and ‘Monolithic’ Actually Good?
What if Agile is the product of getting bored at work, microservices are the result of us needing a challenge and scrums are the result of an obsessive project manager? Do we all think waterfall development is a dinosaur whose extinction is deserved because of what we have been told and not because of reality? Do we think Agile is better because it simply sounds desirable?

Michael DeHann (@laserllama), creator of Ansible, spoke at the 2016 All Day DevOps conference, which espouses the benefits of DevOps, and, by association, Agile and microservices. It essentially says, “Let’s step back and evaluate the past. Maybe waterfall development and monolithic architectures are actually a good thing?”

Michael has been a software engineer and created a key tool in the IT automation toolbox: Ansible. His arguments deserve a look, because re-evaluation can keep you from making mistakes and improve processes going forward.

Michael started his talk by making the point that in this day and age, it is waterfall versus Agile and monolithic architecture versus microservices. The assumption is that if you are still developing software with the waterfall method and/or developing monolithic applications, you might as well be riding a horse and buggy to work and dialing up coworkers on your rotary phone.

Throughout his talk, he asked questions to make his points, beginning with, “Is big design up front bad?” with the conclusion, “I don’t think so.” His argument is that design and requirements at the beginning of the process enable a better production. Marketing and engineers hammered out requirements based on needs, abilities and budgets. There was plenty of time to research and design solutions. Both sides worked together, raising questions and comments and documenting mutual agreement. This presented an accurate view of the strengths of various team members and dependencies between tasks and task orders. And, perhaps most important, engineering was engineering—the engineers had greater responsibility and less hand-holding.

Contrasting that with his view of Agile, big design up front is labeled as bad or counterproductive, engineers are viewed as interchangeable and scope is more unpredictable. Estimates are encouraged to not be time-based, which often results in a constant crunch time. Finally, meetings can consume a significant portion of team members’ time and often include “games reminiscent of Kindergarten, such as Fist of Five or Planning Poker.” Even in lightweight Agile systems, testing can suffer, requirements are more vague and there is no mutual sign-off.

Michael argues that there can, in fact, be a middle ground where teams:
- Do share design
- Have an accessible, effective product owner
- Know hard requirements versus soft requirements
- See test-driven development and test automation as super important
- Understand that Agile and iterative development can be closer than we think
Agile and DevOps are also often associated with microservices versus monolithic applications, but according to Michael, “What’s so bad about a monolith?”

Sample Monolithic Diagram

Sample Microservices Diagram

An example microservices shop has 150 services, many thousands of Jenkins builds per day, eight Amazon regions including dev, staging and production and thousands of EC2 instances. This can result in Jenkins configuration and security enforcement drifts, suboptimal code sharing and DevOps writing lots of custom code to manage it.

Michael contrasts that with monoliths, which have:
- Increased ability to develop on one laptop
- Interfaces that benefit from static typing versus design-by-contract
- Very small numbers of different server/image types in production
- Local function calls that are much faster than RPC
- Few instances sitting around with 5 percent utilization
- No need for “containers” to get around above utilization problem
- Greatly simplified debugging
- Easy code-sharing
- Very simple version upgrades
Rather than argue that all applications should be monolithic, he makes the point that many of the concerns that resulted in microservices could be solved with improved communications. He contends that microservices are a tech solution to a people problem.

Michael concludes that:
- Labels cloud true meanings,
- Monolithic architectures are more efficient,
- Agile can be less efficient,
- Microservices attempt to solve people problems but introduce new technical and people problems, and
- We should try and separate the labels from the true meaning of things.
Most importantly, though, is the fact that there is something to learn from everything. So, even if you don’t agree with Michael’s conclusion, you should listen to his talk online here. You might learn an unexpected lesson.

If you missed any of the other 30-minute long presentations from 2016 All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
September 13, 2017
Liberty Mutual: Taking an Organic Approach to DevOps
Some people are directors, managers, and administrators. Others are disrupters.

Eddie Webb (@edwardawebb) is an IT disrupter for Software Development Platforms at Liberty Mutual and was a presenter at the 2016 All Day DevOps conference. His talk, “Organically DevOps: Building Quality and Security into the Software Supply Chain at Liberty Mutual,” looked at Liberty Mutual’s transformation to continuous integration, continuous delivery and DevOps. For a large, heavily regulated industry, this task can be not only daunting, but also viewed by many as impossible.

Eddie’s talk looks at the origins of DevOps at Liberty Mutual, the challenges of centralizing DevOps, and how to make it easy for developers and operations to do the right thing. He noted that DevOps is not Liberty Mutual’s goal; continuous delivery is the goal and DevOps is the enabler.

It all started at Liberty Mutual in 2004 with a vision: “Black-box reusable services that benefit the business.” The company also had a concept that “developers own the quality of the code; we have no QA team.” Employees were empowered with this “concrete vision but malleable strategy,” as Eddie said. The company gave them time and space to explore solutions, because ownership increases adoption.

The team started with fixing what frustrated them, including CI/CD pipelines, artifact and code repositories, elastic cloud-native runtime and Agile planning and delivery.

But what about centralizing DevOps?

The reality is that tools can be centralized, but culture cannot.

Eddie laid out an example of a large organization’s IT structure, saying that this is how things work at Liberty Mutual. Eddie’s team is the Central IT, and their customers are the Market IT team, not Liberty Mutual’s end customers. As Eddie pointed out, the more layers you create, the more friction that enters the system. Often, organizations try to reduce the friction through microfixes, but Eddie’s team asked how to change the culture to reduce the friction.

This required distributing innovation and centralizing the tools that work. As an example, if you build a manufacturing plant, you wouldn’t build your own power plant. If you are in IT, why build your own cloud?

This is the Netflix way. At Netflix, teams are independent, but responsible. Engineers can choose the best tools for the job. No one single team is responsible for innovation. However, the Engineering Tools team helps direct other teams’ experimentation toward new products without getting in the way of innovation.

Eddie quoted the book, “Drive,” when looking at how to motivate innovation in complex work environments. The author says if you want to incentivize or motivate cognitive tasks, you need three things: autonomy, mastery and purpose.

When you centralize non-value added platforms, it enables teams to:
- Master their specific domains of business;
- See clear purpose in everything they build; and
- Maintain autonomy of choice of frameworks, patterns, and approach.
When you decentralize, it raises the question: What about security? Developers are humans, and humans inherently try to avoid tasks they see as unnecessary or overly burdensome, and this becomes easier when the compliance person isn’t there (think of how we handle speed limits). Traditional security measures erode profit and market share, and, from a stat Eddie quoted from Gartner, 77 percent of IT security professionals agree that security slows down IT.

Whether you are centralized or decentralized, security (and other compliance-oriented tasks) can increase adoption rates by making it easy to do the right thing. How do you do this? Eddie proposes governing with tools (think of speed governors in fleet cars).

You also need to have immutable environments, because, “We can’t trust humans to manipulate production.” So, again, you have to make the path you want them to take, and make this the easiest path. As Eddie notes, “You can’t put too much friction into the pipeline, because developers are good at bypassing friction.” You have to make your CI/CD pipelines the easiest way to get ideas to production by. You also have to abstract complexity and automate policies. You have to have an “opinionated platform, and observe your customer so you can address their pain.”

Eddie concluded with the following final points:
- Don’t mandate DevOps. Give employees the chance to master their discipline with examples to set and follow.
- Favor deep end-to-end accomplishments over broad but incremental steps forward. Focus on taking the right teams far before encouraging broad adoption.
- Centralize the platforms and tools that your teams shouldn’t be thinking about. Provide foundational services/commodities and let teams stay on purpose.
- Incorporate contributions from everyone; don’t stifle autonomy. Stay open to new ways of working.
- Challenge security policies, but respect intentions. Find new ways to enforce concerns without abandoning precaution.
You can listen to Eddie’s entire talk online here. You might learn an unexpected lesson.

If you missed any of the other 30-minute long presentations from 2016 All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 100 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
September 12, 2017
DevOps and Security is Like Smoking Meat
It isn’t everyone who thinks, “Doesn’t Ubuntu remind you of wild boar?” Or labors over his pit of slow-roasted pork shoulder while contemplating containers. Or dwells on e2e testing while mesquite smoke permeates spareribs.

But, Apollo Clark (@apolloclark) does. Apollo is a foodie of smoked meats (is that a meatie or smokie?), working to master the craft: understanding different cuts, what each type of wood adds, and the subtleties of sauces. But, alas, it is a hobby; security and DevOps are his career.

Being passionate about both, he naturally sees parallels between the two, so Apollo presented at the 2016 All Day DevOps conference with a session titled, “What Smoking Meat Taught Me About DevOps and Security.”

Understandably, you are asking yourself, “What does smoking meat have to do with DevOps and security?” Apollo notes both have tremendous complexity and nuances, there are multiple ways of getting the job done, lots of ways to mess it up, a couple of ways of doing it right, and you are always learning.

For both, there are many tools and processes to get the job done. For smoking, your wood is a critical component, along with time. Let’s look at some parallels:
- Oak = unit testing
- Maple = coverage
- Apple = dynamic analysis
- Peach = static analysis
- Mesquite = e2e testing. People love it, but it is difficult to handle
- Wine barrel = browser support. This is when you are doing it really well
- Bourbon barrel = device support. Pretty complicated
- Smoking time = testing. You can test or smoke for five minutes or five hours, but there is a sweet spot. You can oversmoke meats and you can overtest.
Getting hungry?

Meat is critical to a meal of smoked meat, but so are many other components. Likewise, software applications take a suite of other tools and components to deliver the full package. Get your taste buds ready, because Apollo likens all of the goodness that goes with smoked meats to tools you use to keep applications running:
- Sauces = auto-scaling. They both can get pretty complicated. Always a little different, but you have to make it work for you.
- Bread = monitoring, which is the bread and butter of infrastructure. Make sure you have it up and down your stack.
- Salad = system logs. Not the sexiest things, but you can rely on them.
- Fruit = custom application logs. Takes a lot of time to pair them, but takes a really good thing and makes them better.
- Beer = firewall. You should always have both.
- Wine = Intrusion detection systems. These are your fine wines.
- Whiskey = IR training. It takes time and there are so many ways of doing it. When things break, we have procedures on how to deal with them.
When smoking meats and in DevOps and security, Apollo asks and answers, “Do we have to do everything? No, but the more we do, the better it will taste.

“Does it cost money and take time? Yes, but you can do great things even without money and time. You can use cheap meats and you can use open source. Start simple, build up complexity, and always be learning.” Constantly ask yourself, “Am I better today than I was yesterday.” If not, be better.

You can watch Apollo’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
September 6, 2017
Securing Immutable Servers in a Serverless World
Snowflakes are beautiful, unique creations. But, let’s keep them in nature. They don’t belong in our server infrastructure. Snowflake servers, where every configuration is just a little different, can introduce unnecessary security vulnerabilities and complications. While common in IT infrastructure, in the DevOps realm they are gradually becoming ancient history.

At All Day DevOps 2016, Erlend Oftedal (@webtonull), with Blank and head of the OWASP Norway chapter, discussed the benefits of immutable infrastructure practices within serverless architectures. Erlend walked us through the positive effects on security as well as offered insight on potential problems.

Erlend outlined the progress from physical servers to immutable servers, discussing the benefits of each step and the concerns that lead to the next phase, ending with immutable servers—that is, they can’t be changed. Once set up, they are set. Automated processes coupled with immutability principles can eliminate snowflake servers.

Immutability isn’t a new concept. Some classic examples include live Linux CDs, internet cafes, schools, libraries and hotel business centers. To set up immutable servers, Erlend outlines these steps:
1. Find a suitable base image
2. Build a specific image for the application
  1. Dependencies
  2. Application
  3. Secrets
3. Deploy image to the server(s)
4. Repeat steps 2-3 for new versions
Of course, this immediately raises questions:
- What about data? He notes that because data inherently changes, data has to be stored externally.
- How do you deploy changes? Well, you don’t change the server. You take one out and put a new one in. What if that one fails? Roll back to the previous one.
- How do you manage secrets? You can build secrets in, leverage cloud key management, or use third-party services.
- How do we apply security patches? If truly immutable, you deploy a new server with the patch. However, you can have a semi-immutable server that allows for automatically installing patches but still doesn’t allow for logins or manual changes.
- Speaking of security, what are the security advantages? For starters, if an attacker is in your server, it will be thrown out when you change servers, although it may come back if the vulnerability isn’t patched. It also allows for very specialized images by removing all unnecessary packages. With these specialized images, you can more easily audit and monitor the system for unexpected file changes, logins and connections.
Erlend discussed containers because they are a step closer to a serverless world. Containers isolate processes and all run on the same OS. Of course, you can make them immutable, but regardless of whether they are, there are some key security best practices you’ll want to consider:
- Don’t run as root inside container
- Implement user namespaces so the root account within the container is separated from the root account outside the container
- Utilize the least privileges necessary for capabilities and resources
- Keep them up-to-date
- Automate scanning using tools such as Sonatype Nexus, Twistlock, Clair, or Aqua
- Separate untrusted and trusted containers
Erlend also dove into serverless architecture. While there is debate over what exactly qualifies as serverless, there are five principles that Erlend highlighted:
- Use a compute service to execute code on demand
- Write single-purpose, stateless functions
- Design push-based, event-driven pipelines
- Create thicker, more powerful front ends
- Embrace third-party services
Migrating to immutable, serverless infrastructure is a topic to be discussed well beyond this blog post and even Erlend’s session. Whatever you do in the future, though, take his parting words to heart: “The cloud can solve many of your problems, but in the end you cannot transfer your responsibility for security.”

Erlend’s session was packed with information. If this piqued your interest, you should watch his complete session from All Day DevOps 2016 (just 30 minutes). The other 56 presentations from the conference are also available online, free-of-charge here.

Also, this year’s All Day DevOps is Oct. 24. It’s online and free, so be sure to check out www.alldaydevopsy.kinsta.cloud to learn more and register.

— Derek E. Weeks
August 28, 2017
All Day DevOps: Taking Lambda to the Max
Lambda is the 11th letter of the Greek alphabet, and it is also the name of the Amazon Web Services (AWS) service that lets you run code without actually configuring a server. Amazon chose the name Lambda because—well, okay, I have no idea. Comment here if you know.

Thankfully, you don’t need to know that to understand how to utilize Lambda. Matt Williams, DevOps Evangelist at Datadog (@technovangelist), outlined how to set up AWS Lambda at the 2016 All Day DevOps conference in his talk titled, “Hold the Infrastructure: Getting the Most Out of Lambda.”

First, to understand what Lambda is, we must look at the evolution of servers.

We started with physical boxes—we had to buy and maintain these boxes, and no matter how much we used, we had to pay for all of it. We then evolved to multiple virtual machines on a physical server so that we could maximize the resources of physical boxes. Now, AWS and similar cloud services are common. We create a virtual machine that we need, and the service provider maintains the physical servers. We also have containers to keep environments identical across machines and, finally, we arrive at Lambda, which is service that runs code but doesn’t require you to set up the server. It takes care of all of that automatically, and you only pay for the time that the function is actually running. So, if your function takes 500ms to run, you pay for 500ms to run it once.

Lambda is trigger-based. When it receives a trigger, it executes a specified function. The trigger can be a variety of actions with AWS services, such as when a file shows up on S3 or an Alexa skill is requested.

Lambda is serverless, as far as you are concerned.

Lambda is stateless. This means you can’t rely on local states, so, if you need to retain information between function calls, you have to save that data somewhere else.

Lambda executes functions. Each function:
- Is code in a container on LXC on AMZN Linux
- Does not have restrictions on what it can do
- Is priced based on exec time, quantity and memory
Lambda links to various AWS services, such as:
- Files (S3)
- Database Updates (DynamoDB)
- Streams (Kinesis)
- Messages (SNS, SES)
- IoT (Echo, API Gateway)
Matt mentioned several good use cases, such as bootstrapping (the first 1M requests per month are free), cron jobs, file manipulation, API translation, social media stream processing, scaling infrastructure, mobile/IoT backends and Alexa skills, to name a few.

He also mentioned three specific examples to explore:
Matt then outlined how you can get started and noted that setup is simple as long as you think of application development as simple (write the code, set up on AWS, and debug). He walked through a specific example of automatically resizing images on S3 when they are added. You can watch his full presentation with screenshots here.

Matt also provided a list of some AWS Lambda Tools:
Matt rounded out his 30-minute presentation with a discussion regarding the need for monitoring and how to use Lambda for monitoring.

You can watch Matt’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
August 17, 2017
All Day DevOps: Modern Infrastructure Automation
“We’re no longer an airline. We’re a software company with wings,” claims Veresh Sita, CIO, Alaska Airlines. The success of today’s businesses rests on software. It is an integral part of infrastructure, so we need to always ask, “How can it be better? More reliable? More secure?”

Nathen Harvey (@nathenharvey) is the VP of Community Development at Chef and the co-host of the Food Fight podcast. He is also an evangelist of automating software operations, which is what he presented on at the 2016 All Day DevOps conference.

Nathen started his talk by stating that “automation at scale is required for high velocity IT,” making the point that in today’s environment, you can’t wait months between deployments. Enterprises are moving to continuous integration/continuous deployment, and he advocates for continuous automation.

To implement automation at scale, you have to start with a dynamic infrastructure by:
- Provisioning and setting up environments
- Implementing dynamic scaling of compute resources
- Migrating legacy workloads to the cloud
- Deploy in multi-cloud and hybrid-cloud environments
- Support heterogeneous environments
Several speakers of All Day DevOps talked about automation, and all of them said a critical step is to treat infrastructure as code. According to Nathen, this means that to programmatically provision and configure components, such as servers, databases, firewalls, etc., you also have to treat it like any other code base. This includes version control and automated testing, but also:
- Automated, full-stack application policies
- Package and service installation
- Versionable, testable and repeatable workflow
- Scalable application policies
- Management of interdependencies across nodes
When you treat your infrastructure as code and deliver at scale, you can scale elegantly from one to tens of thousands of managed nodes across multiple complex environments.

For software to be effective, users have to trust that it will do what they expect it to do. This is true from handheld calculators to self-driving cars, and it is also true when we automate our development and operations. Nathen advocates that we have to build a pipeline to production that creates faith in us that our code is ready for production. This pipeline must:
1. Test the code/verify locally
2. Verify in pipeline with automated testing
3. Approve with another engineer (aka code review or Nathen’s “four eye rule”)
4. Build an artifact
5. Deploy into acceptance environment to automatically test the artifact
6. Ask someone else, “Do we want to ship this?”
7. Ship the code with a click of a button
8. Union (bring together dependencies and run tests)
9. Rehearsal
10. Delivered (ship to customers)
What does the future hold?

Infrastructure automation is NOT enough. Why? Auditors show up to audit your infrastructure and applications. The problem is, they use different tools—Excel spreadsheets, notebooks, PDFs, etc. This makes it very difficult to audit. Nathen advocates managing compliance documents as code, too. When we translate compliance into code, it can be executed as part of the pipeline so we can test for compliance throughout the entire life cycle.

Finally, Nathen stresses the need to have application automation, too. Modern applications look like infrastructure as code.

Developers can make applications that operations can deliver and run in production. This makes it easier to consume, run, and keep applications healty. Operable applications need to be:
- Isolated
- Immutable
- Configurable
- Built on a common interface for monitoring health
- Rebuildable from sources
- Reside in common packaging
- Maintain runtime independence
As enterprises implement infrastructure, application, and compliance automation, it creates a freedom for people to focus, and people are the key.

Nathen notes that, “DevOps is a cultural revolution and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.” He underscores that people are the future of automation. Automation places the priority on people, which enables great companies and great products.

You can watch Nathen’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online on Oct. 24.
— Derek E. Weeks
August 2, 2017
From ‘Water-Scrum-Fall’ to DevSecOps
As organizations abandon the waterfall method of software development for Agile, many are stuck in what Hasan Yasar terms Water-Scrum-Fall. That is, the organization has not effectively embraced Agile and DevOps principles and remains in silos with no links to business goals. Enter DevOps, an extension of Agile thinking. While Agile embraces constant change and embeds the customer into the process, DevOps embraces constant testing and delivery and embeds operations into the team to internalize expertise on deployment and maintenance.

This is how Hasan started his talk, “Multi Security Checkpoints on DevOps Platform,” at the All Day DevOps conference.

In his talk, Hasan lays out a plan to get organizations to DevSecOps. Really, DevOps is a risk mitigation strategy, built on situational awareness, automation, and repetition. But, security is where a lot of DevOps implementations fall down. The goals for each organization should be:
- Protecting private user data
- Restricting access to data/systems
- Protecting company data/intellectual property
- Standards compliance
- Safeguarding disposition/transition
But, how do organizations get there?

First, integration and communication. Every point of the product development life cycle should be integrated and communicating, including among the tools. Once this is achieved, you can automate many, if not most, of the tasks. The automated steps are the ones that require less human actions/input to the software development process. This allows everyone to focus on innovation and better code and less on tasks that can be automated by autonomous systems. Also, tasks that can be automated are less susceptible to errors.

Of course, it is the team that ultimately designs, develops and delivers the software. Your team consists of development, IT operations, quality assurance and security. Each has its own skill set and focus, and the overlap is Secure DevOps.

The team is in place, processes are automated and development has started. Development in this day and age has evolved tremendously from even just a few years ago. Previously, software was limited to size, function and audience, and the supply chain was practically non-existent. Your team built each component. Now, development has grown beyond the ability of an organization to develop outside of its core competencies. The supply chain now involves many sources for the code. It is more like plug and play, and this creates lots of vulnerabilities.

Hasan notes the software supply chain risk factors:
- Supplier capability – Does the supplier follows practices that reduce supply chain risks?
- Product security – Is the delivered or updated product acceptably secure?
- Product distribution – Does the method of transmitting the product to the purchaser guard against tampering?
- Operational product control – Is the product used in a secure manner?
To reduce your supply chain risk, Hasan recommends:
- Ensure supplier security commitment;
- Evaluate a product’s threat resistance;
- Create a centralized private repository of vetted third-party components for all developers;
- Establish good product distribution practices; and
- Minimize variation of components to make things easier.
Finally, as you transition to DevSecOps, remember that security must be addressed without breaking the rapid delivery, continuous feedback model.

You can watch Hasan’s entire talk online here for more details and other tips.

If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
July 27, 2017
How To Fully Automate CI/CD – Even Secrets
Imagine a world where your continuous integration/continuous deployment environment is 100 percent automated, including the passing of credentials.

Capital One, a leading U.S. bank, has achieved this, and Andrey Utis, director of Software Engineering at Capital One, outlined how the company achieved this in his talk, “Application Secret Management with KMS,” at the All Day DevOps conference.

Andrey started by stating what we all know: Security needs to be at the forefront. Additionally, a message we heard over and over at the All Day DevOps conference is to automate where you can automate. That combination can be daunting and seemingly impossible—how can we automate security when someone needs to enter credentials and we can’t store credentials where everyone can get to them? After all, credentials provide access to databases that often contain personally identifiable information and other protected information. A breach could devastate companies and people, whether intentional or accidental.

Andrey’s team uses Amazon Web Services (AWS), so they give instances IAM roles that allow them access to other AWS resources. What is the key (pun intended)? On an AWS EC2 instance, there is a magic IP address to which you can make an HTTP call and it will return temporary AWS keys. Those keys then make the API calls to different database services.

Here is the solution: AWS KMS is encryption as a service. KMS Context allows you to add “salt” to the encryption. You can only decrypt with the same “salt.” KMS Key Policies restricts which IAM role can decrypt with the key/salt. Master KMS keys can be used only to decrypt their own keys.

Below is a code sample of a KMS policy:

This allows multiple applications to be on one AWS account while limiting access of developers to the applications they are authorized for.

Here is the actual protocol:
1. Create a KMS master key (can be shared by multiple apps)
2. Create an IAM role for your server
3. Add KMS key policy that allows decrypt to your IAM role for a specific context
4. Encrypt your secret with the key and context and store the value in GitHub
5. At deployment or runtime, invoke KMS API to decrypt
There is a broader issue with IAM roles because credentials are generated by calling the “magic” metadata IP address 169.254.169.254. Developers in production, even with “read only” access to the instance, could call the KMS API to decrypt the secret. Developers should not be able to generate production IAM credentials at all, so you block the magic IP address with this code:

To automate it and make it reusable, the team created a Chef cookbook, which they call a “briefcase,” to abstract decryption of secrets. They also have the iptables cookbook to block the metadata IP address for all except a whitelist of user groups, such as root and any application-specific group that makes AWS API calls (see above code).

Andrey mentioned that Vault may have a viable solution soon. It uses signed EC2 Identity Document to verify the caller. The current downsides are that it only supports authorization by AMI ID (but should support more soon) and secrets are not source-controlled/versioned. As Andrey noted, that is not ideal for “configuration secrets” such as database passwords.

In a follow-up question, Andrey was asked, “What were the lessons that were most critical to learn?” He answered, “This is a new field. Most companies either don’t fully automate or don’t fully secure the entire pipeline. So, there is little information out there.”

Well, there is some more information in Andrey’s complete talk, which you can watch online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.

Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online on Oct. 24.

— Derek E. Weeks
July 14, 2017
Continuous Everyone: Engaging People Across the Deployment Pipeline
We have continuous integration and we have continuous deployment, but what’s continuous across all of what we do is people. Even when tasks are automated, someone wrote the automation. So, Jayne Groll evangelizes about “continuous everyone.”

Jayne is the CEO of the DevOps Institute and the author of “Agile Service Management Guide.” She talked about continuous everyone at the 2016 All Day DevOps conference. She describes it as “about people, culture and collaboration mapped into your value streams.”

We all hear about the importance of people, culture and collaboration, but how well are they integrated into our organizations and processes? Is it in everything we do? Is it inherently continuous?

It all starts with culture. The Business Dictionary defines culture as, “The values and behaviors that contribute to the unique social and psychological environment of an organization.” As Jayne points out, culture influences how people think, feel, interact and behave at work.

We all have a vision of a euphoric culture. For most, that doesn’t reflect the reality of the current culture, which is generally Games of Thrones. Everyone wants to sit on the Iron Throne: ITIL/ITSM, DevOps, Security, Lean, Agile, Automation, and CI/CD. Traditionally, each of these functions has operated in isolation, convinced they are the most important cog in the machine. Now, DevOps has forced us to talk for the first time in 40 years.

Once we acknowledge a problem, there are only 11 more steps. So, next, improving culture begins with addressing the root cause of cultural fails:
- Language barriers (industry language)
- Tribal knowledge
- Poor communication
- Thick silos
- Disparate processes
- Incompatible automation
- Inadequate collaboration
These cultural fails—both the cause and a symptom of thick organizational silos—must be broken down because they cage people’s creativity, and creativity is something you can’t automate.

Now, people typically participate in the Continuous People Pipeline like this: Develop → Commit → Build → Test → Deploy and APIs built in between. Jayne advocates that we need to focus on Application People Interfaces between engineers and testers and ops teams. We need to let computers do the boring, repetitive tasks and let people solve problems and focus on high-value activities.

Since not everything can be automated, continuous engagement is critical. While some advocated for “NoOps” (everything is automated), we should focus on “NewOps” to capture the talent and expertise in operations we need to leverage for tasks only people can do, such as:
- Provide input and feedback
- Design pipelines
- Configure automation and APIs
- Review test and other output
- Design metrics
- Facilitate feedback loops
- Interface with the customer
- Solve problems
- Exchange ideas
- Be innovative
The bottom line is that this could be the beginning of a beautiful friendship. We need to reinvigorate the lost art of the dialogue. Actually talk to each other.

This begins with trust. Every aspect of the Continuous People Pipeline is filled with expertise, integrity and people whot want to learn. There is a trust. You have to trust each other’s strengths and take the opportunity to engage on a continuous basis.

It is only together that we can continuously improve.

You can watch Jayne’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here.

Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
June 30, 2017