devopsy.kinsta.cloud

Tag: Nexus

  • From ‘Water-Scrum-Fall’ to DevSecOps

    From ‘Water-Scrum-Fall’ to DevSecOps

    As organizations abandon the waterfall method of software development for Agile, many are stuck in what Hasan Yasar terms Water-Scrum-Fall. That is, the organization has not effectively embraced Agile and DevOps principles and remains in silos with no links to business goals. Enter DevOps, an extension of Agile thinking. While Agile embraces constant change and embeds the customer into the process, DevOps embraces constant testing and delivery and embeds operations into the team to internalize expertise on deployment and maintenance.

    This is how Hasan started his talk, “Multi Security Checkpoints on DevOps Platform,” at the All Day DevOps conference.

    Picture5.png

    In his talk, Hasan lays out a plan to get organizations to DevSecOps. Really, DevOps is a risk mitigation strategy, built on situational awareness, automation, and repetition. But, security is where a lot of DevOps implementations fall down. The goals for each organization should be:

    • Protecting private user data
    • Restricting access to data/systems
    • Protecting company data/intellectual property
    • Standards compliance
    • Safeguarding disposition/transition

    But, how do organizations get there?

    First, integration and communication. Every point of the product development life cycle should be integrated and communicating, including among the tools. Once this is achieved, you can automate many, if not most, of the tasks. The automated steps are the ones that require less human actions/input to the software development process. This allows everyone to focus on innovation and better code and less on tasks that can be automated by autonomous systems. Also, tasks that can be automated are less susceptible to errors.

    Picture6.png

    Of course, it is the team that ultimately designs, develops and delivers the software. Your team consists of development, IT operations, quality assurance and security. Each has its own skill set and focus, and the overlap is Secure DevOps.

    Picture7.png

    The team is in place, processes are automated and development has started. Development in this day and age has evolved tremendously from even just a few years ago. Previously, software was limited to size, function and audience, and the supply chain was practically non-existent. Your team built each component. Now, development has grown beyond the ability of an organization to develop outside of its core competencies. The supply chain now involves many sources for the code. It is more like plug and play, and this creates lots of vulnerabilities.

    Hasan notes the software supply chain risk factors:

    • Supplier capability – Does the supplier follows practices that reduce supply chain risks?
    • Product security – Is the delivered or updated product acceptably secure?
    • Product distribution – Does the method of transmitting the product to the purchaser guard against tampering?
    • Operational product control – Is the product used in a secure manner?

    To reduce your supply chain risk, Hasan recommends:

    • Ensure supplier security commitment;
    • Evaluate a product’s threat resistance;
    • Create a centralized private repository of vetted third-party components for all developers;
    • Establish good product distribution practices; and
    • Minimize variation of components to make things easier.

    Finally, as you transition to DevSecOps, remember that security must be addressed without breaking the rapid delivery, continuous feedback model.

    Picture8.png

    You can watch Hasan’s entire talk online here for more details and other tips.

    If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online Oct. 24.

    — Derek E. Weeks

  • How To Fully Automate CI/CD – Even Secrets

    How To Fully Automate CI/CD – Even Secrets

    Imagine a world where your continuous integration/continuous deployment environment is 100 percent automated, including the passing of credentials.

    Capital One, a leading U.S. bank, has achieved this, and Andrey Utis, director of Software Engineering at Capital One, outlined how the company achieved this in his talk, “Application Secret Management with KMS,” at the All Day DevOps conference.

    iStock-683716072-1.jpg

    Andrey started by stating what we all know: Security needs to be at the forefront. Additionally, a message we heard over and over at the All Day DevOps conference is to automate where you can automate. That combination can be daunting and seemingly impossible—how can we automate security when someone needs to enter credentials and we can’t store credentials where everyone can get to them? After all, credentials provide access to databases that often contain personally identifiable information and other protected information. A breach could devastate companies and people, whether intentional or accidental.

    Andrey’s team uses Amazon Web Services (AWS), so they give instances IAM roles that allow them access to other AWS resources. What is the key (pun intended)? On an AWS EC2 instance, there is a magic IP address to which you can make an HTTP call and it will return temporary AWS keys. Those keys then make the API calls to different database services.

    Here is the solution: AWS KMS is encryption as a service. KMS Context allows you to add “salt” to the encryption. You can only decrypt with the same “salt.” KMS Key Policies restricts which IAM role can decrypt with the key/salt. Master KMS keys can be used only to decrypt their own keys.

    Below is a code sample of a KMS policy:

    Picture9.png

    This allows multiple applications to be on one AWS account while limiting access of developers to the applications they are authorized for.

    Here is the actual protocol:

    1. Create a KMS master key (can be shared by multiple apps)
    2. Create an IAM role for your server
    3. Add KMS key policy that allows decrypt to your IAM role for a specific context
    4. Encrypt your secret with the key and context and store the value in GitHub
    5. At deployment or runtime, invoke KMS API to decrypt

    There is a broader issue with IAM roles because credentials are generated by calling the “magic” metadata IP address 169.254.169.254. Developers in production, even with “read only” access to the instance, could call the KMS API to decrypt the secret. Developers should not be able to generate production IAM credentials at all, so you block the magic IP address with this code:

    Picture10.png

    To automate it and make it reusable, the team created a Chef cookbook, which they call a “briefcase,” to abstract decryption of secrets. They also have the iptables cookbook to block the metadata IP address for all except a whitelist of user groups, such as root and any application-specific group that makes AWS API calls (see above code).

    Andrey mentioned that Vault may have a viable solution soon. It uses signed EC2 Identity Document to verify the caller. The current downsides are that it only supports authorization by AMI ID (but should support more soon) and secrets are not source-controlled/versioned. As Andrey noted, that is not ideal for “configuration secrets” such as database passwords.

    In a follow-up question, Andrey was asked, “What were the lessons that were most critical to learn?” He answered, “This is a new field. Most companies either don’t fully automate or don’t fully secure the entire pipeline. So, there is little information out there.”

    Well, there is some more information in Andrey’s complete talk, which you can watch online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.

    Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online on Oct. 24.

    — Derek E. Weeks

  • Continuous Everyone: Engaging People Across the Deployment Pipeline

    Continuous Everyone: Engaging People Across the Deployment Pipeline

    We have continuous integration and we have continuous deployment, but what’s continuous across all of what we do is people. Even when tasks are automated, someone wrote the automation. So, Jayne Groll evangelizes about “continuous everyone.”

    Jayne is the CEO of the DevOps Institute and the author of “Agile Service Management Guide.” She talked about continuous everyone at the 2016 All Day DevOps conference. She describes it as “about people, culture and collaboration mapped into your value streams.”

    Picture1.png

    We all hear about the importance of people, culture and collaboration, but how well are they integrated into our organizations and processes? Is it in everything we do? Is it inherently continuous?

    It all starts with culture. The Business Dictionary defines culture as, “The values and behaviors that contribute to the unique social and psychological environment of an organization.” As Jayne points out, culture influences how people think, feel, interact and behave at work.

    Picture2.png

    We all have a vision of a euphoric culture. For most, that doesn’t reflect the reality of the current culture, which is generally Games of Thrones. Everyone wants to sit on the Iron Throne: ITIL/ITSM, DevOps, Security, Lean, Agile, Automation, and CI/CD. Traditionally, each of these functions has operated in isolation, convinced they are the most important cog in the machine. Now, DevOps has forced us to talk for the first time in 40 years.

    Picture3.png

    Once we acknowledge a problem, there are only 11 more steps. So, next, improving culture begins with addressing the root cause of cultural fails:

    • Language barriers (industry language)
    • Tribal knowledge
    • Poor communication
    • Thick silos
    • Disparate processes
    • Incompatible automation
    • Inadequate collaboration

    These cultural fails—both the cause and a symptom of thick organizational silos—must be broken down because they cage people’s creativity, and creativity is something you can’t automate.

    Now, people typically participate in the Continuous People Pipeline like this: Develop → Commit → Build → Test → Deploy and APIs built in between. Jayne advocates that we need to focus on Application People Interfaces between engineers and testers and ops teams. We need to let computers do the boring, repetitive tasks and let people solve problems and focus on high-value activities.

    Since not everything can be automated, continuous engagement is critical. While some advocated for “NoOps” (everything is automated), we should focus on “NewOps” to capture the talent and expertise in operations we need to leverage for tasks only people can do, such as:

    • Provide input and feedback
    • Design pipelines
    • Configure automation and APIs
    • Review test and other output
    • Design metrics
    • Facilitate feedback loops
    • Interface with the customer
    • Solve problems
    • Exchange ideas
    • Be innovative

    The bottom line is that this could be the beginning of a beautiful friendship. We need to reinvigorate the lost art of the dialogue. Actually talk to each other.

    Picture4.png

    This begins with trust. Every aspect of the Continuous People Pipeline is filled with expertise, integrity and people whot want to learn. There is a trust. You have to trust each other’s strengths and take the opportunity to engage on a continuous basis.

    It is only together that we can continuously improve.

    You can watch Jayne’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here.

    Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online Oct. 24.

    — Derek E. Weeks

  • Embedding Ownership: A DevOps Best Practice

    Embedding Ownership: A DevOps Best Practice

    From where I sit in the DevOps community, there is often more focus on dev than on ops. Damon Edwards (@damonedwards) of SimplifyOps sought to change that with his talk, “Ops Happens: DevOps Beyond Deployment,” at the All Day DevOps conference.

    Damon dove right into the primary, systemic force behind most DevOps problems: silos. The product development process goes like this: Planning—>Dev—>Release—>Operate. The problem lies in the tendency of many enterprises to place similar functions together. Everyone ends up in a silo. Then walls build up between the silos. Eventually, people only know life in their silo, making handoffs even harder.

    We often find application knowledge and business intent are heavily emphasized on development side but light on the operations side. Likewise, operational knowledge is heavy on the ops side, but light on the development side. Furthermore, development has ownership but limited accountability, while operations has accountability but no ownership.

    While many enterprises are striving toward building cross-functional teams, the reality is that the transformations often stop short of truly integrating operations. The result?  Silos remain.

    So, one has to ask, why is this so hard?

    The reality is that enterprise operations are under tremendous pressure. One side is telling them to go faster and open it up, and the other side is telling them to be more secure and be more reliable. These are often seen as competing priorities.

    To solve this, enterprises need to “shift left” in the product development cycle operations activity as much as possible. They need to do as much as possible during development. For the deploy function, enterprises should be:

    1. Writing/running automated tests
    2. Writing/exercising deploy automation
    3. Running security scanning tools

    For the operate function, enterprises should be:

    1. Writing/exercising automated runbooks
    2. Writing/exercising monitoring/metrics
    3. Operational control (safely!)

    However, shifting operations to the left is much more difficult. How do you do it? Embed ownership.

    Let me repeat: embed ownership.

    First, those who build something define the procedures to fix it, and those who build something fix it when it breaks.

    That sounds simple, but raise questions:

    1. How do you safely and securely give out access?
    2. How do you enable the experts to contribute remediations?
    3. How do you give the experts visibility into operations?
    4. How do you do postmortems days/weeks/months later?

    Damon recommended four steps.

    Step 1: Establish a secure ops portal


    Step 2: Establish a software development life cycle (SDLC) for ops procedures

    Step 3: Connect with enterprise management systems

    Step 4: Make Compliance really happy

    Ticketmaster is a real-life of example of this working at a large, prominent scale. Ticketmaster calls its system, “Support at the Edge” and it involves:

    • Automated Ops procedures written/vetted by the delivery teams
    • Ops remained in full control of what can run and security policy
    • Empowered support teams with self-service ops tasks
    • Empowered developers with limited self-service operations
    • Combined with new incident response model

    Ticketmaster has seen transformative results. Before Support at the Edge, the average mean time to respond was 47 minutes. Now, support at the Edge has reduced that to just 3.8 minutes in addition to decreasing escalations 50 percent and overall support costs 55 percent. Ticketmaster has seen real results.

    Damon has more details in his talk, which you can watch online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.

    Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free, online on Oct. 24.

    — Derek E. Weeks

  • All Day DevOps: Capital One Gives Credit Where It’s Due

    All Day DevOps: Capital One Gives Credit Where It’s Due

    Listening to his talk, it seems as though George Parris and his team at Capital One aren’t keeping “banker’s hours.” George is a Master Software Engineer, Retail Bank DevOps at Capital One. At the All Day DevOps conference, George gave a talk, titled “Meta Infrastructure as Code: How Capital One Automates our Automation Tools with an Immutable Jenkins,” describing how the team automated the DevOps pipeline for its online account opening project for Capital One, a major bank in the United States. Of course, there is a lot to learn from their experience.

    George started by pointing out that software development has evolved, coming a long way even in just the last few years. Developers now design, build, test and deploy, and they no longer build out physical infrastructure—they live in the cloud. Waterfall development is rapidly being replaced by Agile, infrastructure as code and DevOps practices.

    Where we see these technologies and methodologies implemented, IT operations teams are acting more like developers, designing how we launch our applications. At the same time, development teams are more responsible for uptime, performance and usability. And, Operations and Development work within the same tribe.

    George used the Capital One Online Account Opening project to discuss how they automate their automation tools, now a standard practices within their implementation methodology.

    For starters, George discussed how Capital One deploys code (hint: they aren’t building new data centers). They are primarily on AWS; they use configuration management systems to install and run their applications; and they “TEST, TEST, TEST, at all levels.” Pervasive throughout the system is immutability; that is, once created, the state of an object cannot change. As an example, if you need new server configurations, you create a new server and test it outside of production first.

    They use the continuous integration/continuous delivery model, so anyone working on code can contribute to the repositories that, in turn, initiate testing. Deployments are moved away from the scheduled release pattern. George noted that, because Capital One is a bank, regulations prevent its developers from initiating a production change. They use APIs with the product owners to automatically create tickets, and then product owners accept tickets, making the change in the production code. While this won’t apply to most environments, he brought it up to demonstrate how you can implement continuous delivery within these rules.

    Within all of this is the importance of automation. George outlined their four basic principles of automation and the key aspects of each:

    Principle #1 – Infrastructure as Code. They use AWS for hosting and everything is in a Cloud Formation Template, which is a way to describe your infrastructure using code. AWS now allows you to use CFTs to pass variable between stacks. Using code, every change can be tested first, and they can easily spin-up environments.

    Principle #2 – Configuration as Code. This is also known as configuration management systems (they use Chef and Ansible). There are no central servers, changes are version-controlled, and they use “innersourcing” for changes. For instance, if someone needs a change to a plugin, they can branch, update and create a pull request.

    Principle #3 – Immutability. Not allowing changes to servers once deployed prevents “special snowflakes” and regressions. Any changes are made in code and traverse a testing pipeline and code review before being deployed. This avoids what we all have experienced: the server that someone, who is no longer around, set up and tweaked differently than anything else and didn’t document what was done.

    Principle #4 – Backup and Restore Strategy. A backup is only as good as your restore strategy. You know the rest.

    George also dives into how they do continuous delivery/continuous integration in his talk, which you can watch online here.

    If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free, online Oct. 24.

    — Derek E. Weeks

  • DevOps: Trust, Obligations and Promises

    DevOps: Trust, Obligations and Promises

    Mark Burgess (@markburgess_osl) is a theoretical physicist, but in his keynote at the 2016 All Day DevOps conference, he talked more about economics and human interactives than physics. What does either have to do as the keynote for a conference on DevOps?

    Well, for a little more background, Mark Burgess is also the founder and former CTO of CFEngine, a configuration management and automation framework, and is the author of “Promise Theory.” While at CFEngine, Mark worked to apply a theory of how the autonomous agents in software interact with each other. Promise Theory was born.

    Exploring Promise Theory

    Promise Theory sits counter to obligation theories. Obligation theories view human interactions and behavior under the assumption that people (or agents, in the case of software) choose behavior based on their obligation to follow the rules. Agents are obligated to take a certain action. Promise theory contends the desire to follow the rules is voluntary and that agents can only be responsible for their own behavior, not the behavior of other agents.

    In his presentation, Mark started out looking at human behavior and trends in our society. For instance, with technological advances, our need for one another is decreasing, even as our interdependence through communication is increasing. For example, our smartphones enable us to do almost anything without the direct interaction of someone else.

    Looking at interactions in society, trust is the basis for our exchanges, working communities, and function as a society. He contends, “The promise is the source of intent by which all promises and observers may calibrate their expectations.” Money is an example of this. Money does not have intrinsic value, but we use it because we trust the backer will stand behind its value and it is more convenient than bartering goods. He explains that in the old monetary economy, banks and governments were the provider of trust; in the new service/IT economy, the service provider is the basis of trust.

    How Agents Make Promises

    How do you model this trust? Enter Promise Theory, which describes how agents make promises with each other and how this leads to trust between them. Mark outlines the “ingredients” of Promise Theory:

    1. Agents and super agents make promises in a scaling hierarchy.
    2. An agent can never make a promise about another agent’s behavior (only its own).
    3. An agent’s promise need not be accepted or used by its intended or unintended recipient!
    4. An agent makes its own valuations: what you have is only worth what another is willing to give you for it.
    5. Dependency invalidates promises.

    At the core of understanding this, you have to understand that collaboration is made up of semantics, what words mean and dynamics: all how things work.  And that our ever-increasing service economy is driving us to an explosion of semantics, because words matter so much more than the mechanics. As an example, Mark sketched the hierarchy of promise dependencies in the old and new economies. Looking at the old economy, where we have more dynamic money, the hierarchy is:

    Retail→ Money→ Banks→ Central Banks→ Government.

    In the new economy, where we have more semantic services, the hierarchy is:

    Client→ Network Availability→ Provider→ Internet Authorities→ Company/Service Organization.

    The Need for Cooperation

    None of these systems could work without cooperation at the bottom of it all. So, production is a cognitive process between the producer and the consumer, but also between the agents inside those organizations that have to interact. Often we create pipelines to describe these interactions, but the interdependencies within the systems are often much more complex than the pipeline can model.

     

    This is all just the tip of the iceberg. Mark has three books that dive into it in detail: “Thinking in Promises,” “In Search of Certainty,” and, “Promise Theory.”

    In the end, Mark made the point that technology does an excellent job of both binding us together and pulling us apart. We need to be careful that we don’t take ourselves out of the systems we create, or we could scale ourselves out of existence.

    devopsy.kinsta.cloud Supports the All Day DevOps Conference

    If this piqued your interest, you can watch Mark’s full All Day DevOps conference session (just 40 minutes).  When you do, you can view the other 56 presentations from the 2016 conference online and free of charge.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s free and online on Oct. 24.  devopsy.kinsta.cloud is a media sponsor for All Day DevOps 2017.

    — Derek E. Weeks

  • From a Commodore 64 to DevSecOps

    From a Commodore 64 to DevSecOps

    We all know the story: a farm, a kid, a Commodore 64, and a modem maxing out at 300bps. A few unexpected phone bills later, and young Ian Allison is figuring out how to game the system so he can keep using his newfound gateway to the world of tech. According to Ian, that is where he began building the foundation of skills for his career in computer security.

    At the recent All Day DevOps conference, Ian (@iallison), now with Intuit, talked about his history of being “that” security guy. You know, the one who thinks developers don’t care about security or deadlines and, really, are just plain “stupid.” But, don’t worry, he is enlightened now and realizes that we all have the same goal: Everyone wants to build a secure system.

    Ian realized that “security doesn’t understand how developers or operations works. Security solves for security, but that leaves everyone else in their own place.” He started his enlightenment when his career path led him to a place called DevSecOps; that is, DevOps where security plays a more integral role.

    Ian pointed out that traditional InfoSec relies on compliance, regulations, appliances and perimeter (CRAP). He then realized the selfishness of his own and his peer’s perspective: Remediation is left up to the developers, the feedback they get are 200-page scanner reports, and it only solves problems for security and compliance. It doesn’t help developers reach their shared goal of a secure system.

    DevOps creates an opportunity for security to get a better view into our infrastructure, operations and development efforts.  DevOps is not only fast, lean and efficient, but when done right, it is also collaborative and empathetic. Couple speed with collaboration and empathy, and DevSecOps can blossom.

    Here is the reality that Ian was facing: Scanners find the absolute bare minimum; bad default configs are a HUGE problem even with SaaS vendors; manual testing can uncover defects that have been hiding for year; and the attackers are more skilled and motivated.

    How do you implement it and make it better?

    • Allow Dev teams to assume the risk of their decisions.
    • No more Security exceptions or signoffs.
    • Security is everyone’s responsibility.
    • Test the crap out of your own stuff like an attacker would.

    At Intuit, Ian wanted to help build stronger bridges between development, operations and security teams.  To do this, he set up a Red Team to:

    • Use same tactics as attackers.
    • Only scope is “Don’t take down production.”
    • Need to adapt and evolve like an attacker.
    • Prove risks actually exist.
    • Should be writing their own exploits.
    • Should have ongoing campaigns that mimic attackers.

    The Red Team started small and lean and focused on the cloud. It worked like an Agile DevOps team, working manually with the use of some tools. In the end, the team found, reported and fixed thousands of vulnerabilities not found by scanners.

    Ian goes into more details and lessons learned in his full All Day DevOps conference session (just 30 minutes). The other 56 presentations from the All Day DevOps Conference are also available online, free-of-charge here.

    This blog series is reviewing sessions from the All Day DevOps conference from November which hosted more than 13,500 registered attendees. Last week I discussed, “System Hardening with Ansible.” Next week, look for my review of an awesome session by Erlend Oftedal:  “There is No Server: Immutable Infrastructure and Serverless Architecture.”

    — Derek E. Weeks

  • All Day DevOps: System Hardening with Ansible

    All Day DevOps: System Hardening with Ansible

    The DevOps pipeline is constantly changing.  Therefore, relevant security controls must be applied contextually.

    We want to be secure, but I think all of us would rather spend our time developing and deploying software. Keeping up with server updates and all of the other security tasks is like cleaning your home: you know it has to be done, but you really just want to enjoy your clean home. The good news is you can hire a “service” to keep your application security up to date, giving you more time to develop.

    At the recent All Day DevOps conference, Akash Mahajan (@makash), a founder/director at Appsecco, discussed hardening your system’s security with Ansible. In addition to his role at Appsecco, Akash is involved as a local leader with the Open Web Application Security Project (OWASP).

    Misconfiguration

    During his presentation, Akash mentioned the OWASP Top 10 Security Vulnerabilities list, zeroing in on No. 5: Security Misconfiguration. To determine if you comply with the guidelines, No. 5 on the list asks:

    • Is any of your software out of date?
    • Are there any unnecessary features enabled/installed including ports, services, accounts, pages or privileges?
    • Are default accounts and their passwords enabled/unchanged?
    • Are security settings and libraries not set to secure values?

    I am sure no one reading this article still uses the default administrator password, but can we say the same of your peers? Have you gotten around to installing the latest software patches on your server?

     

    Automation

    If a task can be automated, developers automate it. So we should automate our security tasks too, where we can. OWASP provides guidance here, suggesting you should:

    • Have a repeatable security hardening process.
    • Ensure your development, QA and production servers are configured identically but with different passwords.
    • Automate the process to minimize the effort required to setup a new secure environment.
    • Implement a process for deploying all new software updates and patches in a timely manner to each deployed environment.
    • Run scans and audits periodically to help detect future misconfigurations or missing patches.

    This is all part of security hardening, which is, “the process where we identify default configuration present on a system and apply changes that will change the configuration to secure values.” This can be applied to your network, transport, application, and kernel networking parameters.

    Ansible Playbooks

    Ansible is one of the solutions Akash likes to work with, but there are others solutions on the market that provide similar value. Without trying to endorse or evaluate one solution over another, let me share perspectives from Akash’s experience with his toolset.

    Why does he like it? It boils down to playbooks. An Ansible playbook is a codified security document, allowing you to describe the desired state of a system, rather than the specific steps of how to get to that state. As Akash points out, things change—it is better to have the end state described rather than have to change commands when the system changes.

    Other advantages of playbooks include:

    • Playbooks are written in YAML providing us with structure that we can learn and train on
    • Playbooks are text files, so we can use Git for version control
    • Managing playbooks is just like managing any software project
    • Playbooks are infrastructure as code but for security
    • Playbooks consist of roles, a key aspect of security
    • Numerous playbooks are available as open source

    The bottom line is you can—and you should—automate your security hardening process. Your users and other stakeholders will thank you and, most of all, you will thank yourself because you can spend more time on the things you love to do.

    Ansible is just one example of a solution that can be used to automate your security tasks. If you want to know more, Akash goes into further detail on getting started with Ansible in his full All Day DevOps conference session (just 30 minutes). The other 56 presentations from the All Day DevOps Conference are also available online, free of charge here.

    This blog series is reviewing sessions from the All Day DevOps conference from November which hosted over 13,500 registered attendees. Last week I discussed, “DevOps at Massive Scale.” Next week, look for “Operationalizing a Red Team for Fun and Profit,” delivered by Intuit’s own Ian Allison.

    — Derek E. Weeks

  • All Day DevOps: Exploring the New Ordinary

    All Day DevOps: Exploring the New Ordinary

    There you were. You heard about Docker, become mesmerized, floated into its new, special world … and then all chaos broke out. Things were not that easy and you scrambled for knowledge. Some folks saw the light and stepped forward, while others are still trying to figure the out the happy path home where containers will become part of their new ordinary.

    This is the true story of Daniël van Gils (@foldingbeauty), a developer at Cloud 66, and the 12 steps he laid out in his Docker journey. He shared this journey during his presentation at the recent All Day DevOps conference, and I’ll share a summary of his steps here. His journey will take you full-circle, from your “ordinary IT world” to the “very special container world” and back, to where the container world is the new ordinary.

    Containerization continues to be a hot topic in the software world, and one that is only going to grow and mature. As it matures, questions still abound:

    • How can you implement containerization into your organization?
    • Have you already started but run into roadblocks?
    • How can you be the hero of your organization’s journey?

    Daniël addressed this journey in his talk, “Docker Inside/Out,” at the recent All Day DevOps Conference.

    Daniël’s journey started in the Ordinary World (#1), the world you might be in now. You are at a conference, you read a book, you talk to some colleagues and you want to jump into containers. You receive the Call to Adventure (#2) … but you are human, you like the safe, you fear the unknown, so you want to Refuse the Call (#3).

    Thankfully, others—including Daniël—have taken the journey. They can take you under their wings, coddle you and then push you out of the nest. You Meet a Mentor (#4) and follow your heart. The heart is the good stuff—it is the core. Like most people probably would have, Daniël chose an artichoke to illustrate his point.

    The heart of the artichoke is the good stuff, but the outer leaves protect the heart and make up the whole artichoke. Each aspect of an organization has something that drives them—their heart:

    • Development – building elegantly performing code
    • Operations – stable, performing code
    • Business – delivering the right services to customers
    • Customers – consuming the service they want

    Your drive to satisfy the heart’s desire gives you the courage to Cross the Threshold (#5). You have a containerization machine, such as Docker. You put code in, creating nice little containers.

     

    Now that your infrastructure is set up, you test and know where your allies and enemies are (#6). The most important lesson here is that shortcuts are your enemy. Shortcuts are evil, promising a life of ease but delivering a life of pain.

    Daniël offers his Approach (#7) through the Docker journey to ensure you’re successful. You must KISSSSS:

    Keep It…

    • Slim – remove what you don’t need
    • Secure – ensure you have the latest updates; remove all secrets
    • Speedy – Follow best deployment practices and run performance tests
    • Stable – Use version numbers for your Docker files
    • Set – Immutable. Don’t put databases or complicated volumes into your containers. You can, but the the technology is still too new.

    The next steps in his journey cover Ordeals, Death and Rebirth (#8). Once you pass this point in the journey, you’ll understand what it takes to run Docker in production and have shown the proof of concepts. You’ll then reap the Rewards (#9) and have clear visibility to your Road Back (#10) to the new ordinary.

    For the successful heroes on this journey, now is your moment to shine.  The next steps: your Resurrection (#11) and to Return with Elixir (#12) where containers are now the new ordinary.

    If you want to learn more about Daniël’s Docker journey, be sure to watch his full All Day DevOps conference session (only 30 minutes). The other 56 presentations from the All Day DevOps Conference are available online, free-of-charge here.

    This blog series is reviewing sessions from the All Day DevOps conference from November, which hosted more than 13,500 registered attendees. Last week I discussed, “One Team, 5,000 Jobs: Life in the DevOps Jungle.” Next week, look for “Launching Products at Massive Scale: The DevOps Way” presented by Kishore Jalleda and Gopal Moore.

    — Derek E. Weeks

  • One Team, 5,000 Jobs: Life in the DevOps Jungle

    One Team, 5,000 Jobs: Life in the DevOps Jungle

    Damien has 5,000 jobs. While you might gasp at that workload, Damien is not stressing out. All 5,000 jobs are automated within his team’s Jenkins pipelines. How does he do it? Damien follows four key principles to keep his cool in the job jungle: self-service, security, simplicity and extensibility. But you might be surprised that one of his most important survival techniques is treating his pipeline as “not code.”

    At the recent All Day DevOps Conference, Damien Coraboeuf (@DamienCoraboeuf, nemerosa.com) discussed how his organization got out of the job jungle. Even with 5,000 jobs in Jenkins, Damien has a tiny team of six running it while supporting a large band of developers around the world.

    During his session, Damien laid out how a pipeline, the backbone of a Jenkins job, is constructed. Spoiler alert: It is pretty simple; hence, the magic of Jenkins. However, it gets complicated quickly, so Damien shared four principles his team strives toward:

    • Self-service. They don’t want the Jenkins team to be a bottleneck, and they don’t want to have to deploy an army just to run Jenkins.
    • Security. With teams of developers—many of whom they have never met—security and control is critical.
    • Simplicity. Developers can focus on developing rather than Jenkins.
    • Extensibility. As they grow and capabilities are dreamed up or implemented, their system needs to be able to adapt.

    Core to their implementation and their goals of self-service, simplicity and extensibility is the Job DSL Plugin. While Jenkins has a robust user experience, as jobs are added, it can get cumbersome and clumsy. The Job DSL Plugin brings order, automation and consistency to the jobs. Damien walks through the basics of the plugin and its benefits, noting that it treats the pipeline as code.

    Treating the pipeline as code allows the team to reuse code through versioned libraries and easily manage thousands of jobs. Requiring developers to manage more work as code does offer benefits, but giving that many people access to the code also produces security risks. So, Damien’s team actually treats the pipeline as “not code” by describing the pipeline using properties files. It also allows them to use the pipeline as metadata, so queries and reports are almost enjoyable.

     

    Damien’s team also utilizes the Seed plugin, which they maintain, to “help automate the generation and management of pipelines,” tighten security by keeping projects locked to teams and increase automation through hooks.

    The Seed plugin shares some functionality with the popular Pipeline plugin, which is maintained by a large community. However, the Pipeline plugin does not allow for treating the pipeline as “not code.” Damien’s vision is that one day Seed will be an extension of Pipeline.

     

    Damien’s team has done a tremendous job of running Jenkins across the enterprise. In fact, he just wrote me with an update that since November, his company has scaled to 7,000 jobs while still following the same principles.  If you want to learn more about how his team does it, be sure to watch his full All Day DevOps conference session (only 30 minutes). The other 56 presentations from the All Day DevOps Conference are available online, free of charge, here.

    — Derek E. Weeks