Author: Derek E. Weeks

  • DevSecOps: IoT Right and Nobody Dies

    DevSecOps: IoT Right and Nobody Dies

    Dr. Suzanne Schwartz, CDRH Associate Director for Science and Strategic Partnerships at the Food and Drug Administration (FDA), recently released a blog to update us on the FDA’s role in medical device cybersecurity.

    Cybersecurity risks in medical devices are nothing new. As far back as 2012, Sonatype published warnings of security risks in pacemakers that could lead to lethal attacks. Last year, Johnson & Johnson warned of cyber vulnerabilities in its insulin pumps. More recently, our “2017 State of the Software Supply Chain Report” shared details of pacemaker programming machines that were discovered to have more than 8,000 known software vulnerabilities (see page 37).

    When it comes to software in medical and other devices, cybersecurity will be a constant threat requiring constant vigilance. The encouraging guidance I read from the FDA blog was that cybersecurity should not be limited to devices already on the market, but that security needs build in throughout the product life cycle. Schwartz remarked:

    “It is the goal of FDA’s Center for Devices and Radiological Health to encourage a coordinated approach of vigilance, responsiveness, resilience, and recovery that fits our culture of continuous quality improvement.

    “This means taking a total product lifecycle approach, starting at the product design phase when we build in security to help foil potential risks, followed by having a plan in place for managing any risks that might emerge, and planning for how to reduce the likelihood of future risks.”

    In these two brief statements, Schwartz shares views that are common in many of today’s DevSecOps conversations. Perhaps she has been listening in?

    The first view is that we need to “emphasize the performance of the entire system and never pass a defect downstream” (Gene Kim’s first way of DevOps). Schwartz recommends that device manufacturers start by building security in at the earliest stages of the development life cycle. Security cannot be an afterthought.

    Schwartz also recognizes that even when precautions are taken early in the life cycle, risks can emerge over time as new vulnerabilities are discovered. When it comes to device security, vigilance must be continuous across the development life cycle and over the product’s life in the market. As we have said for years, software ages like milk, not wine. Constant vigilance improves our ability to identify risks, initiate feedback to development teams and remediate issues in the device’s software. The faster we can address risks when they appear, the safer we can make the lives of consumers who rely on them.

    I applaud the FDA’s guidance and its proactive stance here to work with device manufactures. What we don’t want is a knee-jerk reaction to medical device security after someone has died. What we need is for security to be ingrained in the planning, design, production and maintenance of medical devices our families rely upon.

    You can read the full blog from Schwartz on the FDA Voice.

    — Derek E. Weeks

  • Learn DevOps from 57 Experts for Free

    Learn DevOps from 57 Experts for Free

    Need to brush up your knowledge of continuous integration/continuous delivery (CI/CD), automated security, modern infrastructure or cultural transformations?

    All Day DevOps just released its Summer School program offering 57 online sessions delivered by DevOps practitioners.  Every session is free to watch, and absolutely no registration is required.

    Image title

    Watch as many or as few sessions as you would like at: https://www.alldaydevopsy.kinsta.cloud/addo-on-demand

    Courses are delivered by these awesome DevOps practioners who will share their real-world experiences and lessons learned with you.

    Image title

    — Derek E. Weeks

  • Scaling DevOps at Pearson

    Scaling DevOps at Pearson

    Pearson might be one of the more influential companies you have never heard of. Its footprint in the educational publishing marketplace is expansive, and its scope was just one challenge when it made the move to DevOps.

    Sean D. Mack (@SeanDMackNYC) is formerly the VP of Operations and Application for Pearson. To give you an idea of the scale of the company’s operations when it began implementing DevOps, Sean oversaw 110 development teams and 3,000 different products across the globe. To further complicate things, Pearson is a mature company with both legacy and new products. This is a case study of how the company approached implementing DevOps with the goal of helping you learn how to do it. Sean presented it at the 2016 All Day DevOps conference.

    As they say, you often learn your biggest lessons with your failures. Sean noted that, ultimately, DevOps was a failure at Pearson due to organizational change. But, he presented the case study with the hope you will learn from what Pearson did and take it forward. He continues to be a passionate evangelist for DevOps.

    Pearson had its challenges. It was a siloed organization operating at a large scale, and had a toss-it-over-the-wall mentality in dealing with problems. Out of sight, out of mind.

    seanmack1.png

    In addition, all of Pearson’s teams were at different levels of maturity, from legacy teams running .Net on Windows 2000 servers while others doing CI/CD to AWS. Obviously, this creates challenges when implementing DevOps, but we will see later how the company tried to tackle that, specifically.

    At the high level, the solution involved several aspects:

    • A DevOps practice area
    • A DevOps steering committee
    • Measuring progress based on global KPIs focused on measuring business impact
    • Measuring teams based on a DevOps scorecard
    • Push from the bottom – plenty of developers and operations who embraced and were eager for the change
    • Push from the top – support from key executives

    Pearson’s DevOps Steering Committee set up several workstreams:

    • Tracking tools
    • Metrics
    • Deployments automation and monitoring practices
    • Test automation
    • App engineering resource contention
    • Operationalization
    • Communications

    While these are specific to Pearson, many may be applicable to your organization. In any case, it is important to step back, look at your organization, and decide what your organization needs.

    Pearson also looked at four core competencies to track and measure: automation; independence (how independent is the feature); operationalization (how easy it is to maintain); and continuous improvement. The below chart outlines each area in more detail.

    Automation

    Continuous Improvement

    Independence

    Operationalization

    • Unit tests
    • Critical Scenarios
    • Dependent Systems
    • Continuous Build
    • Continuous Deployment
    • Test Automation
    • KPI
    • SLA
    • Metric Tracking
    • Metric Reporting
    • Remediation
    • Isolated Configuration
    • Code Dependencies
    • Isolated Infrastructure
    • Monitoring and Alerting
    • Tier 1 Service Desk System Knowledge
    • Tier 2 Service Desk System Knowledge
    • Documentation

    To help each team measure their competencies and improvement, Pearson built a self-reporting tool to empower teams but encourage visibility, and developed business metrics and then a competency model to measure teams against to see how they were progressing. As Sean noted, “We were not trying to get everyone to a 5. Some teams, yes, but we also had legacy products that were running for years, successfully, with little or no maintenance. The investment to bring the teams up wasn’t worth the effort. There had to be a business reason.“

    seanmack2.png

    In the end, Pearson raised DevOps maturity for all of the teams. As noted earlier, though, this was eventually phased out, but Sean presented this case study with the hope you found something to drive DevOps in your company.

    In the end, if you have just a couple takeaways, they are this:

    1. Begin by determining what your desirable outcomes are.
    2. Don’t just do DevOps to be cool. Do it to drive business results.

    You can watch Sean’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online on Oct. 24.

    — Derek E. Weeks

  • Characterizing and Contrasting Container Orchestrators

    Characterizing and Contrasting Container Orchestrators

    Admiral Calcote, also known as Lee Calcote (@lcalcote) or the Ginger Geek to his friends, gave a presentation titled, “Characterizing and Contrasting Container Orchestrators,” at the 2016 All Day DevOps conference.

    Okay, he isn’t really an admiral—nor does anyone call him that—but he used the title “admiral” to describe what container orchestrators do, relating it to an admiral directing a fleet of container ships. You could also say that they are like the conductor of an orchestra, directing the individuals to work together as a group toward a common goal while each musician is still able to play their own instrument.

    lee1.png

    Lee is the head of Technology Strategy at SolarWinds, and for his talk, he discussed four open-source container orchestrators: Nomad, Swarm, Kubernetes and Mesos-Marathon.

    He emphasized the obvious: There is no one perfect solution. Each organization is different, so for each solution, he looked at:

    • Genesis and purpose
    • Support and momentum
    • Host and service discovery
    • Scheduling
    • Modularity and extensibility
    • Updates and maintenance
    • Health monitoring
    • Networking and load balancing
    • Secrets management
    • High availability and scale

    Lee noted that while there are many core capabilities, any orchestrator must have cluster management and scheduling.

    lee2.png

    He then dove deeper into the four solutions. Below are summaries (the full talk is chock full of information and is online here):

    Nomad

    • Designed for both long-lived and short-lived batch processing workloads
    • Cluster manager with declarative job specifications
    • Ensures constraints are satisfied and resource utilization is optimized by efficient task packing
    • Supports all major OSs and workloads
    • Written in Go and with a Unix philosophy
    • Host discovery: Gossip protocol – Serf is used; servers advertise full set of Nomad servers to clients; creating federated clusters is simple
    • Service discovery: Integrates with Consul
    • Scheduling: two distinct phases – feasibility checking and ranking; optimistically concurrent; three scheduler types when creating jobs
    • Uses task drivers to execute a task and provide resource isolation, but it does not support pluggable task drivers
    • Built for managing multiple clusters/cluster federation

    lee3.png

    Docker Swarm 1.12

    • Simple and easy to set up
    • Architecture is not as complex as Kubernetes and Mesos
    • Written in Go – lightweight, modular and extensible
    • Strong community support
    • Host discovery: used in the formation of clusters by the Manager to discover Nodes (hosts); pull model – worker checks in with the Manager
    • Service discovery: Embedded DNS and round-robin load balancing
    • Scheduler is pluggable and is a combination of strategies and filters/constraints
    • Ability to remove “batteries”
    • Rolling updates are supported
    • Managers may be deployed in a highly available configuration, but does not support multiple failure isolation regions or federation

    lee4.png

    Kubernetes

    • An opinionated framework for building distributed systems
    • Written in Go and is lightweight, modular and extensible
    • Led by Google, Red Hat and others
    • Young – about 2 years old
    • Robust documentation and community
    • Scheduling is handled by kube-scheduler
    • Pluggable architecture and an extensible platform
    • Choice of database for service discovery or network driver and container runtime
    • Supports rolling back deployments, automating deployments and rolling updating applications
    • Inherent load balancing
    • Uses Pods, an atomic unit of scheduling. Each pod has its own IP address, no NAT required and intra-pod communication via localhost

    lee5.png

    Mesos-Marathon

    • Mesos is a distributed systems kernel
    • Mesos has been around the longest (since 2009)
    • Mesos is written in C++
    • Marathon is a framework that runs on top of Mesos
    • Mesos is used by Twitter, AirBnB, eBay, Apple, Cisco, and Yodle
    • Marathon is used by Verizon and Samsung
    • Mesos-DNS generates an SRV record for each Mesos task
    • Marathon ensures that all dynamically assigned ports are unique

    lee6.png

    Wrapping up, Lee provided the following overview comparing the different container orchestration solutions.

    lee7.png

    Lee packed a tremendous amount of information into his talk. If you use containers, his talk is worth your time and is available online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online Oct. 24.

    — Derek E. Weeks

  • Speaking for the Dead: Are ‘Agile’ and ‘Monolithic’ Actually Good?

    Speaking for the Dead: Are ‘Agile’ and ‘Monolithic’ Actually Good?

    What if Agile is the product of getting bored at work, microservices are the result of us needing a challenge and scrums are the result of an obsessive project manager? Do we all think waterfall development is a dinosaur whose extinction is deserved because of what we have been told and not because of reality? Do we think Agile is better because it simply sounds desirable?

    Michael DeHann (@laserllama), creator of Ansible, spoke at the 2016 All Day DevOps conference, which espouses the benefits of DevOps, and, by association, Agile and microservices. It essentially says, “Let’s step back and evaluate the past. Maybe waterfall development and monolithic architectures are actually a good thing?”

    Michael has been a software engineer and created a key tool in the IT automation toolbox: Ansible. His arguments deserve a look, because re-evaluation can keep you from making mistakes and improve processes going forward.

    Michael started his talk by making the point that in this day and age, it is waterfall versus Agile and monolithic architecture versus microservices. The assumption is that if you are still developing software with the waterfall method and/or developing monolithic applications, you might as well be riding a horse and buggy to work and dialing up coworkers on your rotary phone.

    Throughout his talk, he asked questions to make his points, beginning with, “Is big design up front bad?” with the conclusion, “I don’t think so.” His argument is that design and requirements at the beginning of the process enable a better production. Marketing and engineers hammered out requirements based on needs, abilities and budgets. There was plenty of time to research and design solutions. Both sides worked together, raising questions and comments and documenting mutual agreement. This presented an accurate view of the strengths of various team members and dependencies between tasks and task orders. And, perhaps most important, engineering was engineering—the engineers had greater responsibility and less hand-holding.

    Contrasting that with his view of Agile, big design up front is labeled as bad or counterproductive, engineers are viewed as interchangeable and scope is more unpredictable. Estimates are encouraged to not be time-based, which often results in a constant crunch time. Finally, meetings can consume a significant portion of team members’ time and often include “games reminiscent of Kindergarten, such as Fist of Five or Planning Poker.” Even in lightweight Agile systems, testing can suffer, requirements are more vague and there is no mutual sign-off.

    Michael argues that there can, in fact, be a middle ground where teams:

    • Do share design
    • Have an accessible, effective product owner
    • Know hard requirements versus soft requirements
    • See test-driven development and test automation as super important
    • Understand that Agile and iterative development can be closer than we think

    Agile and DevOps are also often associated with microservices versus monolithic applications, but according to Michael, “What’s so bad about a monolith?”

    Sample Monolithic Diagram

    dehann1.png

    Sample Microservices Diagram

    dehann2.png

    An example microservices shop has 150 services, many thousands of Jenkins builds per day, eight Amazon regions including dev, staging and production and thousands of EC2 instances. This can result in Jenkins configuration and security enforcement drifts, suboptimal code sharing and DevOps writing lots of custom code to manage it.

    Michael contrasts that with monoliths, which have:

    • Increased ability to develop on one laptop
    • Interfaces that benefit from static typing versus design-by-contract
    • Very small numbers of different server/image types in production
    • Local function calls that are much faster than RPC
    • Few instances sitting around with 5 percent utilization
    • No need for “containers” to get around above utilization problem
    • Greatly simplified debugging
    • Easy code-sharing
    • Very simple version upgrades

    Rather than argue that all applications should be monolithic, he makes the point that many of the concerns that resulted in microservices could be solved with improved communications. He contends that microservices are a tech solution to a people problem.

    Michael concludes that:

    • Labels cloud true meanings,
    • Monolithic architectures are more efficient,
    • Agile can be less efficient,
    • Microservices attempt to solve people problems but introduce new technical and people problems, and
    • We should try and separate the labels from the true meaning of things.

    Most importantly, though, is the fact that there is something to learn from everything. So, even if you don’t agree with Michael’s conclusion, you should listen to his talk online here. You might learn an unexpected lesson.

    If you missed any of the other 30-minute long presentations from 2016 All Day DevOps, they are easy to find and available free of charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online Oct. 24.

    — Derek E. Weeks

  • Liberty Mutual: Taking an Organic Approach to DevOps

    Liberty Mutual: Taking an Organic Approach to DevOps

    Some people are directors, managers, and administrators. Others are disrupters.

    Eddie Webb (@edwardawebb) is an IT disrupter for Software Development Platforms at Liberty Mutual and was a presenter at the 2016 All Day DevOps conference. His talk, “Organically DevOps: Building Quality and Security into the Software Supply Chain at Liberty Mutual,” looked at Liberty Mutual’s transformation to continuous integration, continuous delivery and DevOps. For a large, heavily regulated industry, this task can be not only daunting, but also viewed by many as impossible.

    Eddie’s talk looks at the origins of DevOps at Liberty Mutual, the challenges of centralizing DevOps, and how to make it easy for developers and operations to do the right thing. He noted that DevOps is not Liberty Mutual’s goal; continuous delivery is the goal and DevOps is the enabler.

    It all started at Liberty Mutual in 2004 with a vision: “Black-box reusable services that benefit the business.” The company also had a concept that “developers own the quality of the code; we have no QA team.” Employees were empowered with this “concrete vision but malleable strategy,” as Eddie said. The company gave them time and space to explore solutions, because ownership increases adoption.

    The team started with fixing what frustrated them, including CI/CD pipelines, artifact and code repositories, elastic cloud-native runtime and Agile planning and delivery.

    But what about centralizing DevOps?

    The reality is that tools can be centralized, but culture cannot.

    Eddie laid out an example of a large organization’s IT structure, saying that this is how things work at Liberty Mutual. Eddie’s team is the Central IT, and their customers are the Market IT team, not Liberty Mutual’s end customers. As Eddie pointed out, the more layers you create, the more friction that enters the system. Often, organizations try to reduce the friction through microfixes, but Eddie’s team asked how to change the culture to reduce the friction.

    webb1.png

    This required distributing innovation and centralizing the tools that work. As an example, if you build a manufacturing plant, you wouldn’t build your own power plant. If you are in IT, why build your own cloud?

    webb2.png

    This is the Netflix way. At Netflix, teams are independent, but responsible. Engineers can choose the best tools for the job. No one single team is responsible for innovation. However, the Engineering Tools team helps direct other teams’ experimentation toward new products without getting in the way of innovation.

    Eddie quoted the book, “Drive,” when looking at how to motivate innovation in complex work environments. The author says if you want to incentivize or motivate cognitive tasks, you need three things: autonomy, mastery and purpose.

    When you centralize non-value added platforms, it enables teams to:

    • Master their specific domains of business;
    • See clear purpose in everything they build; and
    • Maintain autonomy of choice of frameworks, patterns, and approach.

    When you decentralize, it raises the question: What about security? Developers are humans, and humans inherently try to avoid tasks they see as unnecessary or overly burdensome, and this becomes easier when the compliance person isn’t there (think of how we handle speed limits). Traditional security measures erode profit and market share, and, from a stat Eddie quoted from Gartner, 77 percent of IT security professionals agree that security slows down IT.

    webb3.png

    Whether you are centralized or decentralized, security (and other compliance-oriented tasks) can increase adoption rates by making it easy to do the right thing. How do you do this? Eddie proposes governing with tools (think of speed governors in fleet cars).

    You also need to have immutable environments, because, “We can’t trust humans to manipulate production.” So, again, you have to make the path you want them to take, and make this the easiest path. As Eddie notes, “You can’t put too much friction into the pipeline, because developers are good at bypassing friction.” You have to make your CI/CD pipelines the easiest way to get ideas to production by. You also have to abstract complexity and automate policies. You have to have an “opinionated platform, and observe your customer so you can address their pain.”

    webb4.png

    Eddie concluded with the following final points:

    • Don’t mandate DevOps. Give employees the chance to master their discipline with examples to set and follow.
    • Favor deep end-to-end accomplishments over broad but incremental steps forward. Focus on taking the right teams far before encouraging broad adoption.
    • Centralize the platforms and tools that your teams shouldn’t be thinking about. Provide foundational services/commodities and let teams stay on purpose.
    • Incorporate contributions from everyone; don’t stifle autonomy. Stay open to new ways of working.
    • Challenge security policies, but respect intentions. Find new ways to enforce concerns without abandoning precaution.

    You can listen to Eddie’s entire talk online here. You might learn an unexpected lesson.

    If you missed any of the other 30-minute long presentations from 2016 All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 100 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

    — Derek E. Weeks

  • DevOps and Security is Like Smoking Meat

    DevOps and Security is Like Smoking Meat

    It isn’t everyone who thinks, “Doesn’t Ubuntu remind you of wild boar?” Or labors over his pit of slow-roasted pork shoulder while contemplating containers. Or dwells on e2e testing while mesquite smoke permeates spareribs.

    But, Apollo Clark (@apolloclark) does. Apollo is a foodie of smoked meats (is that a meatie or smokie?), working to master the craft: understanding different cuts, what each type of wood adds, and the subtleties of sauces. But, alas, it is a hobby; security and DevOps are his career.

    Being passionate about both, he naturally sees parallels between the two, so Apollo presented at the 2016 All Day DevOps conference with a session titled, “What Smoking Meat Taught Me About DevOps and Security.”

    Understandably, you are asking yourself, “What does smoking meat have to do with DevOps and security?” Apollo notes both have tremendous complexity and nuances, there are multiple ways of getting the job done, lots of ways to mess it up, a couple of ways of doing it right, and you are always learning.

    apollo 1.png

    For both, there are many tools and processes to get the job done. For smoking, your wood is a critical component, along with time. Let’s look at some parallels:

    • Oak = unit testing
    • Maple = coverage
    • Apple = dynamic analysis
    • Peach = static analysis
    • Mesquite = e2e testing. People love it, but it is difficult to handle
    • Wine barrel = browser support. This is when you are doing it really well
    • Bourbon barrel = device support. Pretty complicated
    • Smoking time = testing. You can test or smoke for five minutes or five hours, but there is a sweet spot. You can oversmoke meats and you can overtest.

    Getting hungry?

    apollo2.png

    Meat is critical to a meal of smoked meat, but so are many other components. Likewise, software applications take a suite of other tools and components to deliver the full package. Get your taste buds ready, because Apollo likens all of the goodness that goes with smoked meats to tools you use to keep applications running:

    • Sauces = auto-scaling. They both can get pretty complicated. Always a little different, but you have to make it work for you.
    • Bread = monitoring, which is the bread and butter of infrastructure. Make sure you have it up and down your stack.
    • Salad = system logs. Not the sexiest things, but you can rely on them.
    • Fruit = custom application logs. Takes a lot of time to pair them, but takes a really good thing and makes them better.
    • Beer = firewall. You should always have both.
    • Wine = Intrusion detection systems. These are your fine wines.
    • Whiskey = IR training. It takes time and there are so many ways of doing it. When things break, we have procedures on how to deal with them.

    When smoking meats and in DevOps and security, Apollo asks and answers, “Do we have to do everything? No, but the more we do, the better it will taste.

    “Does it cost money and take time? Yes, but you can do great things even without money and time. You can use cheap meats and you can use open source. Start simple, build up complexity, and always be learning.” Constantly ask yourself, “Am I better today than I was yesterday.” If not, be better.

    You can watch Apollo’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s all free and online Oct. 24.

    — Derek E. Weeks

  • Securing Immutable Servers in a Serverless World

    Securing Immutable Servers in a Serverless World

    Snowflakes are beautiful, unique creations. But, let’s keep them in nature. They don’t belong in our server infrastructure. Snowflake servers, where every configuration is just a little different, can introduce unnecessary security vulnerabilities and complications. While common in IT infrastructure, in the DevOps realm they are gradually becoming ancient history.

    At All Day DevOps 2016, Erlend Oftedal (@webtonull), with Blank and head of the OWASP Norway chapter, discussed the benefits of immutable infrastructure practices within serverless architectures. Erlend walked us through the positive effects on security as well as offered insight on potential problems.

    Erlend outlined the progress from physical servers to immutable servers, discussing the benefits of each step and the concerns that lead to the next phase, ending with immutable servers—that is, they can’t be changed. Once set up, they are set. Automated processes coupled with immutability principles can eliminate snowflake servers.

    blog pic 1.png

    Immutability isn’t a new concept. Some classic examples include live Linux CDs, internet cafes, schools, libraries and hotel business centers. To set up immutable servers, Erlend outlines these steps:

    1. Find a suitable base image
    2. Build a specific image for the application
      1. Dependencies
      2. Application
      3. Secrets
    3. Deploy image to the server(s)
    4. Repeat steps 2-3 for new versions

    blog pic 2.png

    Of course, this immediately raises questions:

    • What about data?  He notes that because data inherently changes, data has to be stored externally.
    • How do you deploy changes?  Well, you don’t change the server. You take one out and put a new one in. What if that one fails? Roll back to the previous one.
    • How do you manage secrets?  You can build secrets in, leverage cloud key management, or use third-party services.
    • How do we apply security patches?  If truly immutable, you deploy a new server with the patch. However, you can have a semi-immutable server that allows for automatically installing patches but still doesn’t allow for logins or manual changes.
    • Speaking of security, what are the security advantages? For starters, if an attacker is in your server, it will be thrown out when you change servers, although it may come back if the vulnerability isn’t patched. It also allows for very specialized images by removing all unnecessary packages. With these specialized images, you can more easily audit and monitor the system for unexpected file changes, logins and connections.

    blog pic 3.png

    Erlend discussed containers because they are a step closer to a serverless world. Containers isolate processes and all run on the same OS. Of course, you can make them immutable, but regardless of whether they are, there are some key security best practices you’ll want to consider:

    • Don’t run as root inside container
    • Implement user namespaces so the root account within the container is separated from the root account outside the container
    • Utilize the least privileges necessary for capabilities and resources
    • Keep them up-to-date
    • Automate scanning using tools such as Sonatype Nexus, Twistlock, Clair, or Aqua
    • Separate untrusted and trusted containers

    Erlend also dove into serverless architecture. While there is debate over what exactly qualifies as serverless, there are five principles that Erlend highlighted:

    • Use a compute service to execute code on demand
    • Write single-purpose, stateless functions
    • Design push-based, event-driven pipelines
    • Create thicker, more powerful front ends
    • Embrace third-party services

    Migrating to immutable, serverless infrastructure is a topic to be discussed well beyond this blog post and even Erlend’s session. Whatever you do in the future, though, take his parting words to heart: “The cloud can solve many of your problems, but in the end you cannot transfer your responsibility for security.”

    Erlend’s session was packed with information. If this piqued your interest, you should watch his complete session from All Day DevOps 2016 (just 30 minutes). The other 56 presentations from the conference are also available online, free-of-charge here.

    Also, this year’s All Day DevOps is Oct. 24. It’s online and free, so be sure to check out www.alldaydevopsy.kinsta.cloud to learn more and register.

    — Derek E. Weeks

  • Government Legislation Aims to Secure IoT

    Government Legislation Aims to Secure IoT

    The U.S. Senate is making moves to protect consumer interests, data, and privacy with regard to the internet of things (IoT). Sens. Mark R. Warner (D-VA) and Cory Gardner (R-CO), co-chairs of the Senate Cybersecurity Caucus, along with Sens. Ron Wyden (D-WA) and Steve Daines (R-MT), recently introduced bipartisan legislation called the Internet of Things Cybersecurity Improvement Act of 2017.

     

    Addressing IoT Market Failures

    According to a fact sheet released by the senators, “While ‘Internet of Things’ (IoT) devices and the data they transmit present enormous benefits to consumers, the relative insecurity of many devices presents enormous challenges. This legislation is aimed at addressing the market failure by establishing minimum security requirements for federal procurements of connected devices.”

    The proposed legislation drives home a new reality for IoT device manufacturers: While development organizations currently decide to invest in cybersecurity using a cost-benefit assessment, they are ultimately liable for the security of their data and systems.

    The U.S. Senate’s Proposal

    The proposed legislation requires vendor commitments:

    • To ensure devices don’t contain known security vulnerabilities when shipped.
    • To ensure proper disclosure of new security vulnerabilities discovered within their devices.
    • To prepare remediation plans for any IoT device where known vulnerabilities have been discovered.

    The Internet of Things Cybersecurity Improvement Act of 2017 is tied directly to federal government procurement of such devices but Sonatype predicts they likely will extend into private-sector guidelines moving forward. While the legislation is clearly aimed at consumer protections and privacy, it also focuses on quality, safety and regulatory standards applied to every other major manufacturing industry (“Thou shall not ship products with known defects“). It’s commonsensical in other manufacturing industries, and now Congress is directing development organizations to apply similar quality standards for the software it develops and ships.

    Certify IoT Devices are Free from Known Security Vulnerabilities

    The legislation specifically calls for vendors selling IoT devices “to provide written certification that the device does not contain, at the time of submitting the proposal, any hardware, software, or firmware component with any known security vulnerabilities or defects.”

    Notification When New IoT Vulnerabilities are Discovered

    Following the purchase of any IoT devices, the government wants the seller “providing the Internet-connected device software or firmware component to notify the purchasing agency of any known security vulnerabilities or defects subsequently disclosed to the vendor by a security researcher or of which the vendor otherwise becomes aware for the duration of the contract.”

    Ensure Remediation Paths are Available

    Protections go even further in the post-sale period, calling for clear remediation paths. The proposed legislation would require “any future security vulnerability or defect in any part of the software or firmware [of the IoT device] to be patched in order to fix or re-move a vulnerability or defect in the software or firmware component in a properly authenticated and secure manner.”

    20 Vulnerabilities Per Application

    Recently, Sonatype released its “2017 State of the Software Supply Chain” report that shares empirical evidence of the number of known software vulnerabilities that ship in applications—IoT or otherwise. The report found that development organizations building software with unmanaged software supply chains release applications with an average of 20 known security vulnerabilities, six of which had a Common Vulnerability Scoring System (CVSS) rating between 7 and 10.

    Sonatype’s report also reveals how national governments, federal agencies and industry associations are taking action to help organizations improve open-source hygiene and application security practices. In the past year, new guidelines have been introduced from the White House, Federal Trade Commission, Department of Homeland Security, Department of Health and Human Services and the Department of Commerce to improve the quality, safety and security of software supply chains, related to IoT devices and any other application development efforts.

    The Tide Has Turned

    While the proposed legislation is still open for debate, there is clear evidence that those once ignorant or turning a blind eye to such cybersecurity issues are now paying attention—and they are preparing to make significant changes.

    The full text of the proposed legislation for the Internet of Things Cybersecurity Improvement Act of 2017 can be found here.

    — Derek E. Weeks

  • All Day DevOps: Taking Lambda to the Max

    All Day DevOps: Taking Lambda to the Max

    Lambda is the 11th letter of the Greek alphabet, and it is also the name of the Amazon Web Services (AWS) service that lets you run code without actually configuring a server. Amazon chose the name Lambda because—well, okay, I have no idea. Comment here if you know.

    Thankfully, you don’t need to know that to understand how to utilize Lambda. Matt Williams, DevOps Evangelist at Datadog (@technovangelist), outlined how to set up AWS Lambda at the 2016 All Day DevOps conference in his talk titled, “Hold the Infrastructure: Getting the Most Out of Lambda.”

    First, to understand what Lambda is, we must look at the evolution of servers.

    williams1.png

    We started with physical boxes—we had to buy and maintain these boxes, and no matter how much we used, we had to pay for all of it. We then evolved to multiple virtual machines on a physical server so that we could maximize the resources of physical boxes. Now, AWS and similar cloud services are common. We create a virtual machine that we need, and the service provider maintains the physical servers. We also have containers to keep environments identical across machines and, finally, we arrive at Lambda, which is service that runs code but doesn’t require you to set up the server. It takes care of all of that automatically, and you only pay for the time that the function is actually running. So, if your function takes 500ms to run, you pay for 500ms to run it once.

    Lambda is trigger-based. When it receives a trigger, it executes a specified function. The trigger can be a variety of actions with AWS services, such as when a file shows up on S3 or an Alexa skill is requested.

    Lambda is serverless, as far as you are concerned.

    Lambda is stateless. This means you can’t rely on local states, so, if you need to retain information between function calls, you have to save that data somewhere else.

    Lambda executes functions. Each function:

    • Is code in a container on LXC on AMZN Linux
    • Does not have restrictions on what it can do
    • Is priced based on exec time, quantity and memory

    Lambda links to various AWS services, such as:

    • Files (S3)
    • Database Updates (DynamoDB)
    • Streams (Kinesis)
    • Messages (SNS, SES)
    • IoT (Echo, API Gateway)

    Matt mentioned several good use cases, such as bootstrapping (the first 1M requests per month are free), cron jobs, file manipulation, API translation, social media stream processing, scaling infrastructure, mobile/IoT backends and Alexa skills, to name a few.

    He also mentioned three specific examples to explore:

    Matt then outlined how you can get started and noted that setup is simple as long as you think of application development as simple (write the code, set up on AWS, and debug). He walked through a specific example of automatically resizing images on S3 when they are added. You can watch his full presentation with screenshots here.

    Matt also provided a list of some AWS Lambda Tools:

    Matt rounded out his 30-minute presentation with a discussion regarding the need for monitoring and how to use Lambda for monitoring.

    You can watch Matt’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute presentations from All Day DevOps, they are easy to find and available free-of-charge here.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

    — Derek E. Weeks