Tag: Nexus

200 Billion Downloads Can’t Be Wrong

Laurie Voss, COO and co-founder at npm (@seldo), tweeted recently that JavaScript packages downloaded from their repository has topped 4 billion. On an annual basis, that would be more than 200 billion downloads. Wow.

Just as open source software turns 20 years old this year, these numbers are a testament to the incredible magic that happens when communities of developers openly share innovations.

Laurie estimated that 12 million developers were responsible for the downloads. That means, on an annual basis, each JavaScript developer will download nearly 17,000 components. Imagine the time saved by not needing to code each of those functions from scratch simply by downloading a package in a millisecond from the internet.

Software Supply Chains at Work

What we’re witnessing here is a software supply chain in full effect. Open source projects contribute packaged code to the community, place it in public warehouses, and it’s then consumed by development teams around the globe to create new front-end, back-end or mobile applications for all of us. Every development team on the planet now utilizes a software supply chain that operates at insane speeds with massive throughput.

In a second tweet, Laurie wrote, “Since we have ~12 million users, that means the average user installed the neatly round number of 1000 packages in 7 days. In reality, most individuals installed very few packages, and a bunch of CI build boxes installed 10s or 100s of thousands of packages each.”

Essentially, there is an army of robots (automated CI tools) downloading billions of components from the web. While this type of automation is efficient, by itself it does not serve teams well.

Repositories to the Rescue

At Sonatype, we saw similar behavior by developers in the early days of Maven. Individual developers would download Java components directly from Maven Central. Maven users sitting right next to one another in the same room and on the same team would download the same versions of the same components needed for their builds, repeatedly. It was for this reason, among others, that our founders invented the Nexus repository manager.

While repository managers have been used for quite some time in the Java community, Laurie’s post tells us that adoption of the technology still lags in younger, less mature, packaged code ecosystems.

Companies looking for repository managers can look toward solutions like e-npm, an on-premises technology exclusively for JavaScript packages. For companies with polyglot development teams, universal repository managers provide full-function, (sometimes) free support for npm and 12 other component types, including Docker containers.

Not All Components Are Created Equal

I have often said that if you have 100 developers, you have 100 front doors open into your organization. If you have 1,000 developers? Then there are 1,000 front doors.

If you dissect Laurie’s comment further, you will also realize that any developer can bring any package into your organization at any time. While this does make them more efficient, it also points to security and governance concerns. Every component was developed by someone else, donated out to the internet and consumed in the millions. But you have no true idea of the source or origin of many of the components.

Research I have done in previous years at Sonatype demonstrated that 5.5 percent (1 in 18) of components downloaded by repository managers had known security vulnerabilities. While the percentage is not very large, keep in mind that a repository manager will only download a component once. Once cached, future downloads of that component are unnecessary and the components can be reused infinitely by that team.

Sonatype analysis of more than 40,000 Nexus repositories reveals that the average repository holds more than 1,600 components. Deeper analysis of the 1,600 components housed in the average repository manager found 192 security vulnerabilities were present among the components (some components having more than one security vulnerability).

Within software supply chains, repository managers and private container registries represent procurement gates into the development organization. The gates can be left wide open where component flows are not governed or they can represent opportunities for quality and security checkpoints that ensure defects are not passed downstream.

What’s in Your Repository Manager is in Production

Using data from more than 133,000 websites, researchers from Northeastern University showed that “37% include at least one JavaScript library with a known vulnerability. From a per-library perspective, at least 36.7% of jQuery, 40.1% of Angular, 86.6% of Handlebars, and 87.3% of YUI inclusions use a vulnerable version.

“Alarmingly, many sites continue to rely on npm packages like YUI and SWFObject that are no longer maintained,” the researchers continued. “In fact, the median website in [NU’s] dataset is using a library version 1,177 days older than the newest release, which explains why so many vulnerable libraries tend to linger on the Web.”

What is Wrong With 200 Billion Downloads?

Nothing from an innovation standpoint. It’s awesome and we can all celebrate this achievement with Laurie.

At the same time, it gives us pause to reflect upon other concerns of our day, including cybersecurity threats, technical debt and wasteful context switching.

— Derek E. Weeks

May 1, 2018
DevSecOps: Digging into Root Cause Analysis
We have all been there in a postmortem when someone says, “Let’s get to the root of the problem.” And, we all know what that means: Who or what is to blame? We also all know that no one wants to play the blame game, yet we all do. But it isn’t our fault (no blame, see what I did there?). It has been the default system for solving problems in business for decades. It is called root cause analysis.

We can change—for the better.

I recently watched a presentation from Matthew Boeckman (@matthewboeckman) titled, “There is No Root Cause: Emergent Behavior in Complex System.” Matthew is a developer advocate with VictorOps and a technology strategist with Dryan.io. He grew up a systems guy and jokes that he has been in DevOps for 18 years, even though DevOps wasn’t around, because he has always been nice to developers.

Digging in (pun intended), root cause analysis focuses on what went wrong, and how we can prevent it from happening again.

The core problems with root cause analysis for development is that it doesn’t provide for enough complexity and its natural focus is blame, which can undermine a positive DevOps culture.

Root cause analysis was more applicable when waterfall was the development methodology because states stayed consistent for months or even years at a time. In the age of Agile, DevOps, CI/CD, microservices, etc., states of work are in a constant flux. Root cause analysis can’t provide solutions quickly enough. As Matthew notes, in root cause analysis, things are either good or bad, working or broken, uptime or failure. The reality is that our world is more nuanced.

What Matthew recommends is to look at it through the principle of emergence because it, “separates judgment from the good and the bad binary approach to our system health, and instead focuses on behaviors and interactions, patterns and complexities of our system. With practice and effort we can manage them to more desirable states.”

But what does this look like in practice?

Getting back to the analogy of the tree and its roots, the answer is more of a forest than a tree. Trees are one living organism, forests are ecosystems.

Matthew takes this philosophy and mental picture and gives us a better system: Cynefin. It is a Welsh word that means habitat, and was created by Dave Snowden (@snowded), originally for managing IBM’s intellectual capital. It draws on research in systems, complexity, network and learning theories.

Starting in the bottom right quadrant, working counter-clockwise, its goes from simple to more complex.

Simple

These are patterns or behaviors that don’t require a great deal of understanding. DevOps is increasingly setting up automated systems to respond to simple issues.

Complicated

These are known unknowns. You can imagine a set of realities where they can occur and they are probable, but not certain. For instance, a busy harbor might get a storm that causes damage to boats, docks, etc. It is hard for the harbor manager to manage and they need to think about it. This requires people to do some thinking, and it is difficult—if not impossible—to automate.

Complex

This is where we start to see emergent behaviors occur. We don’t have the metrics need to understand or manage these problems or you haven’t looked at that metric before. We start with probing, going into the system and exploring. Think of any collection of humans at any scale. Things are still in the scope of probable, but things change quickly. There are many moving parts that aren’t predictable and that we didn’t fully encounter in our test methodology.

Chaotic

This is, well, chaos. Matthews’ real-world example was when an entire region for AWS went down, causing other regions to be overloaded as system admins were moving services. In chaos, you act, then get a sense of where things are and then respond.

Disorder

In DevOps, this is where you have lack of communication and collaboration. Here teams need to: reduce: figure out what you agree on; analyze: build consensus; and, iterate: move to a quadrant and continue.

Matthew notes that knowledge and practice move patterns toward more favorable quadrants. But, complacency erodes the process. Complex systems left poorly managed will create increasingly complex processes to manage.

How to adopt Cynefin
- In the moment: Ask, what quadrant does this map to?
- In the post incident report: How did we manage the pattern? Was it complicated, complex, simple? What can we do to change it?
- In your sprint planning: Devote time to manage your patterns clockwise. What can we move with a little bit of work?
The reality is that root cause analysis is really only present after the fact. Cynefin calls us to action.

Convinced that Cynefin might be just what your organization needs or want to dig a little deeper? Share and watch Matthew’s full talk here. You can watch any of the 2017 AllDayDevOps sessions free of charge here.

— Derek E. Weeks
March 16, 2018
DevSecOps: IoT Right and Nobody Dies

Dr. Suzanne Schwartz, CDRH Associate Director for Science and Strategic Partnerships at the Food and Drug Administration (FDA), recently released a blog to update us on the FDA’s role in medical device cybersecurity.

Cybersecurity risks in medical devices are nothing new. As far back as 2012, Sonatype published warnings of security risks in pacemakers that could lead to lethal attacks. Last year, Johnson & Johnson warned of cyber vulnerabilities in its insulin pumps. More recently, our “2017 State of the Software Supply Chain Report” shared details of pacemaker programming machines that were discovered to have more than 8,000 known software vulnerabilities (see page 37).

When it comes to software in medical and other devices, cybersecurity will be a constant threat requiring constant vigilance. The encouraging guidance I read from the FDA blog was that cybersecurity should not be limited to devices already on the market, but that security needs build in throughout the product life cycle. Schwartz remarked:

“It is the goal of FDA’s Center for Devices and Radiological Health to encourage a coordinated approach of vigilance, responsiveness, resilience, and recovery that fits our culture of continuous quality improvement.

“This means taking a total product lifecycle approach, starting at the product design phase when we build in security to help foil potential risks, followed by having a plan in place for managing any risks that might emerge, and planning for how to reduce the likelihood of future risks.”

In these two brief statements, Schwartz shares views that are common in many of today’s DevSecOps conversations. Perhaps she has been listening in?

The first view is that we need to “emphasize the performance of the entire system and never pass a defect downstream” (Gene Kim’s first way of DevOps). Schwartz recommends that device manufacturers start by building security in at the earliest stages of the development life cycle. Security cannot be an afterthought.

Schwartz also recognizes that even when precautions are taken early in the life cycle, risks can emerge over time as new vulnerabilities are discovered. When it comes to device security, vigilance must be continuous across the development life cycle and over the product’s life in the market. As we have said for years, software ages like milk, not wine. Constant vigilance improves our ability to identify risks, initiate feedback to development teams and remediate issues in the device’s software. The faster we can address risks when they appear, the safer we can make the lives of consumers who rely on them.

I applaud the FDA’s guidance and its proactive stance here to work with device manufactures. What we don’t want is a knee-jerk reaction to medical device security after someone has died. What we need is for security to be ingrained in the planning, design, production and maintenance of medical devices our families rely upon.

You can read the full blog from Schwartz on the FDA Voice.

— Derek E. Weeks

November 16, 2017

Scaling DevOps at Pearson

Pearson might be one of the more influential companies you have never heard of. Its footprint in the educational publishing marketplace is expansive, and its scope was just one challenge when it made the move to DevOps.

Sean D. Mack (@SeanDMackNYC) is formerly the VP of Operations and Application for Pearson. To give you an idea of the scale of the company’s operations when it began implementing DevOps, Sean oversaw 110 development teams and 3,000 different products across the globe. To further complicate things, Pearson is a mature company with both legacy and new products. This is a case study of how the company approached implementing DevOps with the goal of helping you learn how to do it. Sean presented it at the 2016 All Day DevOps conference.

As they say, you often learn your biggest lessons with your failures. Sean noted that, ultimately, DevOps was a failure at Pearson due to organizational change. But, he presented the case study with the hope you will learn from what Pearson did and take it forward. He continues to be a passionate evangelist for DevOps.

Pearson had its challenges. It was a siloed organization operating at a large scale, and had a toss-it-over-the-wall mentality in dealing with problems. Out of sight, out of mind.

In addition, all of Pearson’s teams were at different levels of maturity, from legacy teams running .Net on Windows 2000 servers while others doing CI/CD to AWS. Obviously, this creates challenges when implementing DevOps, but we will see later how the company tried to tackle that, specifically.

At the high level, the solution involved several aspects:

A DevOps practice area
A DevOps steering committee
Measuring progress based on global KPIs focused on measuring business impact
Measuring teams based on a DevOps scorecard
Push from the bottom – plenty of developers and operations who embraced and were eager for the change
Push from the top – support from key executives

Pearson’s DevOps Steering Committee set up several workstreams:

Tracking tools
Metrics
Deployments automation and monitoring practices
Test automation
App engineering resource contention
Operationalization
Communications

While these are specific to Pearson, many may be applicable to your organization. In any case, it is important to step back, look at your organization, and decide what your organization needs.

Pearson also looked at four core competencies to track and measure: automation; independence (how independent is the feature); operationalization (how easy it is to maintain); and continuous improvement. The below chart outlines each area in more detail.

Automation

Continuous Improvement

Independence

Operationalization

Unit tests
Critical Scenarios
Dependent Systems
Continuous Build
Continuous Deployment
Test Automation

KPI
SLA
Metric Tracking
Metric Reporting
Remediation

Isolated Configuration
Code Dependencies
Isolated Infrastructure

Monitoring and Alerting
Tier 1 Service Desk System Knowledge
Tier 2 Service Desk System Knowledge
Documentation

To help each team measure their competencies and improvement, Pearson built a self-reporting tool to empower teams but encourage visibility, and developed business metrics and then a competency model to measure teams against to see how they were progressing. As Sean noted, “We were not trying to get everyone to a 5. Some teams, yes, but we also had legacy products that were running for years, successfully, with little or no maintenance. The investment to bring the teams up wasn’t worth the effort. There had to be a business reason.“

In the end, Pearson raised DevOps maturity for all of the teams. As noted earlier, though, this was eventually phased out, but Sean presented this case study with the hope you found something to drive DevOps in your company.

In the end, if you have just a couple takeaways, they are this:

Begin by determining what your desirable outcomes are.
Don’t just do DevOps to be cool. Do it to drive business results.

You can watch Sean’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online on Oct. 24.

— Derek E. Weeks

October 6, 2017

Characterizing and Contrasting Container Orchestrators
Admiral Calcote, also known as Lee Calcote (@lcalcote) or the Ginger Geek to his friends, gave a presentation titled, “Characterizing and Contrasting Container Orchestrators,” at the 2016 All Day DevOps conference.

Okay, he isn’t really an admiral—nor does anyone call him that—but he used the title “admiral” to describe what container orchestrators do, relating it to an admiral directing a fleet of container ships. You could also say that they are like the conductor of an orchestra, directing the individuals to work together as a group toward a common goal while each musician is still able to play their own instrument.

Lee is the head of Technology Strategy at SolarWinds, and for his talk, he discussed four open-source container orchestrators: Nomad, Swarm, Kubernetes and Mesos-Marathon.

He emphasized the obvious: There is no one perfect solution. Each organization is different, so for each solution, he looked at:
- Genesis and purpose
- Support and momentum
- Host and service discovery
- Scheduling
- Modularity and extensibility
- Updates and maintenance
- Health monitoring
- Networking and load balancing
- Secrets management
- High availability and scale
Lee noted that while there are many core capabilities, any orchestrator must have cluster management and scheduling.

He then dove deeper into the four solutions. Below are summaries (the full talk is chock full of information and is online here):

Nomad
- Designed for both long-lived and short-lived batch processing workloads
- Cluster manager with declarative job specifications
- Ensures constraints are satisfied and resource utilization is optimized by efficient task packing
- Supports all major OSs and workloads
- Written in Go and with a Unix philosophy
- Host discovery: Gossip protocol – Serf is used; servers advertise full set of Nomad servers to clients; creating federated clusters is simple
- Service discovery: Integrates with Consul
- Scheduling: two distinct phases – feasibility checking and ranking; optimistically concurrent; three scheduler types when creating jobs
- Uses task drivers to execute a task and provide resource isolation, but it does not support pluggable task drivers
- Built for managing multiple clusters/cluster federation
Docker Swarm 1.12
- Simple and easy to set up
- Architecture is not as complex as Kubernetes and Mesos
- Written in Go – lightweight, modular and extensible
- Strong community support
- Host discovery: used in the formation of clusters by the Manager to discover Nodes (hosts); pull model – worker checks in with the Manager
- Service discovery: Embedded DNS and round-robin load balancing
- Scheduler is pluggable and is a combination of strategies and filters/constraints
- Ability to remove “batteries”
- Rolling updates are supported
- Managers may be deployed in a highly available configuration, but does not support multiple failure isolation regions or federation
Kubernetes
- An opinionated framework for building distributed systems
- Written in Go and is lightweight, modular and extensible
- Led by Google, Red Hat and others
- Young – about 2 years old
- Robust documentation and community
- Scheduling is handled by kube-scheduler
- Pluggable architecture and an extensible platform
- Choice of database for service discovery or network driver and container runtime
- Supports rolling back deployments, automating deployments and rolling updating applications
- Inherent load balancing
- Uses Pods, an atomic unit of scheduling. Each pod has its own IP address, no NAT required and intra-pod communication via localhost
Mesos-Marathon
- Mesos is a distributed systems kernel
- Mesos has been around the longest (since 2009)
- Mesos is written in C++
- Marathon is a framework that runs on top of Mesos
- Mesos is used by Twitter, AirBnB, eBay, Apple, Cisco, and Yodle
- Marathon is used by Verizon and Samsung
- Mesos-DNS generates an SRV record for each Mesos task
- Marathon ensures that all dynamically assigned ports are unique
Wrapping up, Lee provided the following overview comparing the different container orchestration solutions.

Lee packed a tremendous amount of information into his talk. If you use containers, his talk is worth your time and is available online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
September 20, 2017
DevOps and Security is Like Smoking Meat
It isn’t everyone who thinks, “Doesn’t Ubuntu remind you of wild boar?” Or labors over his pit of slow-roasted pork shoulder while contemplating containers. Or dwells on e2e testing while mesquite smoke permeates spareribs.

But, Apollo Clark (@apolloclark) does. Apollo is a foodie of smoked meats (is that a meatie or smokie?), working to master the craft: understanding different cuts, what each type of wood adds, and the subtleties of sauces. But, alas, it is a hobby; security and DevOps are his career.

Being passionate about both, he naturally sees parallels between the two, so Apollo presented at the 2016 All Day DevOps conference with a session titled, “What Smoking Meat Taught Me About DevOps and Security.”

Understandably, you are asking yourself, “What does smoking meat have to do with DevOps and security?” Apollo notes both have tremendous complexity and nuances, there are multiple ways of getting the job done, lots of ways to mess it up, a couple of ways of doing it right, and you are always learning.

For both, there are many tools and processes to get the job done. For smoking, your wood is a critical component, along with time. Let’s look at some parallels:
- Oak = unit testing
- Maple = coverage
- Apple = dynamic analysis
- Peach = static analysis
- Mesquite = e2e testing. People love it, but it is difficult to handle
- Wine barrel = browser support. This is when you are doing it really well
- Bourbon barrel = device support. Pretty complicated
- Smoking time = testing. You can test or smoke for five minutes or five hours, but there is a sweet spot. You can oversmoke meats and you can overtest.
Getting hungry?

Meat is critical to a meal of smoked meat, but so are many other components. Likewise, software applications take a suite of other tools and components to deliver the full package. Get your taste buds ready, because Apollo likens all of the goodness that goes with smoked meats to tools you use to keep applications running:
- Sauces = auto-scaling. They both can get pretty complicated. Always a little different, but you have to make it work for you.
- Bread = monitoring, which is the bread and butter of infrastructure. Make sure you have it up and down your stack.
- Salad = system logs. Not the sexiest things, but you can rely on them.
- Fruit = custom application logs. Takes a lot of time to pair them, but takes a really good thing and makes them better.
- Beer = firewall. You should always have both.
- Wine = Intrusion detection systems. These are your fine wines.
- Whiskey = IR training. It takes time and there are so many ways of doing it. When things break, we have procedures on how to deal with them.
When smoking meats and in DevOps and security, Apollo asks and answers, “Do we have to do everything? No, but the more we do, the better it will taste.

“Does it cost money and take time? Yes, but you can do great things even without money and time. You can use cheap meats and you can use open source. Start simple, build up complexity, and always be learning.” Constantly ask yourself, “Am I better today than I was yesterday.” If not, be better.

You can watch Apollo’s entire talk online here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
September 6, 2017
Securing Immutable Servers in a Serverless World
Snowflakes are beautiful, unique creations. But, let’s keep them in nature. They don’t belong in our server infrastructure. Snowflake servers, where every configuration is just a little different, can introduce unnecessary security vulnerabilities and complications. While common in IT infrastructure, in the DevOps realm they are gradually becoming ancient history.

At All Day DevOps 2016, Erlend Oftedal (@webtonull), with Blank and head of the OWASP Norway chapter, discussed the benefits of immutable infrastructure practices within serverless architectures. Erlend walked us through the positive effects on security as well as offered insight on potential problems.

Erlend outlined the progress from physical servers to immutable servers, discussing the benefits of each step and the concerns that lead to the next phase, ending with immutable servers—that is, they can’t be changed. Once set up, they are set. Automated processes coupled with immutability principles can eliminate snowflake servers.

Immutability isn’t a new concept. Some classic examples include live Linux CDs, internet cafes, schools, libraries and hotel business centers. To set up immutable servers, Erlend outlines these steps:
1. Find a suitable base image
2. Build a specific image for the application
  1. Dependencies
  2. Application
  3. Secrets
3. Deploy image to the server(s)
4. Repeat steps 2-3 for new versions
Of course, this immediately raises questions:
- What about data? He notes that because data inherently changes, data has to be stored externally.
- How do you deploy changes? Well, you don’t change the server. You take one out and put a new one in. What if that one fails? Roll back to the previous one.
- How do you manage secrets? You can build secrets in, leverage cloud key management, or use third-party services.
- How do we apply security patches? If truly immutable, you deploy a new server with the patch. However, you can have a semi-immutable server that allows for automatically installing patches but still doesn’t allow for logins or manual changes.
- Speaking of security, what are the security advantages? For starters, if an attacker is in your server, it will be thrown out when you change servers, although it may come back if the vulnerability isn’t patched. It also allows for very specialized images by removing all unnecessary packages. With these specialized images, you can more easily audit and monitor the system for unexpected file changes, logins and connections.
Erlend discussed containers because they are a step closer to a serverless world. Containers isolate processes and all run on the same OS. Of course, you can make them immutable, but regardless of whether they are, there are some key security best practices you’ll want to consider:
- Don’t run as root inside container
- Implement user namespaces so the root account within the container is separated from the root account outside the container
- Utilize the least privileges necessary for capabilities and resources
- Keep them up-to-date
- Automate scanning using tools such as Sonatype Nexus, Twistlock, Clair, or Aqua
- Separate untrusted and trusted containers
Erlend also dove into serverless architecture. While there is debate over what exactly qualifies as serverless, there are five principles that Erlend highlighted:
- Use a compute service to execute code on demand
- Write single-purpose, stateless functions
- Design push-based, event-driven pipelines
- Create thicker, more powerful front ends
- Embrace third-party services
Migrating to immutable, serverless infrastructure is a topic to be discussed well beyond this blog post and even Erlend’s session. Whatever you do in the future, though, take his parting words to heart: “The cloud can solve many of your problems, but in the end you cannot transfer your responsibility for security.”

Erlend’s session was packed with information. If this piqued your interest, you should watch his complete session from All Day DevOps 2016 (just 30 minutes). The other 56 presentations from the conference are also available online, free-of-charge here.

Also, this year’s All Day DevOps is Oct. 24. It’s online and free, so be sure to check out www.alldaydevopsy.kinsta.cloud to learn more and register.

— Derek E. Weeks
August 28, 2017
Cybersecurity Improvement Act of 2017: The Ghost of Congress Past
It seems like yesterday when when Representative Ed Royce proposed legislation titled the “Cyber Supply Chain Management and Transparency Act.” Almost three years have passed since Congressman Royce introduced his bill in November 2014 and helped educate the world about serious security vulnerabilities (such as Apache Struts and Heartbleed) lurking inside open-source components that are commonly used by anyone and everyone building modern software applications.

Broadly speaking, Royce’s bill would have required technology vendors selling software to the U.S. government to do the following:
1. Provide customers with a bill of materials documenting all open-source components utilized in the software application;
2. Demonstrate that the component versions utilized in the application have no known vulnerabilities (CVEs from NVD) for which less-vulnerable alternatives are available;
3. Provide a mechanism to promptly remediate new vulnerabilities when they are discovered.
Despite dying a quiet death, the Royce bill helped to pioneer an important conversation between government and industry leaders with respect to the age-old question of software liability.

This conversation continues to evolve—and just recently Sen. Mark Warner, a Democrat in Virginia, and Sen. Cory Gardner, a Republican from Colorado, introduced the “Internet of Things Cybersecurity Improvement Act of 2017“—an attempt to force companies selling IoT devices to federal agencies to adhere to new security standards.

Just like the Royce bill before it, the newly proposed legislation from Sen. Warner would require vendors selling IoT-connected devices to government customers to do three simple things:
1. Provide written certification that IoT devices do not contain hardware, software or firmware components with any known security vulnerabilities or defects listed in the NVD or similar databases.
2. Notify government customers of any new security vulnerabilities or defects subsequently discovered.
3. Provide a mechanism that allows for any future security vulnerability or defect in any part of the software or firmware to be patched or fixed in a timely and secure manner.
As Bruce Schneier observed more than a decade ago, there are no real consequences for having bad security or having low-quality software of any kind. Even worse, the market often rewards low quality. More precisely, it rewards additional features and timely release dates, even if they come at the expense of quality.

Although the Royce bill failed and the future of the Warner bill is yet to be determined, there is an increasingly steady breeze blowing from Washington, D.C., that is gently nudging the entire software industry toward a future in which vendors will no longer be immune to liability for damages due to known security vulnerabilities or defects.

About the Author / Matt Howard

A proven executive and entrepreneur with over 20 years experience developing high-growth software companies, Mr. Howard leads Sonatype‘s corporate marketing, strategic partnering, and demand-generation initiatives. Prior to Sonatype, he co-founded, developed and successfully sold two software companies. Earlier in his career, Howard led sales and marketing at USinternetworking (acquired by AT&T) and Groove Networks (acquired by Microsoft), where his teams distributed workgroup collaboration products to enterprise customers. Howard holds a Bachelor of Arts degree from The George Washington University and a Master of Arts from George Mason University.
August 17, 2017
All Day DevOps: Taking Lambda to the Max
Lambda is the 11th letter of the Greek alphabet, and it is also the name of the Amazon Web Services (AWS) service that lets you run code without actually configuring a server. Amazon chose the name Lambda because—well, okay, I have no idea. Comment here if you know.

Thankfully, you don’t need to know that to understand how to utilize Lambda. Matt Williams, DevOps Evangelist at Datadog (@technovangelist), outlined how to set up AWS Lambda at the 2016 All Day DevOps conference in his talk titled, “Hold the Infrastructure: Getting the Most Out of Lambda.”

First, to understand what Lambda is, we must look at the evolution of servers.

We started with physical boxes—we had to buy and maintain these boxes, and no matter how much we used, we had to pay for all of it. We then evolved to multiple virtual machines on a physical server so that we could maximize the resources of physical boxes. Now, AWS and similar cloud services are common. We create a virtual machine that we need, and the service provider maintains the physical servers. We also have containers to keep environments identical across machines and, finally, we arrive at Lambda, which is service that runs code but doesn’t require you to set up the server. It takes care of all of that automatically, and you only pay for the time that the function is actually running. So, if your function takes 500ms to run, you pay for 500ms to run it once.

Lambda is trigger-based. When it receives a trigger, it executes a specified function. The trigger can be a variety of actions with AWS services, such as when a file shows up on S3 or an Alexa skill is requested.

Lambda is serverless, as far as you are concerned.

Lambda is stateless. This means you can’t rely on local states, so, if you need to retain information between function calls, you have to save that data somewhere else.

Lambda executes functions. Each function:
- Is code in a container on LXC on AMZN Linux
- Does not have restrictions on what it can do
- Is priced based on exec time, quantity and memory
Lambda links to various AWS services, such as:
- Files (S3)
- Database Updates (DynamoDB)
- Streams (Kinesis)
- Messages (SNS, SES)
- IoT (Echo, API Gateway)
Matt mentioned several good use cases, such as bootstrapping (the first 1M requests per month are free), cron jobs, file manipulation, API translation, social media stream processing, scaling infrastructure, mobile/IoT backends and Alexa skills, to name a few.

He also mentioned three specific examples to explore:
Matt then outlined how you can get started and noted that setup is simple as long as you think of application development as simple (write the code, set up on AWS, and debug). He walked through a specific example of automatically resizing images on S3 when they are added. You can watch his full presentation with screenshots here.

Matt also provided a list of some AWS Lambda Tools:
Matt rounded out his 30-minute presentation with a discussion regarding the need for monitoring and how to use Lambda for monitoring.

You can watch Matt’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
August 17, 2017
All Day DevOps: Modern Infrastructure Automation
“We’re no longer an airline. We’re a software company with wings,” claims Veresh Sita, CIO, Alaska Airlines. The success of today’s businesses rests on software. It is an integral part of infrastructure, so we need to always ask, “How can it be better? More reliable? More secure?”

Nathen Harvey (@nathenharvey) is the VP of Community Development at Chef and the co-host of the Food Fight podcast. He is also an evangelist of automating software operations, which is what he presented on at the 2016 All Day DevOps conference.

Nathen started his talk by stating that “automation at scale is required for high velocity IT,” making the point that in today’s environment, you can’t wait months between deployments. Enterprises are moving to continuous integration/continuous deployment, and he advocates for continuous automation.

To implement automation at scale, you have to start with a dynamic infrastructure by:
- Provisioning and setting up environments
- Implementing dynamic scaling of compute resources
- Migrating legacy workloads to the cloud
- Deploy in multi-cloud and hybrid-cloud environments
- Support heterogeneous environments
Several speakers of All Day DevOps talked about automation, and all of them said a critical step is to treat infrastructure as code. According to Nathen, this means that to programmatically provision and configure components, such as servers, databases, firewalls, etc., you also have to treat it like any other code base. This includes version control and automated testing, but also:
- Automated, full-stack application policies
- Package and service installation
- Versionable, testable and repeatable workflow
- Scalable application policies
- Management of interdependencies across nodes
When you treat your infrastructure as code and deliver at scale, you can scale elegantly from one to tens of thousands of managed nodes across multiple complex environments.

For software to be effective, users have to trust that it will do what they expect it to do. This is true from handheld calculators to self-driving cars, and it is also true when we automate our development and operations. Nathen advocates that we have to build a pipeline to production that creates faith in us that our code is ready for production. This pipeline must:
1. Test the code/verify locally
2. Verify in pipeline with automated testing
3. Approve with another engineer (aka code review or Nathen’s “four eye rule”)
4. Build an artifact
5. Deploy into acceptance environment to automatically test the artifact
6. Ask someone else, “Do we want to ship this?”
7. Ship the code with a click of a button
8. Union (bring together dependencies and run tests)
9. Rehearsal
10. Delivered (ship to customers)
What does the future hold?

Infrastructure automation is NOT enough. Why? Auditors show up to audit your infrastructure and applications. The problem is, they use different tools—Excel spreadsheets, notebooks, PDFs, etc. This makes it very difficult to audit. Nathen advocates managing compliance documents as code, too. When we translate compliance into code, it can be executed as part of the pipeline so we can test for compliance throughout the entire life cycle.

Finally, Nathen stresses the need to have application automation, too. Modern applications look like infrastructure as code.

Developers can make applications that operations can deliver and run in production. This makes it easier to consume, run, and keep applications healty. Operable applications need to be:
- Isolated
- Immutable
- Configurable
- Built on a common interface for monitoring health
- Rebuildable from sources
- Reside in common packaging
- Maintain runtime independence
As enterprises implement infrastructure, application, and compliance automation, it creates a freedom for people to focus, and people are the key.

Nathen notes that, “DevOps is a cultural revolution and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.” He underscores that people are the future of automation. Automation places the priority on people, which enables great companies and great products.

You can watch Nathen’s entire talk online here, where you can dig deeper into this subject. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online on Oct. 24.
— Derek E. Weeks
August 2, 2017