Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Google’s full-stack, browser based development environment in the cloud. It’s not just a Copilot clone, but aims to help you “get an app from zero to production.” (more…)
The Stack Overflow annual survey of developers came out this month, and it has a lot of interesting data points in it. There is a lot that I wish was different about their questions/methodology, but one thing caught my attention. Docker, Kubernetes and similar tools (Ansible and Terraform are in there at lower percentages, for example) make up more of the responses than can be accounted for by people with “DevOps” in their title.
Sometimes by a lot.
That means we have a fair number of developers that are doing DevOps, but who aren’t DevOps team members. Yeah, there are a few systems admins in the ranks of the respondents also, but not enough to cover the discrepancy. If every respondent that was in DevOps and sysadmin roles used Kubernetes, there would be just enough to cover the number of respondents that said they used Kubernetes “extensively” over the past year. It’s unlikely that all of them are Kubernetes specialists. And that doesn’t touch on Docker, with a whopping 40% of respondents saying they use Docker extensively. Any developer with recent experience can explain away Docker—though “extensive” is a bit strong for your average developer’s usage, it is becoming part and parcel of development, using containers to host dev environments, test, even build tools. And, of course, some deployments. But 40% is still a lot.
So, if your title is developer, and you’re using deployment tools on a regular basis, go ahead and call yourself DevOps. That’s not the only criteria, but if you were going to pick one, the breadth that using deployment tools represents combined with the implication that there isn’t an ops person worrying about that type of thing makes “uses deployment tools” a good stand-in test.
To the nearly 18% (20% of pro developers) of you using Yarn, I tip my hat. I used it for a couple of years early on in its life, and am happy to say I do not need to use it any longer. I assume your usage is not tied to Hadoop and that it has greatly improved over my 2015-2016 experience. Still, y’all rock, just by definition of using it to get resource scheduling down. It ain’t easy.
Across the board, the number of DevOps/Ops tools that make these lists is amazing for a developer survey – nearly 9% of respondents use Ansible, and nearly 9% use Terraform. There is undoubtedly overlap, so the actual number that uses one or the other is less than 18%, but still, nearly one in 10? That’s huge. And that’s DevOps or Ops, not purely developers. Regardless of title.
And that’s been a hallmark of development since the field took off—doing what needs to be done to deliver software. This is just another iteration, and you’re rockstars, regardless of title. You keep delivering apps that run the business—let them call you whatever they like. Money talks; titles are pretty much meaningless. Not like you’ll retire on a title, but bonuses or stock grants? Those you might retire on.
The idea of “full-stack developers” is becoming a fairly common one, although it is still widely contested. But what about full-stack operations?
There are cycles in the IT industry, and if you wait long enough, you can see the same ideas swing in and out of favor. Business applications used to be fairly monolithic, and so it was expected that their programmers would have a good understanding not just of the application’s code and business logic itself, but of its underpinnings as well.
Over time, different parts of those underpinnings were separated, breaking out data silos in a process of modularization. With the increasing complexity of architectures, the expectation became that specialists would be in charge of each module and simply ensure that standard interfaces were available to teams whose modules needed to interact with each other.
While modularization at the architecture level is here to stay, there is now some push back against it being extended too aggressively to the hum
an developers in charge of building the system. This takes the form of a quest for “full-stack developers,” who are able to understand all the nuances of the entire application stack.
There is a similar need on the IT operations side of the house. Different specialized support teams used to be able to focus on one particular area (server, network, database, middleware and so on) and use their own specific tools to analyze and understand the performance of one particular module of a complex enterprise architecture. The assumption was that if each module was up, then ipso facto, the entire stack was up, and therefore the application as a whole was also up.
As it turns out, it’s not that simple.
These days, failures of business services rarely are caused by a catastrophic outage of one particular component. Hardware is cheap, resiliency is built in and experience from web-scale environments has trickled down to become common best practices. What actually causes problems—such as getting you embarrassing news coverage or angry customers demanding refunds—is the unexpected: many seemingly minor and isolated issues that suddenly add up to a big problem.
When this sort of cascading failure occurs, modular operations teams and processes are not able to cope. Multiple tickets are opened for each affected module and routed to that module’s support queue. Each of those tickets contains only partial information, by definition, and may even describe only a symptom of a failure in another module. Because of this, precious time is wasted diagnosing, reassigning and escalating those tickets, until eventually a complete picture of the actual problem is laboriously assembled.
At this point, representatives of all of the various affected support teams assemble, either in person or on a conference call, and try to determine what to do to restore service as quickly as possible. All too often, these efforts are further delayed by unproductive blamestorming and finger-pointing, but eventually, the issue does get resolved.
Given the complexity of modern-day enterprise architectures, it is impractical for any single IT operations specialist to understand the entire stack in depth, including all the interactions between different modular components. Even if it were possible to do so for a snapshot in time, these days most environments are self-modifying to a greater or lesser degree, provisioning or decommissioning capacity in response to demand, balancing load across available resources and performing routine maintenance tasks. This sort of dynamism is impossible for humans to model accurately in real time.
No single person can be a full-stack ops specialist, but that does not mean that the whole team cannot work to deliver full-stack support. The key is to break down the walls between the different siloed teams supporting each module, and make it easier to build an overall picture of, yes, the full stack that is supporting a particular business service.
This bringing-together must happen in real time, because that is the pace of business today. An outage that has a technical duration measured in a handful of minutes may cause hours-long business repercussions. Ops teams need tools that can relate signals from different modules into a comprehensive picture of what’s going on in the IT environment, proactively inform the right people, and give them the support they need to pool knowledge and work together effectively to resolve the outage and restore service to affected users.
These tools cannot be simple incremental improvements in the monitoring of one particular module, important as that is. The responsiveness that the organization as a whole expects from its IT operations requires that silos be broken down and information and expertise be pooled. Real-time collaboration between specialists in different domains is the only way to deliver truly full-stack operations.
The rapidly maturing discipline of AIOps aims to do just that. The beauty of this approach is that, unlike rip-and-replace approaches in the past where moving to the new thing meant ripping out the old thing, AIOps takes advantage of what was done in the past, simply joining together what were previously disconnected islands of knowledge and making information available when it is needed.
Take a look at some of the players building out these capabilities in the AIOps Market Survey from Gartner. Normally these documents cost $1,295, but you can get it for free courtesy of Moogsoft at this link.
Nearly 50 percent of Node.js developers surveyed using container technology, strong growth emerges in cloud, front end, mobile and devices
SAN FRANCISCO, April, 12, 2016 — The Node.js Foundation, a community-led and industry-backed consortium to advance the development of the Node.js platform, today announced its first ever Node.js User Survey Report.
With over 3.5 million users and an annual growth rate of 100 percent, the Node.js Foundation survey identified that Node.js is emerging as an universal platform/language used for web applications, IoT, and enterprise. It features insights on emerging trends like microservices architectures, real-time web applications, Internet of Things (IoT), and also paints a detailed picture of the technologies that are being used with Node.js in production and language preferences (current and future) for front end, back end and IoT developers.
Node.js and Containers Take Off Together
Both Node.js and containers are a good match for efficiently developing and deploying microservices architectures. And, while the surge in container use is relatively new, 45 percent of developers that responded to the survey use Node.js with the technology. Other container-related data points:
58 percent of respondents that identified as IoT developers use Node.js with Docker.
39 percent of respondents that identified as back end developers use Node.js with Docker.
37 percent of respondents that identified as front end developers use Node.js with Docker.
Node.js — the Engine that Drives IoT
JavaScript and Node.js have risen to be the language and platform of choice for IoT as both are suited for data intensive environments that require parallel programming without disruption. JavaScript, including Node.js and frameworks, such as React, have become the de facto choice of developers working in these connected, device-driven environments with 96 percent of IoT respondents indicating they use JavaScript/Node.js for development.
These environments are challenging, and the survey revealed that on average, IoT developers using Node.js have more experience than their front end and back end counterparts with over 40% percent of IoT developers surveyed having over 10+ years of development experience.
Additionally, although Docker is a server technology, many IoT developers (58%) are using Node.js with Docker compared to only 39% of back end developers. This metric is significant as it means that the new IoT world also is quickly adopting containers and microservices.
Node.js Becoming Universal Language
The full stack is no longer “front end and back end,” but rather “front end, back end and connected devices,” which is a combination of everything from the browser to your toaster all being run in JavaScript and enabled by Node.js. The survey revealed that 62% of respondents are using Node.js for both front end and back end development, and nearly 10% are using Node.js for front end, back end, and IoT development.
Node.js Pervasive in Enterprises
Node.js is increasingly used in the enterprise, and used within huge enterprises like PayPal, Go Daddy, Capital One, and Intel. The Node.js Foundation survey respondents backup Node.js’s growing role in the enterprise:
More than 45 percent already using the Node.js Long Term Support release (v4) geared toward medium to large enterprise users who require stability and high performance.
Of those who haven’t upgraded, 80 percent report definite plans to upgrade to v4, with half of respondents planning to do so this year.
Strong interest in enterprise tooling among 34 percent of tech leaders.
Full “MEAN” Stack Explodes
The popularity of real-time, social networking and interactive game applications is pushing a new stack among developers. The MEAN stack is able to handle lots of concurrent connections and extreme scalability, which these applications demand. Node.js, in combination with MongoDB, Express, Angular.js, allows developers to tackle the needs of front end and back end development. Not surprisingly, all of these technologies were commonly used alongside Node.js. Express, cited the most, used by an average of 83% of developers.
Popularity of JavaScript and Node.js
JavaScript and Node.js were popular among back end, front end, and IoT developers. Other languages that were popular for all three developer types included PHP, Python and Java. However, when looking to the future, back end, front end and IoT developers planned to decrease their use of Java, .Net and PHP (PHP averages a 15% decrease) and increase the use of Python and C++.
About the Survey
The survey was open for 15 days, from January 13 to January 28, 2016. During this time, 1,760 people from around the world completed the survey. Seventy percent were developer’s, 22 percent technical management and 64 percent run Node.js in production. Geographic representation of survey covered: 35 percent from United States, 22 percent from Continental Europe, 6 percent India, and 6 percent from United Kingdom with the remaining respondents hailing from Asia, Latin America, Africa, Russia and the Middle East.
Additional Resources
Node.js Foundation User survey infographic
Report summarizing Node.js Foundation User Survey 2016
About Node.js Foundation
Node.js is used by tens of thousands of organizations in more than 200 countries and amasses more than 3 million active users per month. It is the runtime of choice for high-performance, low latency applications, powering everything from enterprise applications, robots, API engines, cloud stacks and mobile websites.
The Foundation is made up of a diverse group of companies including Platinum members Famous, IBM, Intel, Joyent, Microsoft, PayPal and Red Hat. Gold members include GoDaddy, NodeSource and Modulus/Progress Software, and Silver members include Apigee, AppDynamics, Codefresh, DigitalOcean, Fidelity, Groupon, nearForm, New Relic, npm, Opbeat, RisingStack, Sauce Labs, SAP, StrongLoop (an IBM company), Sphinx, YLD!, and Yahoo!. Get involved here: http://nodejs.org.
The Node.js Foundation is a Linux Foundation Project, which are independently funded software projects that harness the power of collaborative development to fuel innovation across industries and ecosystems. www.linuxfoundation.org
###
*Node.js Foundation is a licensed mark of Node.js Foundation. Node.js is a trademark of Joyent, Inc. and is used with its permission.
Deciding on two separate teams, or an integrated end-to-end team
While a lot has been said about “the right way” to do DevOps, what we’re seeing in practice are two different approaches being tried to enable development and operations to work together as a team. On the one hand, there’s the approach to create an end-to-end, integrated development team, which you might call ‘Devops’. On the other, there’s ‘DevOps’, featuring two development teams – one working on application code, with another developing a platform using Infrastructure as Code techniques in a virtualized environment.
DevOps: providing a platform
Perhaps the most obvious way to go is to have a platform team to provide a runtime environment that “just works”, and a separate application team that develops the business logic to run on the platform.
For some organizations, particularly where one of the teams is outsourced, this separation is going to make more sense. You might buy an off-the-shelf product, build your own platform in-house (which seems to be what many organizations are looking to do with Docker right now – warning: it’s harder than it looks!) … or sign up with a PaaS vendor.
Whatever the situation, the platform should be versioned, just like any other piece of software.
This approach can be ideal for developers because they are free to work on code and focus on the business logic, and they really don’t need to worry about how it runs in production. It’s up to the operations team to make sure that the platform is stable and that it delivers what the developers need.
Communication is still key
Even with a physical or organizational separation, the importance of effective dialogue between the two teams cannot be overstated. This won’t work if the relationship is adversarial. There has to be a genuine awareness of common goals. If the operations team doesn’t understand where the development team is going, and what they need from the platform, then there’s going to be trouble. The idea that one platform is going to cater for every development need is often unrealistic, so versioning and the ability to run multiple platform versions side-by-side, or cater for exceptional situations where a particular application might need its own stack, is very important.
There has to be a clearly defined boundary, which should be close to the application code and configuration layer. The application handles the business logic, the platform handles the deployment, logging, monitoring, failover and auto-scaling.
The development team can focus on getting those new features and improvements into the live product, while operations handles the actual roll-out and manages traffic. For this to work, the platform must enable you to quickly determine who is responsible for any problem that crops up. It’s worth investing time in the setup to ensure that errors are explicit, so you don’t waste time tracking them down.
Arguments between the two teams are bad for progress and morale.
Devops: A full stack development team
For some businesses, it’s going to make more sense to put together integrated development teams that work directly on delivering the full stack from a shared code base, from virtual machine to application code. This allows you to design and optimize each system in a way that takes the application’s specific perspective and requirement into account, and can be an especially good fit if your applications differ from each other greatly in terms of the operating systems, middleware, etc., that they run on.
This approach can only work if the overall group is small (Amazon’s “two pizza” teams are a good example) and has sufficient knowledge and experience to see the big picture of the entire system.
It’s not possible for everyone to understand every part of the full stack in detail, but there are areas of overlap where greater collaboration will increase the level of shared knowledge and allow the team to build a better overall system.
Pulling together
The shared foundation of the Devops approach builds mutual respect and enables more free-flowing communication, but it must be based on shared goals and a vision of what the overall purpose of the system is. This integrated approach can give you much more flexibility, but it also means holding the whole team responsible for keeping everything running. Shared code equals shared responsibility.
The right approach
You need to take a baseline, and then put metrics in place to track your progress. You have clear goals in mind and a system to measure them. So how do you find the right approach to achieve those goals?
It would be wrong to say that one approach is better than the other. What you need to do is assess what will work in your organization. Both are means to achieving similar goals, but finding the right approach is about understanding what your company is looking to achieve. You might choose common goals, like reduced expenditure, or reduced error, or you might choose something more specific to your industry. Establish your goals at the outset, and make sure that they’re measurable.
The right approach is going to be the one that gets results for you — that’s why setting up a baseline and metrics is so important. You may find that a hybrid of the two approaches is going to be the right fit, but you aren’t going to know for sure until you have some hard data to back up your expectations.
Be sure to join devopsy.kinsta.cloud editor-in-chief Alan Shimel on Thursday, May 21st as he hosts the webinar, “Ship Faster without Breaking Everything” to further your smarts on Continuous Delivery.
About the Author/Andrew Phillips
Andrew Phillips heads up product management at XebiaLabs, a provider of continuous delivery automation. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.