devopsy.kinsta.cloud

Tag: automation

DevOps Proves No Coaster Ride for Roller
Keeping amusement parks running and the fun flowing is Roller‘s main job. The company provides software for the leisure and entertainment industries, notably theme parks, amusement parks and trampoline parks. Its core functions are ticketing, POS and CRM. Clients include Coney Island Luna Park, Aussie World and Twinlakes Park.

When Roller decided to expand and redevelop its operations in 2015, DevOps was the main attraction. CTO Andrew Brodie led the company’s push, having worked on Roller’s original implementation in his previous role as co-founder and technical director of digital product design studio Bravo!

Automation of Roller’s processes was a major goal. Even small organizations should adopt good practices, Brodie says, and that includes automating everything that can be automated (within reason) to the point of being able to click one button to deploy a system.

Before DevOps

In its early days, Roller used Microsoft tools including Web Deploy to deliver a monolithic application that met the requirements of a minimum viable product. Roller initially ran on Ninefold, a now-defunct Australian IaaS platform.

When the company attracted additional funding in 2015, it was able to expand its development team with people who had various skills including DevOps. Today, it comprises around 50 people—12 developers and various user experience, quality assurance and project management specialists.

The newfound expertise enabled the redevelopment of the system as a series of applications and microservices and eliminating most frequently performed manual operations.

Tools

Roller’s deployment process had become a bottleneck, so the company invested time in adopting the most appropriate tools. Several came from HashiCorp, including Packer (for creating machine images), Terraform (cross-provider infrastructure management), Consul (service discovery, configuration and orchestration) and Vault (centralized storage and control of distributed secrets such as keys).

After Ninefold’s demise, Roller became heavily involved with AWS, which “had everything we needed” in the way of APIs and toolsets “to code out our infrastructure,” Brodie says. But even though “we’re really happy with Amazon,” the company has avoided using exclusively AWS tools to retain the ability to move to other platforms such as Azure.

Implementing load balancing, auto scaling groups and so on as code took about a month’s effort. “It was a great move for us,” he says.

Other tools used by Roller include the Chocolatey package manager, and Microsoft’s PowerShell and its Desired State Configuration feature. After a machine image is tested, that’s exactly what moves into production, providing confidence that it will work as expected.

Roller uses blue-green deployment with autoscaling groups to make it easy to roll back if necessary. Deployment or rollback takes less than 10 minutes. When individual problems arise, Roller’s practice is to kill the unhealthy server and bring up a new one.

The company maintains separate deployment pipelines for each application and service, using Atlassian’s Bitbucket distributed version control system and its Pipelines feature to automate the deployment of checked-in code.

Roller can deploy code within an hour if necessary, but currently works to a weekly release schedule. The goal is to move to multiple releases in a day.

Resources

Major Australian jobsite Seek served as an inspiration for Roller’s DevOps path. “We’ve always looked at the bigger players,” Brodie notes. In addition, Meetups (“there’s something for everyone”) and conferences (he specifically mentioned DDD Melbourne and NDC Sydney) have been invaluable in being able to talk with other companies about their experiences. “We’re keen to stay ahead of the latest practices,” he says, so the company encourages its people to attend such events.

Other resources he finds useful include Stack Overflow and Plural. Plus, he recommends “The DevOps Handbook” by Gene Kim, Jez Humble, Patrick Debois and John Willis, which he describes as “a fantastic resource,” providing practical advice particularly around issues at the organizational level.

Tips

When asked for advice for those starting a new business or major project, Brodie recommends pragmatism:
- Consider DevOps from the start “but don’t overcook it.” In other words, begin with the basics and evolve from there.
- Aim for continuous improvement—people will keep generating new ideas, but picking the right time to implement them is important.
- Look for a mentor, and talk to people about the tools they’re using.
- Take advantage of outside organizations’ capabilities. For example, Rackspace can help architect an application for AWS.
— Stephen Withers
August 22, 2017
Automation: Critical Missing Pieces in the CD Puzzle

As enterprises accelerate the frequency with which they develop and release new code into production to meet the ever-mounting demands of their consumers, there’s growing pressure on networking and security operations teams to keep pace with their DevOps counterparts. And it’s not hard to see why. According to Forrester, nearly four out of five business leaders believe it is IT’s responsibility to ensure the network can support the company’s digital transformation plans.

And yet, 67 percent of businesses believe the network is the bottleneck—and with good reason. The current networks that NetOps and SecOps teams manage are highly complex, distributed entities that have historically been configured and provisioned manually—and often, in silos. Yet, as DevOps accelerates ahead at lightning pace, underpinned by the continuous integration, delivery and deployment (CI-CD) approach, NetOps and SecOps teams are scrambling for new ways to gain speed to more quickly launch, scale and secure applications and code updates in production.

The Need for Speed

Enterprises are increasingly leveraging DevOps principles to execute daily deployments into their multicloud test and production environments. For example, Etsy deploys 80 times per day, while Netflix and Amazon deploy up to thousands of times a day. In this new world order, collaboration, communication and synchronization across development, operations, network and security teams is essential.

This also means that testing of the network and application services infrastructure must happen concurrently with the application development and testing processes—not afterward, which can dramatically slow down the agile CI-CD process. Similarly, security cannot be retrofitted after the development and test phases are complete. A recent study by Puppet Labs found that high-performing teams spend 50 percent less time remediating security issues when security is addressed at every stage of the software development and delivery cycle.

New CI-CD Tools Deliver on the Promise of DevOps

New innovations in virtualization and programmability of network and application service infrastructure are making it possible to test policies and configurations prior to production—even in highly complex environments. Powered by validated templates and APIs, these network orchestration and application services tools help automate task execution for scale, preserve provisioning and configuration policies for consistency and bolster security, while dramatically reducing management time cycles and the risk of human error. They empower NetOps and SecOps teams to leverage DevOps principles and apply them to network and security during the development, deployment and delivery of multi-cloud apps and code updates.

Similarly to how software-defined components such as servers and storage are provisioned, network orchestration tools leverage a software-defined infrastructure (SDI) approach to rapidly provision network services across multiple network devices. These tools support flexible application provisioning across physical and virtual environments, help identify and address issues quickly and improve overall network performance. In the CI-CD model, network orchestration helps NetOps and SecOps teams ensure that all production-critical application and network elements—such as availability, core network services, security services and performance—are accounted for and are rolled out in proper sequence and at scale.

Application services automation focuses on application availability, performance, and security. While automation used to focus solely on simple load balancing, it now also includes a range of solutions that help network teams configure and automate application services during the test phase. These include web app and network firewalling, DDOS protection, application acceleration, WAN optimization, DNS services and TCP optimization, among others. In the dev and test phases of the CI-CD process, automation allows for faster configuration and deployment of network settings and policies. Similarly, in the production phase, automation significantly speeds up and simplifies areas such as policy adjustment and other time-intensive processes by eliminating the need to reconfigure underlying infrastructure. This means that ticket resolution and necessary changes can be completed in a matter of minutes versus weeks. At the time of rollout, application services automation toolsets also help to ensure the infrastructure is ready to perform at peak levels.

Network and application services automation innovations—the critical missing pieces in the continuous deployment puzzle—work together to ensure the entire DevOps team—including network and security—can deliver and deploy applications with the speed, reliability and security necessary to meet today’s business challenges.

About the Co-Author / Ranga Rao

Ranga Rao is Director of Product Management & Solutions Engineering at Cisco Systems, where he leads a team of product managers, architects/TMEs and software engineers to collaborate with partners to define use cases and build software integrations for leading Cisco solutions including Cisco ACI, Nexus 9000 and Tetration Analytics. Ranga earned an M.S. in Computer Science at Stanford after receiving his B.E. from the National Institute of Technology Karnataka. Ranga is also an author of a popular book, Programmability and Automation with Cisco Open NX-OS, published by Cisco.

— Lori MacVittie

August 9, 2017
Jobs-as-Code: The Business Processing DevOps Forgot

Developing web and mobile apps is cool. But in the words of a longtime ballad: “Don’t forget who’s taking you home and in whose arms you’re gonna be …” It’s the back-end services that underpin a lot of those cool apps and make them possible. So yes, servers and relational databases and enterprise applications and sometimes even mainframes are definitely a thing. And in those environments, scripting, batch processing, application integration and operational visibility are a very big thing that, more often than not, is managed by workflow and job scheduling tools.

I argue that the “something—application, code, other” elements used to deliver all those capabilities are logically part of the application and should be treated as such. The fact that “process X” needs to start only after certain data has arrived, been cleansed and inserted into some database is just as much a part of the application as calculating an invoice amount or determining that inventory needs to be replenished.

And if you are still reading, then I submit that you need jobs-as-code.

Simply stated, jobs-as-code means that the jobs—automation rules for business applications—you need to manage and execute systems of record, back-end application components and data pipelines in production are built and managed in the same way as Java, Python, C++ or any other business logic code. In other words, everything that is part of the application should be included in the software development life cycle (SDLC).

Of course, that statement is pretty simple to say but not that easy to accomplish, because traditional workflow, job and batch processing solutions have lived (and most continue to) in the operations world. The practitioners of enterprise scheduling tend to prefer graphical interfaces with a strong emphasis on ease of use and rigid control over access. Traditionally, capabilities such as developer self-service, automated build and test and deploy have not been high on most enterprise job scheduling new feature requirements wish lists. Most commercial solutions, if they do anything, add an API or two and consider the job done. Open-source authors build tools that cater to developers with little to no regard for the multitude of other user types that rely on batch workload.

The resulting state in the industry is that developers spend lots of time building “snowflake” operational plumbing which may be difficult to test, difficult to embed in an automated delivery pipeline and either has to be reworked to make it ready for production or becomes an albatross around the necks of operations teams trying to keep it running. It’s time for the DevOps community and the industry to recognize there is a better way and we all instinctively know what it is.

Jobs-as-Code

To understand jobs-as-code, let’s examine the traditional dev-to-ops handoff from a developer and DevOps perspective.

I write some code and now I want to test. The execution is simple; I want to pull some data from a relational back end, run my code to manipulate the data and push the result set to some other location to serve as a data source for a different application. I need a SQL query, run my Java code and ftp or sftp the result.

I’ve used some interactive tool to extract my test data, debugged my code in Eclipse and just browsed the output with Notepad++. Now I want to construct a flow that either I or any of my teammates can run whenever they want.

Today, I probably brush off my scripting skills, bash or perl or whatever, and write a quick script to run a SQL query, run my java code and then run the file transfer. Easy, right?

Maybe not.

Connecting to the database interactively was so trivial I didn’t even think about it, but doing it in the script is a whole different exercise. I’ve got to get my quotes and escape characters right; I have to make sure I connect without compromising the credentials needed to log in to the database; I even have to figure out what return codes I’m getting and what to do with them. Of course, I need to add some notification so that I’m informed if there was a problem.

OK, got all that done. Now running my Java code is a piece of cake right? Well, it is when running on my machine with my credentials but maybe my teammate has a different environment. Or he’s a .net guy and doesn’t have Java configured the same way as me. Or worse, maybe doesn’t even have it installed. Better add the code to check for all these problems and notification for that, too. And when I get that failure notification, what do I look at to figure out what went wrong? I’d better pipe everything to a log and make sure others can view the log, too, so I make sure I put it on a shared filesystem that others can access.

Let’s not event start on the file transfer! Or, when you have to add additional steps or when you have to run this a 3 a.m. as part of some testing that another group is doing or making sure that database you rely on is up and running, etc., etc., etc.

Eventually, after way more time than I ever thought it would take, that massive script is done. I run it through some test cases and I’m done.

On to the rest of the delivery phases with more testing and probably some more extensions that make the script truly epic.

In those last few stages of getting to production, someone may raise the issue that this new application interacts with some other application that’s managed by some production automation tool. OK, let the rework—or, really, brand new unplanned work—begin. The developers aren’t familiar with the tool and the tool administrators aren’t familiar with the application so you can imagine how well those conversations go. However, they are all professionals and eventually manage to get the job done and the application proceeds to production.

If this long, drawn-out story didn’t already make you uneasy, you may have caught the huge fail that is now just waiting to blow up in our proverbial faces. Most, if not all, the testing was already completed. The new work done at the eleventh hour gets minimal, if any testing, is deployed and—surprise!—has a high failure rate in production.

To add insult to injury, somewhere down the road after all the heroics of getting the application deployed are long forgotten, a failure occurs, as is only natural and to be expected. The poor on-call folks must negotiate through that production tool they aren’t familiar with and isn’t part of their world. And then they encounter that massive, epic script. What’s going on inside of that thing and where did the failure occur? Where are the logs and do we even have all the ones we need? And saving the best for last, after the problem has been identified, how do we restart this process in the middle when half of the processing was already completed?

So What Would Nirvana Look Like?

What if you could create a database query by just coding the SQL statement(s), picking the database you wanted from a list of logical names and requesting notification in case of error and you could express those requirements in a few simple statements? If you want to run it at 3 a.m., you just add that requirement too and then say “run.” The availability of the database is checked, the credentials are secure, there is no scripting, log and output are captured automatically (not just from this run but as many runs as you want, and you can compare one run with another, etc.) and you get whatever notifications you requested. The logic would be solid because it’s been in use for years by organizations like yours, rather than being written under pressure while trying to meet a deadline.

You could write your automation rules (jobs) in some simple notation you were already familiar with such as JSON, store it together with your Java code in git or whatever SCM you use. The jobs could be packaged with the rest of your application and your build, config management and testing tools and could push your entire application including your jobs through the entire delivery pipeline so there are no surprises or additional work required before deploying to production. The facilities to deliver operational visibility, support problem analysis and log and output management would be second to none because they have been evolving with input from community and IT professionals.

Imagine if this magical-sounding solution that implemented Jobs-as-Code was available from github and other public sources? You could download, install and have access to Jobs-as-Code in minutes. You wouldn’t have to submit access requests to IT folks, no purchase requisitions, no messy and complex installations.

You don’t have to imagine. Jobs-as-code solutions already exist. If the tool(s) you’re using doesn’t give you these capabilities, switch! At least start talking to your peers and let them know the better way has arrived. With sufficient demand from the community, the vendors and authors implementing the myriad job management tools may just accommodate you.

— Joe Goldberg

August 7, 2017
Loom Systems Extends Reach of AI into DevOps Realm

One of the things that tends to hold back adoption of DevOps are all the low-level events that need to be correlated by an engineer to generate anything that approaches actionable intelligence (AI). To eliminate that requirement, Loom Systems created a Loom AIops platform to automate many of the tasks that normally would need to be performed by a DevOps engineer.

Loom Systems has upgraded the Loom AIops platform by adding Distributed Entity Tracing, which makes it easier to analyze events across a range of interdependent applications and associated middleware. In addition, Loom has created Correlations View, which simplifies identifying how alerts being generated are related to one another.

Finally, the latest version of AIops adds support for the OpenStack cloud management framework, in addition to sporting a revamped set of dashboards that are easier to navigate.

Dror Mann, vice president of product for Loom Systems, says one of the reasons that DevOps is not widely implemented is that today it requires IT staffs to accumulate and correlate massive amounts of low-level event data to create anything approaching actionable intelligence. Loom AIops is designed to automate the low-level tasks relating to DevOps to enable the IT staff to focus more of their time and effort on fixing issues and reducing the time required to release and update applications.

Loom AIops, says Mann, accomplishes that goal by first streaming all the application log data into its platform, followed by all the log data generated by any dependencies that application might have. Loom AIops then applies machine learning algorithms in real time to identify the root cause of an issue. Armed with that information, it becomes a lot easier to reduce the mean time to resolution (MTR), which then provides the IT organization with the time needed to address more complex management challenges, he says.

Mann notes the single biggest challenge IT organizations encounter with DevOps is that it requires them to change their modus operandi. By automating many of the lower-level DevOps tasks, there’s more time available for IT organizations to think through the cultural changes required to successfully implemented integrated DevOps processes.

Most IT professionals will view the application of AI to IT operations with a combination of skepticism and trepidation. It will be difficult initially for many IT professionals to trust the analysis being generated by AI software. Once they do begin to trust that analysis, there naturally will be some concerns over to what degree an AI program can eliminate the need for a DevOps engineer altogether. But, Mann notes, there are many complex tasks that never get addressed because most DevOps engineers spend an inordinate amount of time and effort trying to correlate the right data.

At this juncture, usage of AI within the context of DevOps is now all but inevitable. The challenge now is not figuring out the line between where humans add higher orders of value compared to machines that are generally going to be limited to identifying patterns. The advantage a machine has it that once it learns something, it doesn’t forget it, take days off or switch jobs. However, being able to identify a pattern is not nearly the same as being able to apply reasoning to what the sum total of those patterns really means for the business.

— Mike Vizard

August 4, 2017
4 Ways to Give DevOps Teams the Support They Need

If it weren’t hard enough for IT infrastructure teams to innovate and keep the lights on, a host of technologies designed to drive efficiencies are introducing new levels of complexity across the data center. Virtualization, cloud computing, software-defined storage and digital are making it much more difficult for the infrastructure team to control, manage and administer an organization’s IT.

Just when IT professionals think they’ve figured out the support model, DevOps steps in. As the number of apps continues to explode, more organizations are adopting a DevOps model to accelerate development efforts. A recent Coleman Parkes Research survey of 1,770 senior business and IT executives revealed 87 percent have adopted DevOps.

Unsurprisingly, the growing emphasis on DevOps is placing an added burden on infrastructure teams already struggling to cope with the digital transformation of their organizations. Traditional storage systems are often unable to match DevOps’ requirements for an enterprise infrastructure that is simple, flexible and automated. Some DevOps teams also work in silos, which produces an additional level of complexity and opportunities for failure.

Also, occasional tensions between development and IT operations can flare up over different approaches to infrastructure and competition for resources.

IT operations teams with traditional storage hardware backgrounds are normally focused on accelerated release cycles, availability, reliability and streamlined data management. Developers, on the other hand, spend little time thinking about what technology they need versus keeping test data up to date with production and having easy access to capacity when needed.

A Support Structure That Works

Set against this backdrop, choosing the right platform to underpin these growing changes within the enterprise cloud can play a big part in ensuring a DevOps model that works and makes life easier for the IT infrastructure team. Here are four ways to help make the enterprise cloud infrastructure fit a DevOps model:

Copy Data Management

Copy data management provides up-to-date, “virtual” copies of production data to the DevOps team, eliminating the need for physical data duplication. Copy data management helps remove the load on production storage and protects the performance of applications. With snapshots, cloning and replication, modern enterprise cloud systems can enable data copies. The same technologies make it possible to protect and manage the development side of the environment in case of failure or data corruption.

QoS and Performance Guarantees

Quality of service (QoS) and guaranteed performance is a significant concern when adopting DevOps. QoS enables IT to allocate different storage workloads to control performance. QoS can set levels of maximum and minimum input/output operations per second (IOPS) or bandwidth. IT can also configure QoS at the virtual machine (VM) or virtual disk (vDisk) level. With the power of all-flash, DevOps can be consolidated on the same platform as production. This makes it easier to access production data sets and reduce the storage footprint.

Monitoring and Troubleshooting

The ability to monitor the infrastructure and correct problems and misconfigurations rapidly is valuable when moving to a DevOps environment with continuous integration and continuous deployment. Ideally, the foundations of an enterprise cloud platform should provide basic and advanced monitoring capabilities, predictive analytics, integration with other monitoring tools and VM-level monitoring.

Automation

From an infrastructure point of view, it is also critical to provide the capability to provision and manage IT infrastructure programmatically. Automation enables organizations to create and break down environments as needed, incorporate advanced features such as snapshots and cloning as daily workflows, and eliminate the potential for error from manual or interactive configuration.

As organizations move relentlessly to a mixture of virtual and physical, on-premises and cloud-based IT, infrastructure teams have to manage, control and administer all those resources while matching them to a diverse range of business, development and user requirements. By implementing an enterprise cloud platform that can work across physical and virtual environments, infrastructure teams can help DevOps teams achieve their goals without constraints around performance, scalability, manageability, resilience or flexibility.

IT organizations are achieving this level of support by replicating the agility of public cloud within the data center to make it easier to manage enterprise and cloud-native applications. This approach can help support the DevOps model and ensure it is implemented with as little friction and disruption to the infrastructure as possible.

About the Author / Chuck Dubuque

Chuck Dubuque is the VP, Product Marketing at Tintri. Prior to Tintri, he was at Red Hat where, as director of product marketing for the virtualization business unit, he was responsible for Red Hat Enterprise Virtualization (RHEV) and Red Hat Enterprise Linux OpenStack Platform (RHEL-OSP) market strategy. Before Red Hat, Chuck worked at a value-added reseller for VMware, NetApp and Cisco and also spent several years in biotechnology in marketing and business development. Follow him on Twitter.

July 31, 2017
Fighting Deployment Anxiety: 5 Tips from the Pros

You know the feeling: You’ve put in hours of work developing a new feature. It has gone through QA, been tested and passed all stakeholder inspections, and now you’re ready to deploy. Despite your experience and preparation, you have a pit in your stomach. What if something goes wrong and it brings your app to its knees? What if it doesn’t function in your production environment the way it’s supposed to? We’ve all been there. Fortunately, we also have some ways to help.

Electric Cloud CTO Anders Wallgren recently participated on a webinar with Gene Kim, CTO, researcher and co-author of “The Phoenix Project” and “The DevOps Handbook.” Hosted by devopsy.kinsta.cloud Editor in Chief Alan Shimel, the webinar addresses lessons learn from large-scale DevOps transformations, the business value of DevOps and some of the best tips and tricks for fighting deployment anxiety. We pulled out the top five tips and tricks from Kim and Wallgren for making deployment worries a thing of the past.

Tip 1: Automate Your Deployment

A theme throughout the webinar from both Kim and Wallgren was “automation.” One of the big reasons for this is that typically, when you automate your deployments, you’re also automating the diagnosis of problems in those deployments. You can’t always just rely on the exit code of a command-line tool to tell you whether something went right or went wrong. To help keep your deployments healthy, you’ll want to do testing along the way, possibly as a blue-green deployment or a partial deployment. However, if you don’t have automation behind that, it can be difficult to get that accomplished at speed. Without automation, you may start to rely on heroic efforts by individuals—so that every time you are doing a deployment, someone is having to do a figurative tightrope-walk to get everything accomplished. This creates a tough environment for your employees, and can lead to extensive downtime (and lost profits).

Tip 2: Enable Self-Service Deployments to Any Infrastructure

You’re probably wondering why this is a separate topic from automated deployments. That’s because the majority of deployments that occur in a given release pipeline aren’t going into production. Most deployments end up in testing environments (for performance testing, regression testing or just developer testing). It’s also important that you give your engineers the ability to use the same deployment methodology (artifacts, process definition, environment definition, etc.) regardless of the location to where they are deploying. From a cost perspective, allowing QA to test on a local Kubernetes cluster might be more cost-effective for your enterprise than running tests on Amazon Container Service. But, allowing teams to use the same approved automation as part of every deployment—to any location—means a more thoroughly rehearsed and therefore less anxiety-inducing deployment.

The ability to do self-service deployments also gives your teams the ability to move substantially faster when they need to deploy something for testing by avoiding the need to open a ticket and get your Ops teams involved. By allowing self-service deployments, you can avoid the process of running around to find the right person to accomplish the deployment you need to run, and empower your organization to react and build more autonomously across different groups. The result is better-tested and built deployments (resulting in lower deployment stress and fewer errors).

Tip 3: Self-Service Environments

We’re continuing to roll with the theme of automation, but this time we’re talking about self-service environments. More often than not, we’re working with complex distributed apps. This means things such as a containerized microservice application, DBMSs, messaging, key value stores, legacy applications and third-party vendors that we need to integrate with—all things that make our deployments complicated. Being able to consistently provision complex environments is a very important part of being able to successfully recreate a deployment as you promote the application through its life cycle.

It’s important to make sure that your engineers, and in particular, your QA and development engineers, have easy access to environments and that they’re not standing around waiting for access. When you don’t have the ability to get VMs or containers provisioned and configured correctly to receive your application, you run into problems. One of these problems is that your ability to test rapidly just became much more difficult. When you can’t get a near-production quality environment stood up in a few minutes, it means that your bug fix that requires just a few lines of code could potentially take weeks to test. Suddenly you need to requisition a set of VMs to test on (which could take weeks) and another day or two to do the deployment, and only then can you complete your tests. Things become painfully slow. And, when it comes to testing and fixing bugs, slow equals stress.

Tip 4: Artifact Repositories

Want to feel more confident about your releases? Look at your artifact repositories. Use of a shared artifact repository in production environments has been shown to be a good are a really good predictor of deployment success. If you’re not versioning your binaries, and have little insight into where binaries are coming from and how they were built, it can lead to failed deployments.

It may not be easy or simple to get your artifact repositories in line, but it’s a labor that will pay massive dividends in the long term. Electric Cloud worked with a massive online mortgage company that had the challenge of deploying large artifacts (gigabytes) into multiple data centers. To get everything up to speed, it was necessary to transfer those artifacts out of their build and test infrastructure and into their pre-production and production infrastructure. Because of the size of the artifacts and concerns about worldwide availability for the files, Electric Cloud worked with this financial services company to use Amazon S3 as the backing store for the repository server to create a long-term solution that is flexible and scalable.

Tip 5: Security and Auditability

If you work in a regulated industry, already know the heartburn auditability and governance can cause. Security is one of—if not the single—most important aspect of many application delivery pipelines. The reality is, we are just beginning to scratch the surface of the problems we will see the in the future when it comes to infrastructures getting hacked. So what’s our tip? Again, it comes back to automation. At ElectricCloud, there is a saying: Automation is auditing, automation is documentation.

The idea behind this is that by automating your deployment tasks, you are essentially creating a very real and tangible paper trail that shows what you intend to do (the automation definition) and a report of exactly what happened (the automation logs). When you rely on manual deployments, you suddenly take an aspect of repeatability and predictability out of your workflow. At that point, you are forced to report based on people’s spreadsheets, emails and memories. This is one of those things that doesn’t become a big deal until it’s a really big deal, and—to borrow another saying—an ounce of prevention is worth a pound of cure. In the case of auditability and security, being able to retrace your steps because your steps are always the same is a fairly foolproof method to making your audits much more smooth.

To provide a real-world example for how automation can make auditing and security much less painful, one of ElectricCloud’s clients is an aerospace customer with very stringent and demanding security standards. In the past, at the end of each release, it would have a two-week process where everybody had to sit down and compile a list of every single check-in deployed as part of that release. They had to understand where it got built and exactly how it was tested, and then assemble all of that information in a report. It was an extremely time-consuming, but seemingly necessary process. However, by automating many of the steps in the pipeline, the audit response effort went from a two-week process to just a few hours. They accomplished this because 99 percent of the information they needed was in the automation system and was simply scraped as part of process. The task went from manually pulling that data to just getting into it consumable formats.

Summary

By now you’ve certainly seen a theme emerge in our recommendations for reducing the stress of deployments—namely, automate! By putting effort into automation and streamlining your pipeline, you can take what was once a very labor-intensive and sometimes stomach-churning process and make it much more enjoyable. Not only will this save you time and money in the long run, but it will also allow you to focus on building better software, doing more deployments and focusing on the tasks that bring your business and your customers real value.

For more details on how you can relieve your deployment anxiety, and to hear the rest of the discussion focusing on the lessons learned from large-scale enterprise DevOps transformations and the business value of DevOps, watch the full replay.

About the Author / Sam Fell

Sam Fell is the Head of Marketing at Electric Cloud, responsible for driving awareness and demand for our DevOps Release Automation and Acceleration solutions. For 20 years, Sam has helped companies achieve success by marketing, selling, and implementing DevOps Release Automation, IT Change Management and Application Performance Management solutions. Prior to joining Electric Cloud, Sam held leadership roles at BMC, CA, Marimba and Nimsoft. Follow him on Twitter and connect with him on LinkedIn.

July 25, 2017
UBI: Facing the Reality in the Era of Automation

Those of us working in DevOps make a good living. According to Glassdoor, the national average salary for a DevOps engineer in the United States is $100,000. Are we worth it? Of course we are; DevOps is hard work. The value we add to an organization more than justifies the salaries we can command. Nobody is giving us the money. We have to earn it every day.

Considering that high school teachers earn on average $48,000, a restaurant manager averages $46,000—about the same as a paralegal—and a forklift driver makes $26,000, DevOps engineers are doing A-OK.

That’s the good news. The bad news is that our average salary, combined with our commercial isolation from others, puts us in a bubble. I am the first to admit that I live in that bubble. I don’t interact with a lot of paralegals, restaurant managers and forklift drivers in my day-to-day work. Most of my interactions are with people who do what I do and live the lifestyle I live. It’s nice to live in the bubble; I’ve done countless all-nighters to be here. But I am very aware that it’s a place that few people live. Keep in mind, the average income in the United States is $51,272.

Here are some other DevOps Chats you may want to listen to:

Automation’s Impact on a Structured Life

Culture Change: Where It Matters in DevOps

An Unanticipated Obstacle to Full Automation

It takes a lot to get into our bubble. You need to be really smart and you need to be well-educated. Most programmers have a college degree. Yes, there are few of us that are part of the 70 percent of the population that does not have a college degree, but those people are rare. And, for as long we keep delivering and learning the new stuff that comes down the pike, we’ll be living in the bubble for the foreseeable future.

However, for those outside our bubble, it’s going to be a different story. As the evolving landscape reveals, those who do predictable, repetitive work are going to be automated away. And, most of the jobs that do remain are going to be on the lower end of the income scale.

So, what’s to become of those outside the bubble? How will they survive?

One solution that has been bantered about is Universal Basic Income (UBI). UBI is income that you get by virtue of being a citizen of state. There are no strings attached. Every month a deposit appears in your checking account and you are free to do with it as you wish. If your refrigerator is empty, you buy food. Should you earn a good salary, you can give the money to a favored charity or buy your kid a new video game. As I said, it’s no strings attached. The intention is to provide financial security to the citizenry without means testing.

UBI has been in play in Alaska since 1976 as implemented under the Alaska Permanent Fund. Every permanent resident of Alaska gets money from the state just because they live there. In 2014 every resident of Alaska got $1,884 from the fund.

Other governments are trying or have tried out UBI. Finland has a limited experimental program that pays €560 ($630) a month to participants. Cyprus pays out €480 a month ($546) to qualifying citizens. Canada had an experimental program back in 1974 that ran for four years in Dauphin, Manitoba. The program was called Mincome. It dispersed $1,200 a year to people living below the poverty line.

UBI has it supporters. Former U.S. president Richard Nixon supported it for a while. Milton Friedman supported it via the negative income tax. And tech leaders including Mark Zuckerberg and Elon Musk, men who have a keen understanding of the impact of automation on human employment, are proponents.

I am a supporter, too, if for no other reason that I believe in the promise of technology—that we will automate our way to a life in which we are free to do the work that brings meaning and purpose rather than going through the motions of existence, laboring away to put bread on the table.

But, for as much as I think UBI is valuable, there is a part of me that finds the notion to be a farce. It’s not so much because I think people will take the money and go down to a riverboat casino to gamble away their monthly allocations while the kids go hungry, but rather because the dollar amounts being considered for allocation are a joke.

Let’s do the math.

The table that follows illustrates how much income is required to support a family of two adults and one child on state-by-state basis. This data is provided using the MIT Living Wage Calculator by way of Mental Floss.

Table 1: A listing of minimum income by select states

State Minimum Income per Year Per Month

West Virginia $45K $3.75K

Iowa $49K $4.09K

Montana $49K $4.08K

Tennessee $47K $3.91K

Alaska $54K $4.50K

The states I choose—West Virginia, Iowa, Montana, Tennessee and Alaska—don’t contain any big cities on the order of New York, Los Angeles, Houston or Chicago. Let’s just say they’re states not subject to a large amount of urban impact. These are states that I can image might find UBI a useful program, particularly as industries—manufacturing, agriculture and energy—become more automated and the employed workforce experiences more displacement due to automation.

As you can see in Table 1, according the the MIT Living Wage Calculator, it will cost a family of two adults and one child residing in any of these states about $4,000 a month to live at a comfortable minimum.

Let’s apply some UBI numbers. As mentioned above, in 2014 the Alaska Permanent Fund distributed $1,884 a year to each permanent resident in the state. That comes out to a monthly allocation of $157. Compare that $157 a month to the $2,250 each working adult in the Alaska household has to earn to achieve a modest standard of living. That $157 is 7 percent of the amount needed. It’s a joke. The reality is that the allocation from the Alaska Permanent Fund will buy about a week’s worth of groceries, if that.

On the other hand, there is Cyprus. As mentioned above, that country is paying out $546 per person in guaranteed income, which is an OK allotment given that the rent on a three-room apartment is about $500 a month and a McDonald’s Combo Meal goes for around $7. Multiply the per-person allocation of $546 by two adults and you have a situation where life is sustainable. But then again, the population of Cyprus is 1.2 million people, a little more than the size of the population of Montana. Still, the Cyrus UBI amount is a viable number. But, what about here in the United States? If we were to implement UBI, what is the dollar amount each person is to be given?

Let’s go back to Table 1. For the residents in those states listed to enjoy a standard of financial security on the order of Cyprus—one in which no matter what, you will have your basic needs met—it seems that we will need to come up with a UBI allocation of about $20,000 a year per adult. Of course, this is assuming that the person does absolutely nothing in the way of earning money beyond the UBI allocation. However, as was reported in the study of the Mincome Project in Dauphin, Canada, once UBI kicked in, only two segments of the workforce worked less—new mothers and teenagers.

Granted, this is very limited study of a very small sample of people from over 40 years ago. But, still the inference may be valid. Also, as we’ve found from people in retirement or forced unemployment, after a period of welcome inactivity, at some point we need something to do. Thus, there is a good chance that those on UBI will want to earn money beyond the allotment. Given that we can project that most people on UBI will want to engage in income-producing work, if such work is available, then maybe that $20,000 a year number is excessive. Maybe a reasonable dollar amount for UBI is between $12,000-$15,000 per year, per person. That number could work in less-expensive areas of the country.

How would we pay for it all? Honestly, at this point, I don’t know. Maybe the same way we paid to put a man on the moon? To quote the old saying, “Where there is a will, there’s a way.”

Sadly though, no matter what the allocation number is and no matter what the consequences of not doing UBI may be, the will to do UBI currently is limited to a bill before the Hawaiian Legislature. The general will is not there for a variety of reasons: politics, moral distaste manifested as a fear of Netflix-induced sloth in the general population, or just plain greed. The notion of having enough money show up in a citizen’s checking account to ensure a modicum of financial security has limited appeal.

I can accept such reluctance. UBI is far afield from where we are culturally or where we have been historically. But, what I cannot accept is formulating a UBI program that is dumb; one in which we create stories that say financial security can be had for $157 a month. UBI allocation needs to be proposed in real, livable numbers. Otherwise, the risk we run is that when we can somehow sell the idea to a willing buyer, should UBI fail due the stupidity of not allocating enough money per recipient, we will be hard-pressed to get a second chance to do it right.

The notion that the impact of automation on human employment will be severe is a tough one for many to accept. While it is true that history shows that advances in technology have led to new forms of employment for those displaced, history does not accommodate thinking machines. Thinking machines are new to the landscape.

Just as automobiles replaced horses, maybe thinking machines will replace humans. Horses did not find new forms of employment when the automobile came along. Rather, their number diminished to levels required for their remaining purpose: providing entertainment. Such an analogy might sound harsh. After all, a human life is worth more than its commercial value. If this is the case, then unless humanity can implement ideas such as UBI as a way to sustain life independent of commerce, the alternative might very well be about the bubble—a place where those on the inside live well, in secure isolation from the less fortunate ones suffering on the outside.

— Bob Reselman

July 18, 2017
When Automation Predictions Come To Be: Transportation as a Service

Back in December of 2016 I wrote an article for devopsy.kinsta.cloud titled, “What Do We Do When Everything is Automated?” in which I made a prediction:

Also, think about this: As Uber and Lyft mature in the business landscape and self-driving cars become commonplace, the ride-share companies will replace human-driven vehicles with self-driving vehicles. Then, Uber or Lyft will start to offer a pricing model that is a flat fee for 24/7 service with unlimited mileage.”

Pretty brazen prediction, huh? Well not really. I thought and still do think that such a situation is inevitable. You just have to think things through to arrive at the conclusion I made above. However, my thinking was that, until yesterday, this sort of inevitability is about five years out. Then this news flash come in from Bloomberg: “Apple Is Working With Hertz to Manage Its Self-Driving Car Fleet”

This caught my eye. Hertz stock went up 14 percent that day. Upon further inspection, I learned that Apple is leasing a small fleet of cars from Hertz to test self-driving vehicles. Turns out the Alphabet Inc., a.k.a., Google, has a similar deal with Avis Budget Group Inc., a competitor to Hertz in the car rental space. Now granted, I thought that Uber or Lyft would extend the scope of their activity by working directly with automobile manufacturers making self-driving vehicles. That the service provider might be a current automobile rental company was a bit unanticipated. But, as I explored further, it gets interesting.

I turns out that Hertz previously was owned by Ford Motor Company, which sold the company to a private equity group in 2005 for $15 billion, a tidy sum. Hertz acquired Dollar Thrifty in 2012. Avis Budget Group, a competitor to Hertz, owns Zipcar as well as the New Zealand rental company, Apex Car Rental and Maggiore Group, the fourth largest car rental company in Italy. There’s a lot going on in the auto rental space.

As I reflect upon events, things become clearer. It makes sense that those non-automotive companies working to create driverless vehicles are attracted to car rental companies. Car rental companies are not in the automobile business. They are in the transportation service business. Few people go to Hertz to rent a Chevy. They go to rent a way to get around. If a customer shows up at the Hertz counter only to find out that there is no Chevy to be had, most people will take the Nissan alternative. They don’t care about the car. They care about the service the car provides.

Now imagine this: Let’s say you are a vehicle manufacturer, new to the marketplace. You finally figure out how to make a driverless vehicle that satisfies the needs of consumers under most driving conditions. Given current trends, when it comes time to make a profit from your investment, are you going to open showrooms with the hope of selling your new automotive brand to consumers? Or, are you going to buy an existing company that has a proven record success and infrastructure for providing transportation services on demand to the masses? In other words, what’s easier to imagine, Apple setting up showrooms throughout the world to sell autonomous vehicles to consumers accustomed to buying Ford, Toyota, Nissan or Tesla? Or, rather, Apple going directly into the autonomous transportation-on-demand business by buying Hertz? Or maybe Hertz will just buy driverless cars from Apple. (Hertz buying from Apple is hard to imagine; Apple is a company that likes to think big. Selling cars to Hertz is thinking small.)

My take? Well, two things: First, as autonomous vehicle technology matures, transportation as a service is going to grow on the landscape. Vehicle manufacturers need to survive. They will adapt. How the actual selling of vehicles fits into it all is still working itself out. I think we’re going to see a lot of vehicles being made to provider of service, not as a property to own. Second, maybe it’s time to buy some stock in Hertz, maybe Avis too, for that matter.

What do you think?

— Bob Reselman

June 29, 2017
Fleetsmith Pulls Apple into DevOps Orbit

For too long now Apple Macintosh systems have been the proverbial red-headed stepchild of DevOps. There are a lot of ways to incorporate Windows systems automatically into a larger DevOps pipeline, but Macintosh systems generally require a lot of manual intervention.

As part of a larger effort to automate application deployments across Apple systems, Fleetsmith has announced that the platform it created for automating the management of Apple systems now supports both the Puppet and Chef automation frameworks. DevOps teams looking to extend the reach of those frameworks to Apple systems can now invoke Fleetsmith to automate the deployment of Chef and Puppet like any other Macintosh application.

Of course, once Puppet or Chef are installed, IT organizations can control what gets deployed on those systems going forward. But in general, organizations that deploy Macintoshes prefer to employ tools that provide a native Macintosh experience when it comes to managing those systems, says Fleetsmith CEO Zack Blum. There are plenty of options when it comes to cross-platform management tools, but Blum notes each of the tools compromises the Apple Macintosh experience such that individuals who prefer Apple systems must manage those systems using a construct that was not primarily designed for use in, for example, a Windows environment.

Over the years, a number of developers have shown a marked preference for Apple Macintosh systems. As a result, it’s difficult to develop a coherent approach to DevOps that does not include some way to automate the management of those platforms, including the ability to deploy DevOps tools such as Puppet and Chef. Blum says that, in time, Fleetsmith plans to increase the number of Macintosh applications that can be distributed via its platform.

Naturally, Apple is too focused on the end-user experience to trifle with IT management issues. It prefers to leave those issues in the hands of partners such as Fleetsmith, JAMF Software, IBM and SAP. The other alternative is to rely on open-source software that has been created to address this same issue. But Blum says open-source alternatives are more complex than commercial software, while rival commercial approaches to provide the same level of fleet management of Apple system at scale.

Fleetsmith was developed when its founders worked at Dropbox and Fandom. Both organizations had evaluated several Mac management options before deciding to develop their own code. The company launched last fall with $3.1 million in funding.

Pricing for the service, which officially is still in beta, is $10 per month per device. The service is free for any organization with fewer than 10 Apple devices. In addition, Fleetsmith integrates with Google G-Suite to make it simple to authenticate users of the service.

Obviously, there’s more to be done in terms of pulling Apple devices into the greater DevOps ecosystem. But as a first step, making it simpler to deploy Chef and Puppet on these systems signals greater awareness of the overall issue.

— Mike Vizard

June 26, 2017
Humans: The Ultimate Exception Handlers. Until …

Back at the turn of the century, I had a job working in the IT department of a Big Bank. The section I worked in was Mortgage Exceptions. My group’s task was to route exceptions that could not be addressed by the mortgage processing system to a human being for resolution.

Most of the intelligence required to execute the Big Bank’s mortgage business processes was automated by the mortgage system. Each month the system processed millions of mortgage payments. The systems sent out the bills for payment. And, when a payment came in, a human operator entered the check payment into the system. (This was before the days when direct withdrawal was the norm.) Most people paid their bills. So, for the most part, the system chugged along without a hitch. But every so often there was a problem the system could not handle—an exception, if you will.

Examples of exceptions were accounts that were significantly past due, account numbers that were unknown to the system, customer complaints and bounced checks. There was a whole array of stuff that could—and did—go wrong that the software could not handle. Thus, the exception ended up in front of a human being. The human was the ultimate exception handler.

I worked on that system more than 15 years ago. I think it’s safe to say that today the bank’s system intelligence is a lot smarter. I imagine that today the IT group I worked in is more concerned with writing more robust, automated exception-handling than routing problems to humans.

That’s the way technology goes. Systems get smarter and, in many cases, become better than humans when it comes to doing work that is well-known and repetitive. The more the machine can do, the more the machine will do. Businesses like machines—always have, always will. Automation is a good investment.

Those of you who have been following my writing on devopsy.kinsta.cloud know that I am very concerned with the impact of automation on human employment. I saw the writing on the wall when I read the Ball State study. As time goes on, more machines will be doing more work. Less human labor will be required. What will those displaced humans do?

Many believe that automation will create new jobs for humans. In some cases, it will. As I said earlier, I am sure the people who were writing the routing code at the Big Bank I worked at years ago are now writing more robust exception handlers that require less human interaction for resolution. Those few developers writing the exception handlers are probably doing just fine. The developers who were writing simple if-then rules, with no ambition otherwise, are probably gone. Also, I’ll bet the the Big Bank has embraced the DevOps sensibility, which probably has resulted in those with lesser talent in terms of implementing automation being shown the door. As for those humans who were entering checks into the system and turning delinquencies over to collection agencies, their number is probably now a handful, if any. Hopefully, the poor soul who was hired as a teller out of high school and worked her way into mortgage exception resolution adapted, despite the odds.

As I said previously, many believe that automation will create new jobs. There always will be new things to do or things that need to be done that machines can’t do. Fair enough. Still, for me it’s not about new jobs being created. It’s about the number of new jobs that will be created.

Many of the new jobs will require a lot more smarts than did the jobs they are replacing. And my gut tells me that, while there will be new jobs, there will be fewer than the number of old jobs eliminated. And these new jobs will fall into two categories: “Cool New Work” and “Doing Work the Machines Have Yet to Master.” The “Cool New Work” is just that. “Doing Work the Machines Have Yet to Master” is handling exceptions, the work I used to route to humans back at the Big Bank. Except this time, as systems become smarter, many of the exceptions these systems throw will become a lot more difficult for humans to resolve. The systems will throw fewer errors, but when they do, watch out!

Exception-handling has been in the realm of advanced human activity since the days when the ancient Mayans needed to figure out a way to build pyramids that didn’t fall down. A lot of buildings fell down before a few stood up. Eventually, they figured it out. Exceptions gave way to learning, and today we have the art and science of architecture. The industry has matured. Today, constructing a building is more about process and technology than figuring out what went wrong with the last building.

So, too, will it be with automation exceptions. Just as over time fewer pyramids collapsed, with automated systems fewer exceptions will be raised and fewer humans will be needed to figure out what went wrong. Then, we’ll be back to where we seem to be going: more machines doing more general work, a smaller number of very intelligent people doing more advanced work, and the rest of us doing the tasks the machines can’t do yet—aka, the exceptions. We’ll be working on exceptions until the time that the thinking machines will be able to resolve those exceptions. Then what? Dunno.

Now, don’t get me wrong. Technology is good; automation is good. There’s more to life than spending most of one’s waking hours entering check deposits into a mortgage system. Humans were meant for more. But, when earning a living means being really smart or being just one step ahead of a machine, it’s going to be a real challenge for the Average Joe to the find opportunities to earn a buck, let alone find a job. How will we meet this challenge? This I leave for you to determine.

— Bob Reselman

June 21, 2017