devopsy.kinsta.cloud

Tag: automation

  • Oracle Advances AI Effort to Automate DevOps

    Oracle Advances AI Effort to Automate DevOps

    Oracle today announced the general availability of an expanded Oracle Cloud Platform service that promises to automate most back-end DevOps processes by relying on a combination of machine learning algorithms and other forms of artificial intelligence (AI).

    The Oracle platform-as-a-service (PaaS) environment has also been expanded to include Oracle Mobile Cloud Enterprise, Oracle Data Integration Platform Cloud and Oracle API Platform Cloud alongside existing continuous integration and continuous deployment (CI/CD) cloud service. Oracle also revealed today that later this year it will also add blockchain, security and management and support for additional database workloads, including OLTP and NoSQL, that expand on existing Oracle Database, Oracle Autonomous Data Warehouse Cloud, Oracle Analytics Cloud, Oracle Integration Cloud and Oracle Visual Builder services.

    Siddhartha Agarwal, vice president of product management and strategy at Oracle, said via the cloud service, Oracle plans to automate DevOps processes to the point where organizations will be able to allocate most of their time and effort to building applications. Today, developers are spending as much as 50 percent to 60 percent of their time on DevOps processes rather than writing application code, he said, and the Oracle Cloud platform automates a broad range of operational functions including tuning, patching, backups and upgrades.

    Oracle Cloud Platform is designed to be an infinitely elastic autonomous cloud service that can both anticipate application requirements and dynamically adjust to changing workload conditions. Over time, the platform also will be able to increasingly provide recommendations based on the characteristics of the application workloads being built and deployed, Agarwal said.

    The Oracle API Platform Cloud service, for example, already continuously learns about usage patterns to recommend allocation limits, configurations and policies using predictive algorithms, he said.

    Meanwhile, the Oracle Cloud Mobile Enterprise service provides access to machine learning algorithms that make it much easier to incorporate chatbots into mobile applications, Agarwal said. Machine learning algorithms are now also relied on to create Big Data lakes and the data pipelines needed to make that data available to application developers.

    Those services are all extensions of an Oracle Developer Cloud service for building applications that supports build frameworks such as Maven, Ant, Gradle, npm, Grunt, Gulp and Bower, as well as command line interfaces for Docker, Kubernetes and Terraform. Frameworks such as JUnit and Selenium for both logic and UI testing are also included.

    Agarwal said one of the primary reasons DevOps process have not been as widely adopted across the enterprise is that it’s simply too complicated for most IT organizations to set up and then continuously manage all the underlying infrastructure required. By moving more of the responsibility for the operations side of the DevOps equation on to an autonomous cloud service managed by Oracle, he said, it becomes simpler for enterprise IT organizations to focus their energy on application development processes.

    It may take some time for enterprise IT organizations to get comfortable with ceding that level of control to a cloud service provider. But whatever DevOps path they choose, it’s apparent that machine learning algorithms coupled with other AI advances are about to automate broad swaths of the DevOps universe.

    — Mike Vizard

  • Don’t Forget to Automate Cloud Security

    Don’t Forget to Automate Cloud Security

    Cloud has been around long enough, and we’ve had enough breaches stemming from unsecured cloud (mostly data objects, but other items, too) that we should know better. I cringe whenever I see a security headline along the lines of “… Unsecured S3 Bucket.” Would you publicly expose a server in your data center and not lock it down? No.

    So, what is the problem? Frankly, “cloud” now consists of so many separate-but-integrated offerings that it can be daunting, even improbable, for security staff to lock it down. Add in dozens or even hundreds of instances/storage locations/routers, and you have a separate data center with different requirements—at least, that’s what it essentially is for your security team. Then add agile and DevOps, spinning up new infrastructure and changing the parts at a rapid pace, and you have a fast-moving, complex data center environment. Yet, if you are a typical organization, your security staffing hasn’t significantly changed, nor do you have requirements for securing the physical infrastructure. In short, same group of individuals, a ton more work to do.

    Before it was purchased by VMware, I had the pleasure of working with CloudCoreo, and truly liked what I saw. The problem in highly automated environments, be they DevOps or just automated, is that a lot of parts are moving as the system is maintained/updated. Some of those parts are changing security settings in a complex (data center-size complex, in many cases) environment. For security teams to keep up and be part of DevOps—or just keep up with automation—quite often corners must be cut.

    That’s bad for all of us—security staff, the company and customers. If anything, when the cloud environment grows complex, security needs to be more cautious, not less. And yet, the number of data leaks/exploits that involve “misconfigured” security settings in the cloud says we have a problem.

    So what to do? One option is to check out security control and auditing software from the likes of RightScale, Dome9 or Evident.io. (DISCLAIMER: I have not used any of these tools. They were chosen because each shares attributes with CloudCoreo, which I have used.)

    The ability to scan and apply policy to a broad array of cloud artifacts—from users to back-end storage—is essential in allowing security to keep up with DevOps. There is just too much change in a fully automated cloud-based environment to expect security to keep up without the same level of tools as developers and ops get. “Shift left” doesn’t stop misconfigurations, but scanning can detect them, and some tools can even auto-fix those misconfigurations.

    In the end, protecting systems and data is what the security team does. Bringing them along on the DevOps grand tour and offering them the same type of automation opportunities as the rest of IT is benefiting from is mandatory. It is not “would be nice to have” to give security the chance to keep up; it should be a requirement.

    And you might still have rogue S3 buckets that security doesn’t know about that might be misconfigured, but any that are paid for by the corporation will be available to security and those automated tools and, thus, will be locked down.

    And if security isn’t enough of a reason for you to want to implement one of these tools, “Come for the compliance, stay for the security.” Most solutions offer auditing that helps with compliance.

    — Don Macvittie

  • Coming Apart in the Age of Automation

    Coming Apart in the Age of Automation

    One of the fundamental tenets of DevOps is that working in closed, departmental silos is counterproductive for creating quality software. Rather, work is done best when groups are composed of members from across the business working openly and cooperatively. Experience shows time and time again that open, multifaceted teams made up of a variety of viewpoints work better than ones that are closed and monolithic.

    Yet, when we move beyond the landscape of DevOps into American society, things look different. In his book, “Coming Apart,” published in 2012, author and political scientist Charles Murray makes the following point: Sixty-three percent of the population of the United States—white Americans—is separating into a two-class structure in which one class enjoys an increasingly stable, healthy, knowledge-driven prosperity while the other experiences declining wealth, health and educational achievement accompanied by a loosening social cohesion. This loss of social cohesion perpetuates the continuing decline. And, there is no relief in sight.

    Murray points out that while class distinction in the United States is nothing new, what is different now is that these two classes exist in silos that are closed and disconnected. While in the these past classes have intermingled, now each lives apart from one another—so much so that the classes have become almost unidentifiable to each other.

    For example, back around 1960, it was entirely conceivable that a CEO of a manufacturing business shared many of the same experiences as the less-wealthy employees of the company. They might go to the same restaurants, watch the same TV shows and maybe even drive the same make of car. Their children might attend the same schools. Today, however, it’s each to his own; shared experience among classes is becoming more rare.

    There definitely was a class system in force onboard the ocean liner Titanic. The well-to-do traveled first class, while the poor went down to steerage. But everybody boarded together, and Leonardo DiCaprio’s pauper character had a slight chance of interacting with Kate Winslet’s wealthy socialite—maybe even fall in love and get and stay married. And, when the ship went down, all aboard were at risk, rich and poor alike.

    Today, the average Joe flies in a cramped seat in economy. The CEO flies private. The economy flier probably has no idea what it’s like to board a private plane, let alone fly in one. The thought of packing a lunch beforehand to save money while traveling is completely foreign to the CEO. Each lives in a world the other can’t imagine. Again, this trend is not reversing.

    What Does This Have to do with Automation and DevOps?

    Tech culture is a prime example of this exclusive segmentation and clustering according to class. The average salary of a DevOps engineer is ~$138,000 a year, which puts the profession in or around the top 10 percent of wage earners. And, just getting hired at most tech companies requires a college degree. This is a self-selecting dynamic. Birds of a feather fly together: People who make six figures tend to hang out with others who make six figures. Likewise, people with college degrees tend to hang out with others who have college degrees. If you’re in tech, chances are you do not know many people in your day-to-day life who had difficulty in school, have worked on a factory floor or smoke cigarettes. When you and those around you make money by thinking, it’s difficult to have empathy for those whose muscles ache after spending each day stocking shelves at Walmart. You just can’t imagine it.

    Intrinsically, such homogeneity is not bad. The cause for concern is that now, once the boundaries are set, it’s difficult to go beyond them. In the past, the prevalence of shared experiences made moving up into that other, affluent world possible. Today, however, it is becoming nearly impossible. And we in tech are not making it easier. Why? One reason is automation.

    Automation replacing human labor is nothing new. It’s been going on since before Otis figured out how to make elevators go up and down between floors without requiring human intervention. Yet, this is simple stuff by today’s standards. As automation becomes more sophisticated, the degree of smarts required to work in an environment in which automation prevails grows. An engineer capable of refactoring a piece of software that does speech-to-text transformation has had significantly more education than a stenographer—which, by the way, is a profession automation is replacing.

    Automation is making it more difficult for those with only a high school education be employed. Yes, some sources claim that automation will actually create more jobs. But, how many of these new jobs will require nothing more than a high school diploma? And, once hired, how long will these new jobs be available before they are replaced with another piece of automation?

    Now, combine this trend in automation with Murray’s assertions about the growing extreme in class separation. One group is happy, healthy, pretty wealthy and productive. The other group stagnant, poorer, in ill health and continually being replaced by automation. Add in another troubling statistic: The rate of divorce in the happy, healthy, well-educated group is going down. The divorce rate in the other group is going up. (Page 146, Coming Apart, Murray) In addition, the number of single parent households among the less educated group is increasing, while the rate of single parent households for the happy, healthy, wealthy group remains low and substantially less than the other. (Page 159, Coming Apart, Murray). One group has the stability necessary to promote health and wealth in its offspring. The other group is barely keeping its head above water with little hope for better life now, and for generations to come.

    What does this mean? It means that things are coming apart.

    Probably one of the most important lessons we’ve learned in DevOps is that open systems, genuine accountability and blameless reflection combined with a cross fertilization of ideas between different groups makes for a better, more productive work environment. This thinking can and does go well beyond software. Anybody who has experienced the waste and frustration of a closed, siloed organization understands that open is better. We in DevOps understand the the smarter we all are the better off everybody is. Sharing knowledge and expertise is not a zero sum game.

    According to Murray, we are no longer all passengers on a common ocean liner. Today two ships are sailing side by side. One has an able crew with enough supplies and lifeboats for everybody aboard. The other sails along somehow, with dwindling resources and a few lifeboats. We in DevOps are are snug and secure in the first ship. For those in the other ship, the clock is ticketing and the ship is taking on water. Those of us who care about the Big Picture understand that things need to change, that we need to come together rather than come apart. The question, how? This is the question I will leave to you to answer.

    I look forward to reading your ideas about how to bring us together in the comments section below.

    — Bob Reselman

  • Jump-Start Your Kubernetes Journey with JenkinsX

    Jump-Start Your Kubernetes Journey with JenkinsX

    Containers are very quickly being adopted as the stable way of deploying services. More and more organizations are adopting immutable architecture and containers provide an excellent way to achieve that.

    But along with the stability comes the complexity of deployment, along with managing workload, scaling and updating. That’s in addition to having to create a reliable and repeatable process to continuously build and ship these containers to a production ready state. Kubernetes is one option: It provides features to reliably manage production workloads for your services.

    So now developers have to learn Kubernetes, as well as how to prepare their application for Kubernetes and create the pipeline for continuous integration and continuous deployment (CI/CD). Developers also must determine what is the best approach to get this done and where to start.

    JenkinsX: CI/CD for Kubernetes

    JenkinsX is an open source CI/CD solution for the modern cloud platform for developing and managing applications on Kubernetes. It’s the automation that helps an application move to Kubernetes, deploying various services on Kubernetes to manage an application’s CI/CD journey.

    Developers simply focus on their product and its features and JenkinsX will manage the Kubernetes deployment. It’s opinionated yet flexible, and provides extensive customization. Users can create a Kubernetes cluster using JenkinsX on AWS, Azure, Google or even locally on minikube or minishift. Developers who already have their own Kubernetes cluster and its RBAC (Role Based Access Control) enabled can simply install JenkinsX on top of it.

    JenkinsX provides various templates, and, since it’s open source, more templates are added daily. Developers can do a quickstart with a templated version of spring boot application, a golang project, node, open-liberty, python-http and more. It will create a git repo for a template application and then connect to developer’s github account to push this template to a repo. Also, it sets up the required webhooks to talk to Jenkins, which is installed on Kubernetes as part of the JenkinsX setup.

    JenkinsX also creates namespace environments in Kubernetes. By default, it will create a staging and production environment that can be modified.

    Once changes are made to a templated quickstart application and a pull request is created, the process kicks in and pushes the payload using webhooks to jenkins. The pipeline jobs then kick in to start the build process.

    After the application builds in its own container, the next stage begins to create a docker image and push it to the internal docker registry. Helm charts are generated for the application and stored in the internal hosted chart museum. It is then deployed to a preview environment and developers can view or run tests against the application even before the pull requests are merged.

    As the pull request is reviewed and merged to the master branch, a job kicks in to deploy that application on to the environments. JenkinX implements GitOps, so any and every change is triggered from the github repository. It creates a pull request to the environment repository and adds a change to deploy this application in this environment., validates the pull request and automatically merges it, kick-starting the final deployment stage.

    An environment is mapped to a specific namespace, the application container is then deployed in that namespace, a service is created and all the required mappings are configured. Also, a URL is spit out for developers to hit their application.

    Developers then push their changes, grab a coffee (this step is optional) and few minutes later they get the URL to access the application in that environment.

    That’s the standard workflow. As you might have noticed, JenkinsX is very different from the traditional Jenkins approach. Jenkins provides complete flexibility, whereas JenkinsX provides an opinionated approach. Given everything is in the template repo it creates, you can always modify them to your requirements.

    A Jenkinsfile is created for the pipeline jobs that can be edited to add another stage. Helm charts are created and can be updated if required. Users can disable the default environment it creates the first time and have their own set of persistent or disposable environments.

    This is a great approach toward automating of various different tools we use today to build and deploy our applications on Kubernetes. There are a few features I would love to see added, such as different deployment strategies, support for internal docker registries and maintenance features to clean up obsolete images. That said, JenkinsX removes much of the learning curve and jump-starts your Kubernetes and container journey.

    — Saurav Sen

  • Using CALMS to Assess an Organization’s DevOps

    Using CALMS to Assess an Organization’s DevOps

    Of the tried-and-tested frameworks that allow enterprises to assess DevOps in their organization and how it can be improved, the CALMS model remains particularly useful.

    CALMS, which stands for Collaboration, Automation, Lean, Measurement and Sharing, is particularly helpful for analyzing an organization’s DevOps structure, and ultimately, its utility in any organization. The CALMS framework covers all stakeholders within DevOps, including business, IT operations, QA, InfoSec and development teams, and how they collectively deliver, deploy and integrate automated processes that make business sense.

    “DevOps as a technique is slowly but steadily maturing, so it’s good to see general assessment methods like the CALMS model,” Holger Mueller, an analyst for Constellation Research, said. “The CALMS model gives a good frame of reference to compare the maturity of a DevOps team, and as such,  is invaluable for assessing the state of teams for the transformational change that goes with it.”

    Applying CALMS

    Here are a few ways CALMS can be applied to DevOps for enterprises either thinking about shifting to a DevOps structure or seek to make improvements to an existing deployment.

    Culture

    It is long-established that technology adoption must be to serve a business need rather than investing in technology for its own sake. This shift in mindset applies to how the “culture” component of CALMS supports how the forecasted ROI associated with automating a process must be championed across the DevOps team. This mindset is especially important for getting the non-technical business teams onboard, Mueller said.

    “DevOps must have the backing from the business executives. That’s a crucial factor in the culture part,” he said. “One thing software teams can do is use successful software deployments to help convince the business executives in DevOps to make the investments. Those are the kinds of tribal events that every DevOps organization needs.“

    Automation

    Netflix is well-known as an exemplary case study for DevOps automation. Among other things, it can deploy code and software updates in a matter of minutes across its entire massive infrastructure, many times throughout the day. But Netflix’s example of automation does not take into account what can go wrong at firms that want to follow Netflix’s example when automating their software deployments and updates, Mueller said.

    “The risk is always deploying too fast and you end up deploying bad code,” Mueller said. “Automation, of course, needs to take this into account and compensate. But not everybody can do that.”

    Lean

    Once completed, it is relatively easy to document cost savings by eliminating resources. Within the CALMS framework, the corresponding “lean” component in DevOps is thus relatively easy to assess, Mueller said.

    The good thing about DevOps, Mueller said, is companies can come to a consensus ahead of time regarding what they consider to be a lean operation.

    However, he added, “There is always the risk of being so lean you begin to err on the QA side.”

    Measurement

    The success of software deployments is relatively easy to measure once completed. However, attempting to measure future deployments as part of a forecast model is, of course, much more challenging.

    “It can be surprisingly tricky to measure how good DevOps is ahead of time,” Mueller said. “Measurements should thus allow errors to be pinpointed  quickly during the post-deployment stage to fix the code if needed very quickly.”

    Sharing

    Sharing, in many ways, overlaps with culture. This is where assessing whether all of the different teams within DevOps are working together and communicating comes into play.

    “Thanks to  DevOps, the days of the lone developer are long gone. But sharing is not always easy,” Mueller said. “Even within separate teams, it is easy to fall into the trap of wanting to get your own work done without necessarily thinking about the team as much as you should.”

    The End Result

    While applying CALMS to assess DevOps never hurts, the framework is still not as important as one of the main objectives of DevOps—namely, using technology to make organizations’ operations more agile and reactive to customer needs, Mueller said.

    “At the end of the day, assessment methods such as CALMS are great, but they are never as important as focusing on implementing next-generation applications,” he said.

    — B. Cameron Gain

  • ChefConf 2018: Chef Grows Beyond Infrastructure to Apps

    ChefConf 2018: Chef Grows Beyond Infrastructure to Apps

    chefconf keynoteHere I am in Chicagoland for my third ChefConf. I caught the keynotes today with Nathan Harvey, Barry Crist, Adam Jacob, Corie Scobie and more. The big news at this year’s event is that Chef is focused on more than just infrastructure and compliance—Chef is bringing automation to apps and the entire software delivery life cycle.

    I spoke with Chef CEO Barry Crist about this and he said there is a sea change taking place with software is delivered. Chef has seen this wave coming and, over the last few years, has been retooling its offerings to match this. Today, Chef Automate and Chef Habitat represent the heart of Chef’s arsenal in its mission to offer the premier platform to automate the software development, deployment and security life cycle.

    I personally have seen this shift over the last few years. I think back to when we launched devopsy.kinsta.cloud. Chef was already a popular solution within the DevOps community; however, most people will tell you that they used Chef for configuration management. It was great to be able to make sure that as you spun up new instances in the cloud, your infrastructure was configured exactly as you wanted. By automating and using a Chef-type of solution, you could spin up new instances at-will and their configuration would be exactly the same or exactly as you desired.

    At the time, that was pretty cutting-edge. People were writing Chef recipes for all kinds of tasks—again, usually around configuring infrastructure.  With the advent of microservices, containers, Docker and Kubernetes, this type of automated deployment has become commonplace. But the need to automate, especially with security and compliance and app deployment, is perhaps an even bigger mission.

    Crist acknowledged these facts during his keynote. In speaking about Habitat, Crist noted that not only did many of Chef’s customers not understand the real story behind Habitat, but that half of Chef’s own employees didn’t, either. One person who did was Julian Dunn, director of product marketing, who I interviewed on DevOps Chats about Habitat when it first launched. Tony Bradley wrote that it could be a game changer back then, as well.

    Habitat has come a long way since then and still has a ways to go to grow into its ambitious potential, but it is very functional right now. Chef has several clients here at ChefConf talking about their Habitat deployments and plans. Chef now tags Habitat with Automate Any App Anywhere – Build, Deploy, Manage.

    Perhaps further along than Habitat, however, is Chef Automate. In fact, Chef Automate 2.0 was announced ChefConf, and it seems to be an almost complete rewrite of the program, as it is now built on Microsoft Go. Here is Chef’s take on Automate 2.0:

    Chef Automate has been rebuilt from the ground up to provide performance, scale and advanced analytics required by enterprises using DevOps at scale to meet the demands of post-digital transformation. Chef Automate now provides a single control plane for infrastructure and compliance automation to enable a closed-loop ‘detect, correct, automate’ process to make infrastructure effortless.

    New features include:

    • Enhanced operational visibility and debugging – New tools and visualizations, including a streaming event feed, trend graphs and a rich query language, help users analyze infrastructure and compliance automation data from a unified dashboard.
    • Compliance scanning and reporting in any environment – Integration with cloud compliance capabilities of InSpec 2 enhances scanning, which can now be used to evaluate cloud configurations (AWS, Azure, Google Cloud) via APIs for compliance on either an ad-hoc or ongoing basis. Chef Automate 2 also supports scanning of Cisco IOS equipment as a significant start on extending modern compliance capabilities to network devices.
    • Re-architected for speed and flexibility – A Go-based microservices architecture, with a modern, performant web UI atop a REST API, enables Automate 2 to support tens of thousands of nodes from a single installation. In-place upgrades with automatic data migration from Chef Automate 1 ensure that customers retain all historical information.

    Additionally, Chef has released something called Chef Workstation. From the release:

    Chef Workstation enables DevOps teams and practitioners to complete ad-hoc DevOps tasks using all of the familiar Chef capabilities, without agents. This provides a simple and effective way to use Chef to complete tasks and then seamlessly scale those tasks through Chef Automate.

    Capabilities include:

    • New standard desktop/laptop experience for Chef – Includes a new ‘target mode’ for executing Chef recipes or entire cookbooks remotely and Introduces ‘chef’ command as the single point of entry.
    • Foundation for a better workstation experience – Bundling all the tooling previously included in the ChefDK provides an improved user experience and cohesion between tools to deliver a seamless experience for any infrastructure automation task.

    Finally, new enhancements to Inspec, Chef’s compliance and security solution, were announced. Besides security and compliance being built into most everything Chef does, Inspec now sports:

    • Cloud configuration compliance – Verify AWS and Microsoft Azure policies, with Google Cloud Platform support in beta. More than 10 new AWS resources bring it to CIS Level 1 benchmark completeness.
    • Improved out-of-the-box experience – More than 30 new resources, including support for Cisco IOS network devices as agentless targets, are built in.
    • Drastically improved performance – Users now see up to 90 percent performance gains on Windows and 30 percent on Linux/Unix.

    I will report more as it comes here in Chicago and ChefConf 2018.

    — Alan Shimel

  • Code Delivery: Pipeline Automation for the Speed to Innovate

    Code Delivery: Pipeline Automation for the Speed to Innovate

    Many developers steeped in the world of agile startups view continuous delivery pipelines as an accepted standard requirement for software development. Yet many companies, particularly large enterprises with traditional infrastructure, still struggle to make this approach a standard part of their development process.

    Whether your company is an enterprise looking to make continuous delivery pipelines a standard project element to increase agility and speed time to market, or your firm is looking to simply implement code delivery pipeline best practices, optimizing code delivery pipelines through DevOps-based automation will increase developer output and strategic contributions and deliver substantial business value.

    Efficient code delivery pipelines that balance security and agility can be achieved by using automation coupled with cloud infrastructure. Together, these technologies and processes can deliver big dividends in the form of lower development costs, higher-quality results and faster times to launch. By focusing on optimizing a delivery mechanism for value versus just delivering more code, code delivery pipelines can meaningfully and measurably impact business objectives.

    Continuous Delivery in Modern Development Environments

    Continuous delivery is a design practice wherein developers use automation to accelerate the software delivery process. It is a high-value process because it enables programmers to cut the overall cost of software development. Faster delivery of high-quality code makes it easier for companies to improve customer satisfaction, capture new markets and gain a competitive advantage.

    To optimize this development model, companies need to automate their build, integration and testing processes as well as provisioning and retiring infrastructure components.

    Let’s look at a typical scenario. Code delivery starts from a feature or bug request filed with a developer. The developer takes existing code, modifies it, does local testing and commits it to the code repository. The code in the repository is then tested by QA and tagged for production, where IT or operations professionals deploy the code to production servers.

    This creates loops at the local testing, QA, and production stages of the process. Each of these loops has a cost, which is directly affected by developer productivity and how long it takes to execute each loop.

    The local testing loop executes most frequently, so it’s critical to make this part of the testing process fast. Production bugs are the least frequent but most expensive for businesses when they exist. Consider the cost of a significant disruption to the workflow, or the negative customer experience, or the inability to deliver a service, or security vulnerabilities. Imagine if you could keep delivery time and costs under control while minimizing total effort at every stage.

    Ideally, code should be pushed in smaller batches. The downside to this approach is it creates overhead and room for error as more and more things move through the process. To reduce the overhead of too many pushes, automation can play an important and meaningful role. Indeed, automation makes it cheaper, faster and more reliable to do things in smaller batches and helps streamline delivery.

    Stories from the Front Line: Automated Code Delivery Pipelines

    DevOps automation such as automated code delivery pipelines can be more easily achieved when coupled with cloud technology. For example, Flux7 recently worked with a Fortune 500 enterprise to set up an automated code delivery pipeline using AWS CloudFormation, AWS CodeCommit, AWS CodePipeline and Jenkins. Developers would use a Git flow methodology, and once they merged code to the master branch in CodeCommit, polling would alert to a new commit. CodePipeline would create a CloudFormation change set and queue it up for review by the Ops team. Once the Ops team approved the change set by manually pressing a button, CodePipeline would trigger a CloudFormation stack update. This allowed for very high full-stack agility while not losing central governance.

    Many leaders in technology organizations are moving to Agile and DevOps as methods of improving the speed of delivery, of which an automated code delivery pipeline is foundational. Optimized development pipelines deliver significant value by minimizing the time and costs of local test, QA and production processes and by promoting code from one development stage to the next.

    When you engage in this process, you reduce the time it takes to identify problems and their causes. And when you automate development steps, you dramatically reduce the cumbersome and time-consuming tasks that slow down the release of software code.

    Impact to Your Business

    Efficient developer workflows in a cloud-based environment are the foundation of pipeline optimization. Their importance relates to the need to:

    • Control ever-increasing developer labor costs. Your monthly developer labor cost is almost guaranteed to exceed the monthly cost of your cloud server setup. Keeping your developers working efficiently means you get the most out of that labor and can focus that on what projects are bringing the most value to the business.
    • Get a clear understanding of developer activities. When we interview developers and IT team members, what emerges from these discussions is astonishing. Companies learn that they really don’t know as much as they thought they did about their employees’ day-to-day activities and workflow.
    • Find developers with the required skill sets. Hiring the right talent is costly and time-consuming. If you can help your developers work more efficiently, you can avoid the search costs. And, building a reputation as a company that supports developers helps attract new talent.
    • Accelerate time to market. It’s possible to improve the performance of less-productive developers by accelerating their work process with optimized code delivery pipelines and by building best practices in through automation.
    • Avoid rework-related slowdowns. Rework comes at a cost of time and effort. Automating development processes can avoid many of the human errors that cause rework.

    An effective code delivery process is highly valuable as it reduces risk from each release, ensures that rogue code is not introduced and helps organizations effectively manage global deployments. Most importantly, optimized code delivery pipelines can significantly impact business objectives and business outcomes, making DevOps a strategic contributor to achieving business imperatives.

    — Aater Suleman

  • Trust and Safety in the Age of Automation

    Trust and Safety in the Age of Automation

    “As with any new technology, it’s really important that we be thinking now about how to do [it] ethically and responsibly.” —Robert High, Vice President and Chief Technology Officer, IBM Watson

    Here’s something to think about: According to an article in The Wall Street Journal, more than 30 countries have defensive weapons run by automation. South Korea has robotic sentries along the DMZ, while Israeli robots patrol the border at Gaza. Russia has robotic vehicles armed with 30mm cannons and anti-tank guided missiles that are well-equipped to battle it out with NATO forces should the need arise.

    Got your attention? Wait, there’s more.

    The SeaHunter is a U.S. military vessel that can travel the high seas on its own for months without a crew to feed or protect. It’s completely unmanned. And, it costs only $20 million per boat, less than half of the $49 million it costs to fire 59 Tomahawk missiles. At that price, the SeaHunter is practically throwaway.

    The SeaHunter, an unmanned oceangoing vessel, can remain at sea for months at a time.

    The implications are profound. The value proposition is hard to resist, Why spend billions on building and manning traditional warships when, for the same expense or less, you can have a navy of predominantly robotic ships patrolling the oceans of the world? And, you can have robotic aircraft, capable of automated takeoff and landing, using these ships as mobile bases. An air force untethered to any land mass is a game changer, particularly when non-state actors can get their hands on a few of them. It sort of brings new meaning to the word “pirate”—no need to board a vessel by force, just get the access credentials.

    And, at the core of it all this modern weaponry is automation and growing capabilities in AI. That should give you something to think about.

    We in DevOps are all about automation. It’s our raison d’être. Our fondness for, and dedication to, automation is admirable. We really have changed the way tech works. Yet, many of us think of our activities as mundane. It’s akin to the nonchalance of a Major League baseball player. They’ve hit the ball so much that many players have forgotten how skillful one needs to be just to foul off a bad pitch. Most people on the planet can’t hit a Big League pitch, let alone make it go foul by intention. But for the pro, it’s no big deal—it’s just part of a day’s work.

    It’s the same for us. We’re so accustomed to automating our world that it’s become no big deal. But, it is a big deal. It’s a very big deal.

    The easiest thing in the world is to trivialize the incredible impact our work has on others. After all, we’re “just working on” an ecommerce site, a dating app, or IoT endpoint for wired bicycles, right? Those of us working in the defense industry might grasp the significance of the work, but for most of us, it’s business as usual.

    Here’s the raw truth: Those of us doing the automation are changing the world. Each line of code we write is intelligence that alters the digital infrastructure of some part of the planet. The code we write will live on well after we’re gone, whether it’s from a company or from the planet. Are we really clear about the impact we have? Yes, some of our tech superstars are asking us to pay attention, but how many of us really are?

    Every day that passes we give automation more power. It might be a little piece of power, such as letting only authorized users access to an application. It might be significant power, such as determining whether a person qualifies for a loan to buy a house. It might be enormous power, such as manipulating political forces toward a desired behavior. Or, it might be the ultimate power: Deciding who gets to live and who gets to die. But, regardless of degree, it is power and we have it. And, we continue to make automation more powerful. As surprising as it may sound, despite appearances otherwise, we’re the ones the world trusts intrinsically to do our work safely and responsibly. A lot of the world’s safety depends on how we do our work. Political leaders don’t write the code. We do.

    Originally,  I was going to call this piece “Death and Destruction in the Age of Automation” and write about how we’ve taken automation way beyond commercial IT into the realm of life-and-death decision-making. I decided otherwise. The world does not need another gloom-and-doom piece, nor does it need another piece of sensational journalism. We’d do better to focus more on the positive things we’ve achieved. Technology is doing a whole lot more good than bad.

    But the fact remains that technology has accompanied death and destruction since Roman catapults assaulted the frontiers of ancient Europe. The thing that’s really different now is that the technology thinks—or, at least, it’s doing a really good job of emulating thinking. And, we’re the ones making it think better. Thus, we should ask, Are we making technology think in ways that are safe and trustworthy? Are we acting in ways that are safe and trustworthy?

    These are big questions. Do most of us really care about them? Or, is it easier to accept our work as mundane, with little consequence in the Big Picture? The nice thing about mundane is that it doesn’t require a lot of ethical contemplation. But, when you consider that the script that deploys an updated container to a Kubernetes cluster is not that far from the one that deploys an unmanned warship to a regional hotspot, things look different. And, contemplating the implications thereof becomes more than an academic exercise in a computer science ethics class.

    More than 2,000 years ago the philosopher Socrates said, “The unexamined life is not worth living.” Given the immense role that technology plays in today’s world and the power it wields—power that continues to move further away from our direct control—we might do well to say, “The unexamined technology is not worth doing.” It’s something to think about, while we still can.

    — Bob Reselman

  • Puppet Expands Scope of Pervasive Automation Initiative

    Puppet Expands Scope of Pervasive Automation Initiative

    Puppet today unveiled a series of initiatives as part of a continuing campaign to make IT automation available pervasively across the enterprise.

    Omri Gazitt, chief product officer for Puppet, said the first step in that effort is making it easier for IT organizations to discover what assets they need to manage. Puppet Discovery is an agentless tool through which organizations can monitor their IT environments continuously. Any changes or additions are captured immediately in a database.

    Scheduled to be available in May, Puppet Discovery replaces a traditional configuration management database (CMDB) with a platform that more accurately reflects the state of the IT environment because it’s always up to date, Gazitt said.

    Puppet is also extending the reach of its IT automation framework to include modules spanning platforms from Barracuda Networks, Cisco Systems and Nutanix, as well as the Azure cloud service from Microsoft.

    Modules for Cisco ACI software-defined networks (SDNs) and Cisco Meraki wireless networks are now available, along with a private beta of a module for the Cisco IOS operating system widely employed on the Cisco network infrastructure.

    Similarly, Puppet modules are now available for security and data protection offerings from Barracuda Networks and the hyperconverged infrastructure (HCI) platform developed by Nutanix.

    Puppet, in the context of the cloud, is now adding a module for Azure Resource Manager, as well as adding support for Windows PowerShell Desired State Configuration (DSC) and the Microsoft Task Scheduler 2.0 application programming interface (API) used in Windows.

    In related news, Microsoft, Barracuda Networks and Puppet also announced they are combining their respective capabilities to create an Azure Accelerator program to automate and secure mission-critical workloads. Organizations that participate in the program can get started at no cost, with free software from Puppet and Barracuda and no Microsoft Azure infrastructure charges while migrating workloads to Azure Cloud.

    Other new members of the Puppet Technology Alliance Partner program include Lenovo and xMatters.

    Finally, Puppet announced scheduled June availability of Continuous Delivery for Puppet Enterprise (PE), an update to Puppet Enterprise that includes role-based access control for Tasks, which can define node groups corresponding to business functions or units to limit who can run tasks within a specific node group. Puppet also has added Puppet Bolt Task Plans, which enable customers to orchestrate a set of tasks across thousands of Puppet Enterprise nodes using Puppet Orchestrator. Continuous Delivery for PE is based on a continuous delivery platform developed by Distelli that Puppet acquired last year.

    Puppet Enterprise has also been enhanced with easier backup and recovery capabilities to automate disaster recovery and support for a Puppet Development Kit (PDK), which includes a new resource application programming interface (API) that Gazitt said can be used to create custom Puppet modules to automate the management of any platform.

    In general, Gazitt noted that IT automation and DevOps increasingly are becoming synonymous. The biggest issue now, he said, is unifying all the independent silos of DevOps implementations that now span many enterprise organizations.

    Of course, IT organizations can apply IT automation without necessarily embracing DevOps. But once an organizations begins to adopt DevOps, it’s pretty clear IT automation quickly becomes indispensable.

    — Mike Vizard

  • 5 Things That Will Be Automated in 2018

    5 Things That Will Be Automated in 2018

    We may not have fleets of robots waiting on us hand and foot yet, but the automation revolution is in full swing. With automation becoming more apparent in daily life, some of the misguided fears over the concept and its potential to eliminate jobs are slowly being replaced with the view of automation as an opportunity. And given that IT departments are powering today’s businesses, few can capitalize on this opportunity more than enterprise IT.

    Across industries, the pressure to deliver innovation faster continues to grow, and with budgets and headcounts remaining relatively stagnant, organizations not adopting an automation strategy are being set up for failure. Automation allows computers to do the mundane tasks—and increasingly, even the not so mundane—freeing staff to focus on innovation.

    Automation Everywhere

    While 2017 demonstrated that automation is key to IT optimization and digital transformation, 2018 will be the year IT automation truly hits its stride, with widespread, systematic adoption across industries. So what trends in automation should organizations be looking to embrace next? These five areas in enterprise IT are prime to be automated in 2018:

    Networks

    Networks are arguably the bedrock of enterprise IT, yet true automation of the network stack has been nearly nonexistent. In fact, managing networks hasn’t changed significantly in the past 30 years. Network engineers still are logging in to individual devices and hand-crafting configurations, leaving network teams increasingly isolated from the DevOps revolution and slowing digital transformation.

    This year, network automation will move full steam ahead in the enterprise—and there isn’t really an option for it not to.

    With the ongoing march to cloud, new and potentially massive IoT deployments and increased pressure to modernize internal IT for efficiencies, network engineering must have the same focus and attention that server and app engineering has had in recent years. Network teams cannot remain isolated and resigned to merely maintaining the status quo; we should embrace them and show them how to reap the same benefits combining Dev and Ops has provided.

    Bringing the automation of networks into the fold of DevOps will enable tasks to be executed with increased cadence and with a higher degree of accuracy, while increasing collaboration between teams that are critical to the success of modern application architectures.

    Infrastructure

    To run business systems, you need infrastructure, and that infrastructure must be set up to have an impact on an organization’s day-to-day operations. Regardless of whether your systems are running in the cloud, virtualized environments, containers or on bare metal, if you still have to configure and provision infrastructure manually, you are moving too slow. Weeks or even days to provision systems means holding up changes that your business needs.

    While automated provisioning has been around for a long time, in 2018 we will see this amplified, primarily due to the ease of use of automation tools and platforms. With automation, infrastructure can be scaled automatically or, using precanned scripts, an operations engineer can quickly affect hundreds or even thousands of devices. This type of technology will allow IT departments to rapidly expand their infrastructure without expanding manpower.

    Once automated provisioning takes over the heavy lifting, focus can move to automating management, maintenance and troubleshooting.

    Microsoft

    Traditionally, in the enterprise, Microsoft environments have been managed by discrete teams. Recently the company has been doing a great job of helping users learn from wider experiences within the industry (joining the open source initiative, for example), and it’s helping break down some of the walls within IT departments. Microsoft also has focused on enabling Azure-backed services in the enterprise, offering more opportunities for customers to adopt automation.

    With SQL Server accelerating in popularity—helped by installs on Red Hat Enterprise Linux as well the traditional base of Windows—automation is an opportunity to ease some of the traditional burdens of scale. We’re already seeing open source development of SQL Server Ansible modules, which is only going to help fit this piece of critical application infrastructure into DevOps-style workflows.

    Microsoft has also taken a broader interest in open source projects. This has included contributing to the Ansible community, helping users integrate open source projects with their Windows environments.

    Security

    Make anything simpler and easier and it is far more likely to be adopted. Over the last 25 years, the connection of our lives to the internet has only increased complexity. This complexity has made it more difficult for security professionals to keep us all safe.

    With configuration management, we’re able to build security in from the start rather than as an afterthought. Changes can be logged and enacted rapidly, giving us the ability to respond to incidents in a timely manner—even across massive scale.

    As our colleagues in cloud and networks have automated more, they’ve reached new levels of scalability. This year we’re already seeing growth of the DevSecOps movement, and expect it to continue. When we bring systems and processes together with automation, doing so securely becomes easier.

    Orchestration

    It comes as a surprise to no one when we say modern applications are complex. They comprise the latest technologies as well as legacy systems. They run in the cloud, across multiple databases and need to be accessed from a variety of devices. This makes it exponentially harder to determine root causes of issues, let alone tell these resources what to do. Since application performance can impact a company’s bottom line, maintaining it is a top priority. However, performance tuning often can’t be an IT team’s full-time job.

    Orchestration is the automation of processes. It delivers relief from the administration, monitoring and help-desk chores that eat up time and resources. IT teams can define how an application is supposed to behave in its running state, and orchestration can bring about that bigger picture. Orchestrated automation is what will really set companies apart from one another, allowing teams to deploy applications quickly and reliably with an easy maintenance path ahead.

    — Justin Nemmers