devopsy.kinsta.cloud

Tag: automation

  • A Saner Way to Build a DevOps Cloud

    A Saner Way to Build a DevOps Cloud

    There’s a popular definition of insanity: doing the same thing repeatedly but expecting different results. Yet, too many enterprises implementing digital services and apps in the cloud fail to heed this warning as they turn to point-to-point integration of their cloud components. In the process, they’re introducing tremendous complexity that looks eerily similar to the point-to-point application integration headaches companies faced before the concept of centralized mediation emerged some 15 years ago.

    Consider the scope of the challenge. A cloud deployment can contain 30-plus product components that need to be integrated to run together smoothly. At any time, an update to one of these products can break processes that rely on the other components. DevOps compounds the complexity, since teams need to ensure consistency across development, test and production environments. It may be fine for a dedicated DevOps team in the group to handle these environments when there’s just one project. However, as the number of applications utilizing cloud technology grows, it is expensive to have DevOps teams build each environment as a wild west project.

    Standardizing on Stacks

    A saner approach is to implement a cloud architecture that standardizes on stack-level operations. Implemented effectively, a stack serves as the reference architecture for all the components and automation needed to create a replica of your production environment. This “full stack” is in essence much like the configuration management database (CMDB) once thought to be the holy grail of enterprises.

    With a meta-description of your full stack for a service or application in place, it becomes possible to deploy identical copies for development, test and production, as well as automate DevOps across your application. This ability to duplicate stacks removes the barriers to, for example, deploying multiple test environments or multiple copies of the stack for security compliance or disaster recovery.

    The stack is an important concept for understanding how you are building and running your software. If something goes wrong, it also provides visibility into what is running, how it was deployed and what the problem might be, facilitating the ability to systematically correct the problem and deploy the fix to all needed replicas. Developing a stack concept also lets you track applications—plus the associated tools, development practices, cloud practices and vulnerabilities—across your organization, providing a critical step in the maturity of your enterprise’s cloud development.

    Standardizing on stacks enables development teams to focus on building new applications and services rather than spending six to 18 months implementing the cloud architecture. At the same time, different groups may require different capabilities, making a one-stack-fits all-approach impractical. The middle ground is to provide a set of “sanctioned stacks” or a portal for configuring stacks, which limits options to products that have been pretested to confirm they work together.

    Structuring DevOps Cloud Stacks

    A DevOps cloud architecture will consist of at least two main stacks: the core infrastructure stack and the DevOps stack. Increasingly, these stacks run in containers, which provide two key advantages: a greater agility through the ability to spin up apps faster and more cost-effective use of resources through operating system-level virtualization.

    The core infrastructure stack supports a service or application with services that ensure security, reliability and scalability. It includes cloud infrastructure, such as networking and storage, which may be handled by a public cloud platform or a private cloud, along with container-specific functionality, such as a Docker runtime with orchestration provided by Docker Swarm or Kubernetes. The stack also has a layer for handling performance and management, including networking, load balancing, service directory, log management, monitoring, backup and recovery and vulnerability scanning, which are supported by a range of different products.

    At the top of the core infrastructure stack is an application layer or stack that includes development tools, such as Java Stack or Node.js Stack. More advanced application stacks will go beyond tools to provide the blueprints for migrating from monolithic apps to containers and microservices.

    The DevOps stack supports continuous integration/continuous deployment (CI/CD). It runs across multiple infrastructure stacks and codifies the tools, processes and procedures to test and deploy or rollback changes to the infrastructure stacks. It typically incorporates DevOps instructions using Chef or Terraform, Ansible, Puppet, Swarm, Helm and half a dozen other tools. Because automated testing is critical, DevOps stacks also should include tools that help with development, building, integration, automated testing. Popular options include WireMock and Selenium for testing web apps and services, Artillery for performance testing and Blackduck or Clare for security testing.

    Additionally, there are key functions within the DevOps cloud platform that extend across development, test and production stacks. These include security, a catalog of stack templates, a registry for certified containers and usage data collection to apply predictive analytics for performance and cost optimization.

    Introducing Stack Automation

    Enterprises implementing a DevOps-enabled cloud need to consider automating as many processes as possible, particularly for large installations. For example, a bank may have 150 to 200 groups developing applications. With each app requiring environments for development, test and production, that means anywhere from 450 to 600 deployed stacks to maintain.

    Significantly, updating stack components needs to be a coordinated process. It’s not enough to rely on automated upgrades and security patching provided by individual product vendors. That’s because if the upgrade or patch changes something that affects a component in the enterprise’s stack, the vendor probably doesn’t know that and this may break the stack.

    Automating stack functions in a coordinated way is best accomplished using an intelligent, centralized “command center” portal that incorporates a central orchestration hub. This eliminates much of the complexity associated with manually running multiple scripts by performing whole operations on a stack. Whether your organization builds it or you use a third-party solution, this hub for automating the execution of scripts should incorporate an orchestration engine, visual map, domain model, proactive monitoring and cost analysis.

    Importantly, a command center incorporating an automation hub lets you take advantage of improved visibility provided by stacks to track and report on cloud costs. Moreover, you can capitalize on “spot” instances—costing five to 10 times less than persistent instances—by using the hub to automatically handle periodic interruptions and redeploy the components taken away.

    Through automated stacks, your organization now has a sane way to cut the complexity of implementing a DevOps cloud while providing an agile platform to support your development team’s innovations.

    — John Mathon

  • The 2 Best Ways to Automate Your TechOps

    The 2 Best Ways to Automate Your TechOps

    The DevOps community is rapidly building tools and processes to automate more of the DevOps workflow. One key is to put engineers on the front line who can find root causes of problems and automate more of the common tasks. Here are the two key areas to focus on when automating operations processes.

    Automate Your Most Expensive Tasks First

    The first step is to do some analysis on where the team spends most of their time. You will want to know if they’ve done a good job around the tickets and how many requests of what type have they gotten.

    After analysis, one company found it had more than 500 tickets a day for user access requests from users who were locked out of their account. This is a common problem, especially in midsize to large companies that are doing user management or account management and have very manual processes for onboarding and offboarding.

    There’s a ton of money that can be saved by automating these processes. And there’s a whole host of tools for everything from full orchestration systems to pipeline stripping to make it possible. But the first step is to analyze where time is being spent on the repetitive, easily duplicated tasks and automating those.

    Automating Monitoring and Self-Healing

    There are a lot of different tools out there, and the amount of data that’s coming together is enormous. With the emergence of big data and machine learning, we have the ability to put all the different points of monitoring into one system and bring all the data together to provide a single picture of how everything is performing.

    Data sources can include anything from systems monitoring tools to application performance monitoring tools to social media feeds such as Twitter and Instagram. In addition, it can and should include data from your IT service management (ITSM) tools. Once you’ve got that you can correlate your performance data with your incident data to actually predict when a problem is more likely to occur so you can fix the root cause or the underlying issue before it impacts your users.

    There’s some really neat stuff that can happen with automation, whether that’s just a simple script that can save you time and money or advanced machine learning to actually predict issues and address them in real time. Take the first step to make your organization more efficient and effective.

    — Sean Mack

  • The ‘Blockbuster Effect’ in the Age Of Automation

    The ‘Blockbuster Effect’ in the Age Of Automation

    Saying that new jobs will always be created to replace those that have been eliminated by automation is a mechanism for avoiding a real problem that’s lurking on the horizon.

    In 2004, at the height of its popularity, Blockbuster Video had more than 9,000 stores and employed more than 80,000 people worldwide. About 50,000 of the total workforce was employed in the United States. Going down to Blockbuster to pick up a movie was a Friday night ritual for many, this writer included.

    Blockbuster and other video rental stores, along with the advent of affordable Big Screen TVs, were significant contributing factors in the growth of the home theater. People now had an alternative to $10 movie tickets and battling weekend traffic to get to the cineplex. An entire family could view a recent movie in the comfort of their own home for about $5. The only thing that was needed to complete the movie going experience was the popcorn, which frugal families bought at Costco by the caseload. Those who were less thrifty could buy a box of microwave popcorn at the checkout counter of the video store. The $25 cost of a Friday night date was reduced to roughly $6.

    Blockbusters kept people on the couches and out of the movie theaters. Television had already put a damper on movie-going. In 1930, 65 percent of the population went to the movies once a week. By 1964, the number dropped to 10 percent. Despite the introduction of 3D projection technology and movie theaters offering added features such as in-seat dining, movie attendance is now at its lowest point since 1992.

    Blockbuster was a transformational business on the entertainment landscape. Today, the company is gone. Streaming technology combined with an “all you can eat” subscription model killed it. (See Figure 1.)

    Figure 1: As Blockbuster stock declined, Netflix’s grew.

    Just as Blockbuster did away with the need to go to a movie house to see a film, Netflix, Amazon and other video streaming services did away with the need to leave the house to get the evening’s entertainment. America stayed home and enjoyed movies on the internet for a fixed monthly fee. Blockbuster went out of business.

    Netflix employs around 5,500 people with gross yearly sales of $11.69 billion. Blockbuster’s gross sales in 2004 were $6 billlion. It’s astounding that Netflix does almost twice as much business as Blockbuster did at its height, with less than 10 percent of the Blockbuster workforce. Such is the benefit of automation.

    Whereas Blockbuster needed to staff a store in every neighborhood to do business, Netflix and other streaming services work from a central, virtual location without the added expense of redundant staffing. Blockbuster had 9,000 cash registers to man. Netflix has one, which is run by some sort of AI.

    So, where did all the out of work clerks and store managers go? Hopefully, they went on to other jobs, maybe ones that were far better than the day-in, day-out tasks that go with processing video cassette rentals. After all, that’s the conventional wisdom: There will always be new jobs to take the place of the ones eliminated by automation. But, let’s face a cruel truth: It takes a lot more knowledge and skill to work on video delivery at Netflix than it does to check out customer purchases at Blockbuster. And Netflix requires fewer people to make the business go. Would a Blockbuster employee in Sioux City, Iowa, have the skills and mobility necessary to go on to to work at Netflix in Los Gatos, California? Seems pretty far-fetched.

    Right now, things are pretty rosy in terms of the economy. So the conventional wisdom prevails: As automation eliminates one job, another new type of job takes its place.

    Yet, there is another way to look at it. There are good arguments to be made:

    • As automation eliminates jobs in the modern world, new jobs for humans will NOT be created at levels required to keep a workforce fully employed. Rather, AI will do more and require less.
    • The lead up to large-scale unemployment will not be slow. It will happen fast, over a period of a year or two. And, it very well might happen within a decade rather than decades in the future.

    Blockbuster’s demise came about pretty quickly. The company was at its peak in 2004. Six years later it filed for bankruptcy with $1 billion in debt. The music retailer Virgin Megastore closed its last North American store in 2009, a year after music streaming Spotify went public on the internet. (Spotify generated $3.8 billion in revenue in 2016 with an employee count of 2,162.)

    Productivity due to automation continues to accelerate. In the early 1980s, it took 10.4 man-hours of labor to make a ton of steel. By 2014, the number was down to 1.9 man-hours per ton. The increase in productivity is due to increased levels of automation (think: robots). Given the choice between buying a machine and hiring a worker, companies are more likely to choose the machine. Thus, it’s entirely possible to imagine that the day when a few humans will be supervising the work of many robots making steel is close. Not too soon after that, a single human will be supervising an entire robotic workforce. Then, finally, as AI continues to improve and further dominate the production landscape, no humans will be required. The robots will supervise themselves.

    How far off in the future is this time? Dunno. But I can tell you that robots are now assisting with surgery on humans. How long will it be until one surgeon can use automation to do the work of 10? Then, how long will it be until AI can do surgery unsupervised, five years? Ten years? Only 10 years ago driverless cars were still on the test tracks; today they are roaming the streets of Mountain View, California. An autonomous robot with a scalpel is no longer the stuff of science fiction. Once AI can do surgery unassisted, where will all the displaced surgeons find work? Maybe a few will go into research. What of the others?

    At one time there were 200,000 horses in the City of New York. They provided the labor for a good part of the city’s transportation system. Today there are fewer than 1,000 horses. Cars, trucks, planes and trains have taken over. There’s no benefit to employing a horse when a car can do the work better. As General AI continues to develop and improve its ability to connect the dots as only a human can, how long will it be until the labor of most humans is not needed? What will it be like when there is no benefit to employing large parts of the human population because a machine can do the work better? Yes, the ramifications are scary. But, the possibility needs to be considered. The sad fact is that few people are considering such a scenario, let alone coming up with ways to address the impact on the horizon. Saying there will always be employment for humans is an avoidance mechanism, particularly given the fact that modern machines are going beyond purely mechanical labor and doing more thinking work.

    In 1990, people thought there would always be movie rental until there wasn’t. In 2000, people thought there would always be a Virgin Megastore until there wasn’t. Today, Sears keeps closing stores. It will soon be gone from the face of the earth. Yes, Amazon is opening brick and mortar stores. This might be good news. But the fact that these stores are highly automated and have few employees cancels out the good feelings of the moment.

    These days when a company comes to an end, it’s not a drawn-out process. It happens fast, over a period of a year. Many of the people left behind are confronted with low-wage, “show up and do it” jobs. The more educated have access to a pool of better paying work that comes with an excruciatingly steep learning curve. Whether the job is low-wage retail work or high-paying tech work, each worker is—or will be—competing against AI-powered automation that can—or will—do the job better.

    It’s not a pretty picture. But it is one that can avoided if we decide to think about alternatives and put those ideas into action. Sadly, there are more people thinking about how to get to Mars than solving the problem of unemployment on a massive scale due to the proliferation of industrial automation powered by general AI. The problem is that we still seem to be stuck at, “There will always be jobs to replace the ones eliminated by automation.” To say otherwise is deemed Cassandra talk. Maybe so. But to not give the problem consideration is beyond folly. It’s a danger that will exact a heavy price from everyone who has to work to earn a living.

    — Bob Reselman

  • CA Technologies Advances DevOps Automation Agenda

    CA Technologies Advances DevOps Automation Agenda

    CA Technologies further advanced its staged goal of unifying its disparate portfolio of DevOps automation tools by announcing today a new release of CA Workload Automation AE (AutoSys) software now directly integrated with the CA Automic One Automation platform the company acquired early last year.

    CA AutoSys is the older of two offerings and is positioned by CA Technologies as being able to support more complex workloads running on a larger number of platforms, including mainframes. CA Automic One Automation was acquired to bolster the automation capabilities CA can offer optimized for distributed computing environments.

    Gwyn Clay, vice president of product management for CA Automic, said it’s no secret CA Technologies has multiple automation engines. The long-term goal is to enable organizations to employ any mix of automation engines and associated DevOps tools as they see fit to create what the company describes a software factory.

    Other new capabilities being added to CA AutoSys include the ability to centrally manage all the various agents the company provides, support for connecting to Amazon Web Services (AWS) and Azure Database Services, improved load balancing and the ability to set up more custom email notifications.

    Clay noted the rate at which IT organizations are embracing DevOps varies widely. Just about every organization has at least one DevOps project or pilot in place, but most of them are still relatively nascent compared to the level of sophistication being applied by web-scale companies, he said.

    CA Technologies is betting that, rather than stitching together components themselves, enterprise IT organizations will prefer a more modular platform that requires less effort to set up without locking them into a specific tool.

    DevOps is coming to the enterprise in one form or another. But unlike web-scale companies, where developers and engineers have tended to work closely together from the inception of the company, enterprise IT organizations will navigate myriad culture issues. There has been a historic predisposition in enterprise IT organizations for developers to “throw code over the wall” to IT operations, with little regard for how that code might run in the actual production environment. IT operations teams, meanwhile, have not always been as forthcoming about the state of that production environment as they might be. That lack of trust and visibility means both sides often make decisions that are not always in the best interest of the overall organization. For example, an application workload might wind up being deployed on a public cloud, even though that platform may be both more expensive and less secure.

    At the same time, it’s still not clear to what degree warring cultures inside the enterprise will be able to put aside their historic animosities to create a coordinated approach to automation and DevOps. Mainframe and distributed computing teams to this day still battle for control over workloads inside the enterprise. Whatever the outcome, there’s no doubt various teams will eventually be forced to work more closely together to achieve DevOps goals regardless of whether they like it.

    — Mike Vizard

  • Standardization, De Facto Style

    Standardization, De Facto Style

    The first decade or so of DevOps had one truth to it: It was about change. Not just change in the way that IT worked, but change in the way DevOps worked—particularly in the tools space, where products and product preferences both are in flux. We’re still seeing a massive change, but increasingly the effect is more like the ripples created when a rock is thrown into a pond than a tsunami, dumping change upon change.

    You could look at this one of two ways (and no doubt those who have more reason to dig into details than I have already done so): Either the slow but steady standardization typified by tools such as Jenkins and Puppet have enabled participation of enterprises in the DevOps life cycle, or the demands of enterprises for some amount of stability in their massively complex systems have driven standardization.

    Either way, the result is the same. We are seeing a lot of change further out, but the core technologies for build/test/deploy are settling into a few products by a few vendors. That is good for adoption, even if you would argue that it is bad for innovation. It means that skill sets are, at least to a certain extent, portable across DevOps teams and organizations, which for a long while was not true. When I was knee-deep in deployment technology, knowing how to use product A did not mean you knew how to use product B. The more people using one or the other, the better for employee mobility and the pool of available resources, as both people and support tools like standardized scripts.

    I recently was going through the offerings of a service provider in a space that doesn’t traditionally have DevOps enablement. The company’s offering included DevOps, so I drilled down. Seeing the company supported Jenkins, Jira and Xebia Labs made me feel better about the program it was putting together. It meant the company was adapting mainstream tools to a specialty market.

    That’s what spurred this blog. It’s been a long time that these things were seen as “new and shiny”; now, for a service provider to include them as proof it is serious about customers’ problems shows just how far we have come.

    There’s still a long way to go. The DevSecOps community is currently in the throes of growth, and lots of outer-edge DevOps tools are incompatible with each other and chosen per shop without a clear market leader. But we’re getting there. And at this point, a person going into DevOps who asks me what they should know will actually get an answer of “Learn X, Y, Z” rather than “You should learn X, but pick up some of X.1, X.2, and A too, then there’s Y …”

    The thing is, though de facto standardization is happening, to many (particularly enterprise) shops, this is all still new. So if you don’t know the recognizable tools yet, don’t worry too much; there will be opportunity to learn as they are rolled out across a given org.

    — Don Macvittie

  • Automation Levels the Playing Field for Continuous Compliance

    Automation Levels the Playing Field for Continuous Compliance

    At this point, I doubt there is any company doing business anywhere that doesn’t fall under at least one compliance framework. One universal truth of compliance—in intent, if not in practice—is that it is a process, not an event. Checking boxes and passing an audit is a requirement of most compliance mandates, but the real value of compliance lies in maintaining that level of visibility and security all the time—which is why continuous compliance is important.

    The Need for Continuous Compliance

    Hopefully I don’t need to continue to make the case that compliance is an ongoing requirement. Because it is something that needs to be consistently monitored and maintained, it also makes sense to automate the process as much as possible.

    In a report titled, “How to Avoid Compliance and Audit Concerns When Using DevOps,” published in November, Gartner stated, “Auditors favor the consistency and traceability of automated systems that have strong logging capabilities and transparent auditable controls.”

    A recent survey of more than 1,500 users conducted by Chef found that 74 percent of cross-functional application, infrastructure and security teams do exactly that, though: They assess software for compliance manually prior to production. Once violations and vulnerabilities are discovered, half remediate manually instead of automating the process. Manual processes result in teams’ detecting and remediating security issues in days (31 percent) or weeks (19 percent), instead of hours (18 percent).

    “At scale, manual reviews or out-of-band security tools bolted onto the DevOps pipelines won’t be able to keep pace with new velocity of change,” said Derek Weeks, VP and DevOps Advocate at Sonatype. “We saw this first with paper-based open source governance policies in the application security realm and it’s exciting to see similar approaches applied to infrastructure.”

    Andrew Storms, VP of Product at New Context, agreed. “Infrastructure automation and configuration management are key requirements for securely storing and transmitting data. … In DevSecOps, bringing security into the development process early and often helps ensure more cost-effective data security.

    “Think about this: By 2020 there will be some 20 billion IoT devices connected to the internet,” he continued. “Most organizations can’t even handle automating security and compliance of their existing infrastructure. The growth demand is far outpacing human abilities. Organizations simply cannot afford to rely on the methods of treating our infrastructure as cute and cuddly pets—and spending weeks producing evidence for security audits is not going to happen.”

    “Rather than force cumbersome security processes on top of the DevOps pipeline, security and compliance professionals need to adopt a new mindset of integrating controls as code,” Weeks added. Codifying controls initially improves the performance of the checks, but, over time, serves to expand the breadth of security and compliance practices as more controls are coded.”

    Speaking a Common Compliance Language

    Effective communication is a critical part of the process. “If you were ever in doubt on the importance of testing, then you should ask NASA about how they lost the Mars climate orbiter,” Storms said. “Years of development and likely some millions later, the space probe burned up because two software components weren’t using same units to communicate.”

    Chef just released InSpec 2.0, which advances the capabilities of compliance automation. Mike Vizard wrote, “InSpec provides a mechanism to automate compliance management using a set of declarative tools that don’t require anyone on the IT team to possess programming skills.”

    The leveling of the playing field by removing the need for programming skills is crucial for DevOps organizations, noted Julia Dunn, Director of Marketing for Chef. “One of the main challenges to DevSecOps (getting developers, security and operations to work together) is communication: not only seeing each other’s point-of-view but having a common language to collaborate on requirements,” she said. “InSpec is a framework that allows for those requirements to be expressed unambiguously in something all three groups can understand. This is in contrast to the previous state, where security throws lengthy scan reports over the wall at operations, or developers toss insecure code at security folks for manual inspection.”

    Jon Williams, CTO of niu Solutions and a Chef InSpec user, has seen the benefits of such a technology. “InSpec has helped us unify our compliance, security and DevOps teams and streamlined audits, reducing the thousands of staff hours usually required by as much as 95 percent and eliminating duplication of effort and data throughout the process,” he said. “It has given these teams more control over compliance policies and enabled business units to be more active in maintaining their own environments. Most critically, it allows us to continually monitor for audit compliance, ensuring desired state and eliminating change drift between nodes.”

    Regardless of what technology you choose, it is important you have a process or framework in place that enables the various parties involved to speak the same compliance language. It is also crucial to automate as much as possible to provide continuous compliance rather than just passing a compliance audit and assuming everything will stay that way.

    — Tony Bradley

  • Survey: Not Much Compliance Progress in DevOps World

    Survey: Not Much Compliance Progress in DevOps World

    A survey of 1,500 customers conducted by Chef illustrates the lacking state of DevSecOps in the enterprise today, finding that nearly three-quarters of IT organizations still manually assess whether applications comply with various regulations before being deployed to production. Worse yet, half the respondents also manually remediate any issue once it arrives, which takes days or sometimes weeks to accomplish.

    In fact, according to the survey, more than half the respondents (52 percent) have applied automation to fewer than 25 percent of the servers they manage.

    Manual processes conspire to make IT organizations less agile at a time when the pressure from the business to deploy applications and associated updates faster has never been higher.

    Julian Dunn, director of product marketing for Chef, said compliance—much like security—needs to move much further left into the DevOps process as a subset of DevSecOps. After all, most compliance regulations are trying to set a bare minimum standard for IT security.

    The first line of IT security defense is detecting vulnerabilities before they go into a production environment. Whenever a vulnerability gets exploited, most organizations incur the initial injury. But after the cybercriminals have come and gone the regulators show up, demanding explanations. Those explanations usually don’t do much to mitigate the fines and penalties that are assessed, especially in highly regulated industries.

    Chef has been making the case for applying the same constructs it developed to automate configurations to compliance for some time. Most compliance issues typically stem from one setting or another that wasn’t configured properly, noted Dunn.

    The issue is that most compliance officers inside organizations are unaware that many of these issues could be automatically resolved, and DevOps teams are too focused on daily operational issues to make anyone aware of what’s now possible. But now that DevSecOps is becoming more mainstream, many of the individuals involved are starting to realize they are dealing with the same issues time and again, said Dunn.

    Dunn added that, much like IT security professionals, compliance professionals who typically reside in a risk-management function need to be brought into the application development process earlier, but without slowing down the pace of application development. However, given the current level of manual remediation occurring, earlier participation mostly likely will reduce the level of manual work occurring right before the application is supposed to be going into production.

    When it comes to compliance, most IT organizations are their own worst enemy. Compliance requirements typically are spelled out in arcane language in documents that not many read, much less comprehend. Asking developers to make sense of those requirements is not the best use of their time. Besides, few of them would interpret the requirements correctly. An automated approach to compliance is not only going to create a better application, but it also should reduce a lot of unnecessary aggravation for all concerned.

    — Mike Vizard

  • Testing and Certification in the Age of Automation

    Testing and Certification in the Age of Automation

    If we do not change the way we teach, 30 years from now we will be in trouble. Because the way we teach, the thing we teach our kids are the things from the past 200 years. It is knowledge based. And, we cannot teach out kids to compete with machines, they [machines] are smarter.

    We have to teach something unique so that a machine can never catch up with us.

    —Jack Ma, CEO Alibaba

    tl;dr synopsis

    Given the complexity of modern DevOps, technical certification is becoming more granular. Also, its use is exploding as a requirement for employment. Most technical certification is awarded by passing a test composed of multiple choice questions. Passing a test with multiple choice questions requires the test-taker to absorb a significant volume of knowledge for a short period of time. This acquired knowledge does not need to be retained. Hence, the certification can only attest to the bearer’s knowledge of a technology at the time the test was taken. And, essentially the test measures only the bearer’s ability to take the certification examine. Finally, testing by multiple choice does not ensure the certification bearer’s qualification in terms of required experience or higher-level thinking capabilities.

    Technical certification is valuable in the current commercial landscape. However, at some point, as machine intelligence is able to perform more of the activities presently done by humans, the format and methods by which certification testing is administered will need to change. Future certification testing methods will need to put greater emphasis on analysis and creative problem-solving rather than the current focus on knowledge acquisition and rote application. Also, what and how we teach humans will need to be unique, beyond the capabilities of artificial intelligence.

    Over the last 10 years, DevOps has become a much more complicated landscape. Regardless of the service provider of choice—Amazon Web Service, Microsoft Azure, Google Cloud, OpenStack, IBM, Joe’s Homegrown Cloud Service—you gotta know a lot to play.

    At a high level, service providers offer products that are surprisingly similar. For example, they all provide some sort of machine virtualization, storage services, database services, queuing technology, authorization service, provisioning technology and API management. But the way you manipulate the conceptual knobs and switches to get any of these products to work can vary dramatically between providers. There’s a lot of minutiae involved. Absorbing the amount of detail required to be productive in a provider’s product line takes time. And, ensuring that someone can actually work competently with a service provider’s products is a crapshoot. For example, a deployment engineer may understand the concepts behind dynamic provisioning, but getting a Kubernetes orchestration to work in Google Cloud is where the rubber meets the road. Doing is always much more difficult. As I said, there’s a lot of detail involved.

    So, how do companies ensure that the people they hire to fiddle with their multimillion dollar, mission-critical technology stack will deliver the goods without blowing things up? Most have an extensive technical interview process accompanied by a stringent reference check. Some just hope. Others require certification.

    The Rise of Micro Certification

    The notion of certifying that a person has the skills and experience required to be competent in a profession has been around since 1885, if not before. That was the year Massachusetts implemented the first written bar exam. Before that time, all a person needed to do lawyering was to get a note from a court saying the person could be a lawyer. That’s how it went for Abe Lincoln. Eventually the profession matured. Professional qualification was determined through a test-based approach. Early bar exams used essay questions. In 1972, with the addition of the Multistate Bar Exam, multiple choice questions were included.

    Going to multiple choice questions may seem like a trivial change, but it’s not. Whereas a knowledgeable professional is required to grade an essay question, multiple choice answers can be evaluated by anybody with the answer key. Hence, the popularity of multiple choice questions on the SAT. It’s the most efficient way to grade millions of high school test-takers each year. Evaluation is fast, cheap and a machine can do it.

    As I mentioned at the beginning of this piece, IT in general and DevOps in particular has become a place of increasingly granular complexity. There are a lot of technologies on the landscape. Each of those technologies has a significant learning curve to be overcome to achieve mastery. The conventional wisdom is that the best way to determine mastery is to test a person’s knowledge of the given technology and then measure the results of the test—high score, good; low score, bad. As with early testing practices, the fastest, cheapest way to make an evaluation is to administer a test made up of questions with multiple choice answers. This is sort of like the SATs, except most technical certification is geared to a particular vendor’s product line, not a general body of knowledge. (You can view a sample of test for AWS Certification as a DevOps Engineer Professional here).

    This type of professional testing paradigm is exploding. It’s infiltrating every corner of the technology landscape. And, more employers are requiring certification as conditions of employment, both for prospective employees and those already on staff. This would be OK, except for one thing: After you scrape away all the preconceptions and assumptions about the value of test-based certification, when it comes down to the bare-bones validity of test measurement, the only thing a test really measures is the ability to take the test. For example, a deployment engineer might be sitting for a certification test after having been up all night working to get the company back online after a system failure. The taker, who is exhausted, ends up failing miserably. Yet, some slacker just out of high school who has never seen in inside of a data center can take the test using stolen answers and pass with flying colors. Tell me, please: Who is really the most qualified professional?

    Testing by Regurgitation

    Certification is not the problem. After all, I really do feel more comfortable knowing that the surgeon about to open up my wrist to alleviate my carpal tunnel syndrome is certified by the American Board of Surgery. Rather, the problem is the way that certification testing is conducted. In my opinion, a certification testing process that emphasizes the multiple choice question-and-answer format is flawed. Even when questions are answered correctly, the test format does not really verify the competence of the taker, only that some facts are known. It’s just testing by regurgitation.

    Such testing requires the taker to absorb a lot of knowledge and then dump it out according to the test requirement at hand. There is no guarantee of long-term retention. It’s the age-old process of cramming for the test. Once the “time’s up!” bell rings, the chances are that the test taker will forget most of what she absorbed. Those with photographic memories might remember it all, but for most of us mere mortals, it’s a one-shot deal.

    Few people in tech work using the cram-it-in method. Most of us just use this thing called the internet to get the answers we need, when we need them. It’s what Bill Gates calls “information at your fingertips.” Yet, we still test for the knowledge in your head at the moment.

    There is a better way.

    A Better Way

    Certification is good provided it ensures that the bearer knows what he or she is doing. When certification is about nothing more than checking the right checkbox on an online test, there’s a problem. Put a monkey in front of a multiple choice test for eternity and eventually the animal will score 100 percent.

    So what do we do?  We do the following:

    Support Open Book Testing

    If the purpose of the test is to ensure that a person has a certain body of knowledge at the time he or she takes the test, make the body of knowledge required available at test time. It’s called open book testing. That’s how it works in real life. For example, you need a NodeJS package to provide mock objects. Do you rely upon your existing knowledge about the packages you know? Great, but what if you don’t know? Does work come to a grinding halt? No. You go to NPM to find one that meets your need. That’s how it is in real life. A competent professional is expected to have a baseline, high-level knowledge of his or her professional domain. For the day-to-day details, a pro has the ability to find accurate answers to questions quickly.

    Instead of having a certification testing process based on cramming, maybe we should eat the dog food we make in real life and go open book. There is little downside to researching what you need, when you need it. Besides, the test question created a week ago might be best answered by a new technology created today.

    Put the Emphasis on Essay Questions

    The actual mechanics of testing vary according to the level of cognition being examined. I wrote an article about this a few years back. The best way to test higher-order thinking—analysis, synthesis, evaluation—is by using essay questions. After all, what is a Ph.D. dissertation if not a very big essay question? Essays require a level of thinking that is hard to capture using lower-order test methods such as multiple choice. Essays reveal not only what the test taker thinks, but also how the test taker thinks. There’s a reason why your high school math teacher wanted you to show your work when taking an algebra test. Sometimes how you get the answer is just as valuable as the answer itself.

    A good certification test will provide a question format that allows test takers to illustrate their thinking along with the way they arrived at an answer. Thus, the need for a bias toward essay questions. Again, how you think is just as important as what you think.

    Put a Human Back in the Examination Process

    I’ll share an intimacy with you: I’m not very good at answering multiple choice questions. When confronted with one, I spend a lot of time trying to figure out what the question is really asking. You’d be surprised at the number of test questions out there that are vague and confusing.

    This was a real problem for me back when it was time to go to college and do the SAT dance. Fortunately, my alma mater focused more on my essays than my SAT score. Also, a lot the exams during my collegiate years were administered as essays or conducted as interviews with a professor. Later, when it was my time to be the Guy Awarding the Grade, my process was to evaluate students by interviewing them about one or many of the projects they did. I based my grade on the quality of the project itself and the thinking that went into the work. I tested my students by talking to them and reviewing their work. I like to think that I got a pretty accurate idea of a person’s competence, more than a multiple choice test could ever provide.

    Providing the opportunity for human interaction in the examination process can go a long way to providing the information required for making an accurate evaluation of a person’s competence.

    Require Testimony of a Candidate’s Experience, Character and Competence

    Think about this: It’s possible to be an AWS Certified Solutions Architect Professional without ever having done a day’s worth of paid, real-world work. All you need to have done is pass the certification exam. Yes, of course passing the exam is meaningful. It’s a grueling test. At the least, the test might prove that should be able to work the conceptual dials and switches that go with a given AWS service. But, as far as having the real-world experience, competence and character to play a critical path role in a mission-critical enterprise, some other instrument of certification is required. This instrument is sometimes called a reference—or, as the Wizard said to the Tin Man in “The Wizard of Oz,” a testimonial.

    As you read above, back in Abe Lincoln’s day, a letter from a court attesting the good moral character of the bearer was all that was required to practice law. You can think of good moral character as the ability to show up on time, do reliable work, demonstrate a commitment to excellence and get along with others. Such a description might sound melodramatic, but it counts.

    While it’s true that some certifications require that bearer to have a modern version of “good moral character,”  in addition to good test scores, many do not, particularly in tech. An employer wants to hire people who know what they’re doing and have the wherewithal to work effectively in an organization. Potential candidates want to provide evidence that they have what it takes to be a good employee. Building a testimonial of the bearer’s experience, character and competence into a technical certification will add a great deal of value for all involved. Also, the testimonial needs to affirm the bearer’s ability to adapt to change. In the not too distant future, your commercial value will not be about what you know today, but rather how fast you can work with technologies that will appear tomorrow.

    How to Win the Race

    For better or worse, we’re in a race with machines to keep our jobs. They don’t really want our job. They’re incapable of desire. But, that’s not to say they can’t have our job should the machine prove to be able to do it. Sadly, the current practice of technical certification is focused on passing tests concerned with the acquisition of knowledge.

    Right now, we’re winning the race. Machines still have problems with knowledge-based tasks involving advanced image recognition and reading comprehension. We still know how to differentiate between a dog, cat and an apple tree better than a machine can.

    But, it’s not always going to be this way. Keep in mind that talking machines were very primitive 20 years ago. Their voice had an artificial, staccato rhythm that was distinctly robotic. Today, it’s hard to tell the difference between a human or machine when a robocall disturbs your evening meal. It’s only a matter of time before the machine can do mid- to high-level knowledge-based work better than any human. Jack Ma’s quote at the beginning of this article supports this assertion.

    We need to change. The commercial value we’ll bring to the workplace in the future will be about analyzing, synthesizing and evaluating. Machines will be able to to the lower-level thinking faster and more accurately than we ever can. The realm of human excellence will be about better understanding the world around us and bringing new things into that world. The way we start is by teaching something so unique that the machines can never catch up. This thing is creativity. The paradox is that creativity can always be learned, but rarely be taught. Still, we must try. We have little choice not to.

    — Bob Reselman

  • Red Hat Rises to Data Management Challenges

    Red Hat Rises to Data Management Challenges

    Red Hat is moving to reduce or eliminate many of the manual tasks DevOps teams routinely encounter via a beta release of an update to Red Hat Enterprise Linux (RHEL) that addresses multiple data management issues related to DevSecOps and DataOps.

    Steve Almy, principal product manager for RHEL, said version 7.5 of RHEL adds support for integration between the Ansible IT automation framework that Red Hat now bundles in the RHEL operating system and OpenSCAP, a security auditing tool originally developed by the National Institute of Standards and Technology (NIST). OpenSCAP provides IT administrators with a baseline of security settings to compare with the current state of their Linux deployment. They then can invoke the declarative Ansible framework to automatically resolve any of those issues raised by the OpenSCAP audit, said Almy.

    In a similar vein, Red Hat is also making it easier to securely unlock Network Bound Disk Encrypted devices at boot-time while adding time-stamping and synchronization needs with the addition of failover with bonding interfaces for Precision Time Protocol (PTP) and Network Time Protocol (NTP).

    Additionally, to reduce data management headaches, Red Hat is now embedding support for data deduplication and compression software based on technology it gained by acquiring Permabit last year. The goal is to enable IT organizations to perform those tasks on data running on the server before it is ever stored on a disk, said Alby. In many cases, that may eliminate the need to acquire, deploy and manage separate data deduplication and compression software for the storage system. That approach also serves to indirectly benefit DataOps, an emerging discipline through which many organizations are trying to modernize the management of data and the pipelines employed to feed that data into applications.

    Other additional management capabilities being added to RHEL 7.5 include an easier-to-use cockpit administrator console for managing storage, networking, containers and other services, and a “known-good” bootable snapshot to help speed recovery and rollback after patching software.

    Almy said Red Hat is focusing more on eliminating much of the friction that gets in the way of enabling IT administrators to accomplish more tasks in a specific amount of time. The goal is not to eliminate the need for IT administrators, but rather to eliminate tasks that conspire to make managing IT more time-consuming than it need be. Much of that Red Hat strategy is anchored around an Ansible automation framework that doesn’t require IT administrators to know how to program to automate the management of underlying IT environment.

    It’s not clear just how much ease of data management impacts IT decisions. All things being relatively equal, however, IT administrators in the agile programming age are trying to manage more workloads than ever. Most organizations are not adding additional IT operations staff mainly because it’s hard to find and retain them. That means make sure existing IT administrators are more efficient than ever by embracing IT automation whenever and wherever possible to streamline DevOps processes.

    — Mike Vizard

  • The Case Against Human Employment in the Age of Automation

    The Case Against Human Employment in the Age of Automation

    A few months back, the small Japanese restaurant down the street from me was renovated. The owner redesigned the layout of his business to solve a vexing problem: How to stay in business when faced with a continuing pattern of employees not showing up for work.

    The layout of his restaurant went from this:

    Figure 1: The pre-automated local Japanese restaurant

    To this one:

    Figure 2: The neighborhood Japanese restaurant after automation

    The new layout eliminates the need for servers. It turns out, the people he hired to take orders and bring food from the kitchen to a customer’s table just weren’t showing up for work. We’re not talking one or two random no-shows; rather, it was habitual behavior among the workforce he hired, mostly high school grads and college students. Each day was turning into a crapshoot. He’d get a text an hour or two before the beginning of a shift reporting some other excuse: sickness, lost dog, whatever. Usually, the note closed with the inevitable, “Sorry, won’t be in today. See you tomorrow.”

    So he automated.

    Related Content: An Unanticipated Obstacle to Full Automation

    The new design is working out well. You come in. He directs you to a table. There’s a touch-screen monitor at each table diners use to place their order. The orders go directly to the kitchen, which is separated from a row of tables by a glass partition. When a dish is ready, the owner (who also happens to be the chef) or an assistant passes the food out to the diner’s table under an opening at the bottom of the glass (see Figure 2, above). At the end of the meal, the owner meets you at the front station where you pay the bill. No more wait staff. No more cashiers. No more “I can’t make it today” texts.

    Automating to Stay in Business

    Removing the servers and using automation was not an easy decision for the owner. His business is not some large-scale, multi-unit corporate enterprise looking to automate away human labor to increase profit margins. This guy has worked seven days a week for 10 years to build his business into a mainstay of the neighborhood. He prefers to employ humans but he can’t. He’s automating because the available human labor is unreliable. He believed that using such an unreliable workforce would eventually put him out of business. He really needed to make the change.

    So he did.

    No Employees, No Employee Retention

    So far, things are going well. However, now the kitchen staff is beginning to join the No Show Club, despite the fact that he is hiring kitchen workers at a rate of $14 an hour—a good wage for kitchen work. What’s causing people to go MIA? Maybe it’s the product. After all, how many California rolls can you make before going nuts out of boredom? Maybe he’s difficult to work for. And I doubt they’re jumping ship to work at Mickey D’s; the Golden Arches is only paying $10 an hour. In any event, the owner has a employee retention problem that, my guess is, he’s going to solve using automation.

    What Does This Have to Do with DevOps?

    At some point, every human being becomes unreliable. Some have poor work habits, but most simply are being human and humans make mistakes. That’s how we’re built. Some of us make a few mistakes, while others make a lot. But, we all make mistakes. Most of us learn from our mistakes, and we still make others. Our human nature to make mistakes makes us inherently unreliable.

    Machine intelligence doesn’t have this problem. It will get things right, eventually. And then, once operating reliably, it’s rare for that intelligence to revert to a state of unreliability. A human can become unreliable at any point in time—taking a sick day, resigning from the job or all the mistake-making that comes with learning a new skill or technology.

    We in DevOps have a lot in common with the owner of my neighborhood Japanese restaurant. Most of us have worked really hard to keep our customers happy and coming back, well beyond the typical 40-hour work week. For the most part, we’ve been successful.

    Our success increases demand. When demand goes up, the complexity of our work increases, as does the need for more reliable operations. But at some point, our inherent human unreliability kicks in. We make mistakes. To keep things going reliably, we automate. Then we move on to the next thing. Fortunately, for the foreseeable future, there will be a “next thing” to move on to.

    Still, intrinsic to our work is a motivation to eliminate human labor from the technical landscape. We have no evil intention. We just want to do the best job possible. Given the choice between a script and a human doing the work, we’re going to choose the script every time. The script will evolve to 100 percent reliability. The human won’t.

    As I said, for now, it’s not really a problem. There will be a next thing to move on to. And, in terms of my neighborhood sushi joint, maybe the out of work no-shows will get the message about the value of showing up at a job on time and ready to work. Or, maybe they won’t. Maybe as automation becomes the standard of labor fulfillment for repetitive work, immature workers will never be given the opportunity to get the experience they need to enable them to master basic employment skills. After all, they’re human and some point will become unreliable. So why hire them at all? Why not get a few computerized gizmos to do the redundant work and then focus on keeping customers satisfied and staying ahead of the competition?

    Having no employees translates into having no unreliable employees. A business without unreliable employees is an efficient business. Efficiency in business is what it’s all about.

    And this, my friends, is the case against human employment in the age of automation.

    — Bob Reselman