devopsy.kinsta.cloud

Tag: automation

  • Minor Crimes and Misdemeanors in the Age of Automation

    Minor Crimes and Misdemeanors in the Age of Automation

    Giving automation the power to detect crime and enforce punishment has ramifications, even for minor infractions

    In November, I broke the law. I crossed over a solid white line to make a right turn at a traffic intersection. At the time I was unaware of my violation. I was on my way to a shopping mall in an unfamiliar part of town to buy my wife a gift for her birthday. My only defense is that I was following the instructions emitted from the map app on my cellphone. It told me to make a right turn. So I did. Little did I know I was being watched.

    Automation enhances law enforcement capabilities

    A few weeks later, I received a citation in the mail. The citation had photos of me, my car’s license plate and the car crossing the solid white line. Turns out, there were cameras at the intersection watching all traffic activity. I was busted. I was also a bit shocked: My experience getting a traffic ticket 10 years earlier involved a real police officer writing me up in real time. Now it was as if Big Brother was indeed watching.

    In addition to the eerie feeling that goes with being charged with a crime anonymously well after the fact, I also felt baffled. The traffic citation did not list the amount of the fine I had to pay. The only information it provided was that I needed to go to the traffic court website to learn the details. So I did.

    I logged into the website and entered my citation number. The site had no record of the ticket, but noted that it can take up to 30 days for an automated ticket to make it into the system. I felt powerless; I was charged with a crime, liable for a fine and yet didn’t know how much my transgression was going to cost me.

    I decided to call the telephone number provided on the citation for more information. I dialed the number using the same cellphone that had gotten me into trouble in the first place. I was connected to an automated answering system. After navigating through various Yes/No and “enter the proper number” prompts, I learned that the system had no record of my citation. My only option was to wait 30 days for the systems to get their collective act together.

    A month later, a formal citation did indeed arrive in the mail, noting the amount of my fine ($480) and options for payment. Also, the citation provided instructions to go online and pay the ~$500 required to attend traffic school, should I so desire. (Traffic school is an option in California in which you attend a one-day class about traffic safety and your violation is kept private from insurance companies. The result is, your automobile insurance rates don’t go up. Here in Los Angeles, a lot of professional actors teach traffic school as a side job. As you can imagine, traffic school can be quite entertaining. But, I digress.)

    At this point, I was completely bewildered. I’d been busted for a crime I didn’t know I committed, by machines I didn’t even know were there. The only way I could get the details about my transgression was by interacting with other machines on the internet or on the phone. And, should I decide to plead guilty and settle matters, my first, best option is to do so via automation.

    Still, the technology was failing me. I needed to talk to a person, any person. So I went down to the courthouse in Santa Monica. There I found a special window on the exterior of the courthouse building for processing traffic inquiries. (One of the nice things about living in sunny Southern California is that providing service outdoors throughout the year is a viable option.)

    I waited in line. Finally, I got called forward to the window—which, by the way, was a pane of blackened glass. I had no way of knowing who or what I was talking to; all I saw was my reflection in the black mirror. A voice from the speaker at the bottom of the window asked me to state my business. I did. The conversation continued. The responses from the speaker seemed brusque, if not a bit preformulated. I started to wonder, Am I talking to a machine or a person? So I asked a random question, “How much is two plus two.” The voice in the speaker responded, “Sorry I am not programmed to respond to that information.”

    Now I was really thrown. Was I confronting yet another piece of automation? After a minute or two, the voice in the speaker said, “I’m just messing with you. I get this all the time. They all think I’m a machine.”

    It turns out that the voice did indeed belong to a human, and the human’s name is Dante Pipkins. Dante shared an interesting fact with me: Even though my violation was caught on camera, the actual OK to create the citation was issued by a human police officer behind the scenes. Go figure.

    Thus, having been enlighted about the details of enforcing traffic law via camera, I finished my business by having Dante give me a date to appear in traffic court, before a human judge, to plead my case. I knew there was no way to proclaim my innocence. I had been caught red-handed. My intention was to throw myself on the mercy of the court to get a reduced fine.

    The Power of Automation

    Automation is a great thing. It empowers us. It saves us time and money. Without automation, there’s no AWS, Google, Azure, Digital Ocean, Netflix, Etsy, Shopify, Facebook, Instagram, Slack, Airbnb, New York Stock Exchange … the list goes on. Automation is more than a technological feature. It’s now a way of life. And, for those of us who work in DevOps, automation is fundamental—none of us want to go back to the days of manual deployment and guesswork system monitoring, to say the least.

    Yet, we do walk a tightrope. As my recent travails with a very minor aspect of the legal system demonstrate, automation is becoming a key factor in everyday law enforcement. On face value, this might seem like no big deal, but consider this: Today, charging a citizen with a crime is still subject to some sort of human intervention. There are a lot of details to consider that go beyond the current capabilities of AI and machine automation. Human judgment is still the most reliable form of judgment. Yet, it’s time-consuming and expensive. Eliminating time-consuming, expensive tasks with cheap, fast, accurate machine-driven processes is a key motivator for using automation.

    As automation becomes more familiar with the activities it is processing, it becomes an expert. And, as automation achieves expert status, it replaces human activity. Just think about detecting credit card fraud; these days, practically all of it is done using automation. While we still have a long way to go in terms of totally automated processes, automation is creeping its way into law enforcement. My recent experience is proof.

    Now, imagine a world in which crimes such as tax evasion, financial fraud, embezzlement, regulatory non-compliance and yes, moving traffic violations are detected, reported and, most importantly, prosecuted using automation. Let’s go further: Let’s imagine that automation becomes the default method of prosecution. Imagine that the justice system becomes so automated that the defendant is presented with an offer to plead guilty to a sentence already predetermined according to an AI algorithm. Or, the defendant can choose to opt into a human interaction but will need to bear the burden of the increased court costs, very much in the same way that an automated audio transcription service charges extra for a human to do the work.

    Is such a scenario possible? Well, think about this: My interactions around the traffic violation were conducted without me ever having once had to interact with a human being. I was guilty as charged and automation proved this beyond a reasonable doubt. Not only did the technology detect my transgression, but it also connected me to my car, thus making citation possible. Hence, there’s a good argument that, when it comes to automation-enabled law enforcement, the distance from catching misbehaving drivers in the act to actually going after real crooks is just not that far a leap.

    So, what does this have to do with DevOps? When you scrape away all of the technological marvels, at some point there’s going to be someone in DevOps either creating, deploying or fixing the systems that make this all possible. It’s a big deal to give automation the power and technical wherewithal to enforce “the law.” As the feasibility of putting complex, comprehensive law enforcement systems into play increases, are we in DevOps going to consider such deployments as just another day in the CI/CD pipeline, or are we going to think through the game-changing impact of what we’re about to make?

    For now, I’ll leave it up to you to decide.

    — Bob Reselman

  • How to Automate HIPAA Compliance with DevOps

    How to Automate HIPAA Compliance with DevOps

    DevOps can help make HIPAA compliance more achievable

    Automating the provisioning of HIPAA-compliant server infrastructure enables compliant hosting service providers with the ability to provision and deploy infrastructure-as-code with minimal human intervention. Providing the automated process is thoroughly tested, revised and updated, it can offer a healthcare organization huge flexibility when enforcing a HIPAA-compliant server or serverless infrastructure within a private, hybrid or public cloud offering.

    Organizations that practice centralized operations, also known as DevOps, have a unique team who have the desirable skills of software development, IT operations and quality assurance (QA). The DevOps objective is to create a performance-based IT solution within a greatly reduced provisioning lead time using technical server blueprints within a continuous delivery environment.

    DevOps within a healthcare organization must be able to meet the stringent administrative, physical and technical safeguards required by HIPAA. In all cases, the required safeguards must be met, but with DevOps, it can be significantly easier to meet the addressable requirements of HIPAA.

    Planning

    Each healthcare organization can present differing business needs which they need a HIPAA-compliant service provider to resolve. It is not always possible to provide a cookie-cutter solution for every healthcare organization, often one design does not fit all; however, it is possible to create a baseline of operations using automation.

    A great deal of planning must be completed with all relevant stakeholders to not only understand the technical solution but to gather critical information about unique aspects of the architecture and then plan ways to automate the delivery process.

    It may not be possible to automate every workstream, but getting the DevOps team involved in the early planning stage will allow thorough and accurate playbooks to be created.

    With advanced planning, significant gains can be made in defining the networking security layout for the healthcare clients virtual private cloud, work out what isolated cloud resources can be defined and make decisions about potentially leveraging network layer protection services such as DDOS protection or a software-defined web application firewall.

    Provisioning

    The next step to creating the infrastructure-as-code is to write the automation playbooks that will provide the infrastructure. There are several provisioning applications that can be used, such as Terraform, Chef, Puppet, Google Deployment manager or AWS Cloud Formation, and playbooks will need to be written (or converted to) a compatible language such a python, YAML or JSON.

    The machine-readable code interacts with the service provider’s API front end; this may be an on-premises VMware vSphere cluster or a public cloud provider such as AWS, GCP or Azure. The API interprets the code and uses the descriptor commands to provision and deploy infrastructure in a cloud environment.

    A DevOps engineer will select the cloud provider they using and then define all the environmental variables to use to create the infrastructure-as-code. Connection variables are usually configured first and will define the cloud provider locations, regions and availability zones, as well as the encrypted account passwords or IAM configuration to interact with the provider API.

    Once the connection variables have been initiated with the provisioning software of your choice, they are used throughout the project life cycle. The next stage is to define the HIPAA infrastructure environment, including storage, network and compute resources. This will include provisioning resources and setting private and public networking.

    At all stages of the design it important to define where ePHI data will be stored and processed. The process must include step-by-step documentation and a customer responsibility matrix. At this stage, it is imperative to incorporate technical safeguards such as server access control lists (groups), introduce activity logging and provide auditable controls.

    Server hardening policies must be created for servers, active directory, workstations or providing least privilege authorization accounts at multiple levels, such as user profiles, that enable or restrict access to individuals or third-party systems. Preapproved software is typically installed at this step as well.

    Next, the inclusion of boot volume encryption and encryption of disk objects and storage volumes must be defined, this is easily achievable on mass provisioning of servers, simply providing a volume ID and an encrypted $TRUE statement.

    Throughout the playbook designing phase, testing is conducted to close loopholes, refine automated network configuration and create shared VPN tunnels with passwords that automatically cycle. The playbooks set the properties which make an organization HIPAA-compliant, and can also be used to create secured bastion servers if VPN is not an option.

    Playbooks allow DevOps teams to create rotating security keys, SSL certs and log aggregation and filtering. This can also be configured to trigger automated messages that can be emailed in case of any errors or failures in the automation process.

    Continuous Delivery

    When all the playbooks are streamlined and accurate, they can be used to define a healthcare clients service catalog, which enables the use of standardized templates to be used within the HIPAA environment. This may include server templates, container configuration, storage templates or any predefined software application.

    Security tools can be used to track and inventory resource usage and any system changes, track all user activity and any API activity requests.

    Monitoring tools can be used such as Cloud Watch, Nagios or Nimsoft to monitor server resources and application usage, these monitoring tools can also pick up errors or predictive errors and alert against predefined thresholds.

    Many organizations choose to use serverless workloads such as Docker or Kubernetes. These massively scalable solutions also bring about a huge benefit that aids the DevOps continuous delivery approach of automated update rollouts—essentially, applications can be isolated over differing pods, allowing the underlying hardware to be updated or versioned with zero downtime.

    DevOps is changing the automated delivery of HIPAA-compliant solutions. It is important to remember that DevOps does not have all the answers for HIPAA, especially many of the administrative regulation. However, it does empower managed service providers with the ability to provide tried and tested automated solutions that meet or exceed the technical and physical safeguards of HIPAA compliance.

    — Moazzam Adnan Raja

  • Red Hat Adds Support for Nested Workflows to Ansible Automation Framework

    Red Hat Adds Support for Nested Workflows to Ansible Automation Framework

    Red Hat today announced it is adding support for nested playbooks created using the open source Ansible automation framework within one another, which, in turn, will make it easier to automate complex IT processes.

    Bill Nottingham, product manager for Ansible, said the goal is to enable DevOps teams to reuse existing Ansible playbooks to kick off a series of nested IT jobs and workflows using version 3.4 of Ansible Tower, the console Red Hat provides for creating and managing Ansible playbooks.

    Version 3.4 of Ansible Tower also now makes it possible to slice an IT job into a series of smaller jobs that can be distributed across a cluster to accelerate execution. That capability makes it possible to scale out the processing of an Ansible playbook, he said.

    Finally, Red Hat also announced that Ansible Tower. 3.4 is now able to run on Red Hat Enterprise Linux in Federal Information Processing Standard (FIPS) compliant mode.

    Nottingham said Red Hat expects the ability to reuse Ansible playbooks within a set of nested processes will encourage IT organizations to create more playbooks now that it’s easier to reuse them as a set of modular components. IT organizations can even make launching jobs dependent on the completion of other jobs using a scheduling mechanism within Ansible Tower, he added.

    Although adoption of IT automation frameworks has been uneven, the last few years has seen IT organizations rely more on them to manage tasks as IT environments become more complex. That level of complexity will only increase as IT organizations deploy more applications that are based on microservices created using containers that need to be updated frequently. IT automation frameworks also go a long way toward making up for what has become a chronic shortage of skilled IT administrators. That capability is especially critical as both the number of applications and the rate at which they are being updated continue to accelerate.

    Most IT administrators have limited programming skills, which is why a declarative approach to IT automation in the form of an Ansible open source project has been gaining significant traction. In fact, thanks to automation, traditional IT metrics such as the number of servers being managed per administrator increasingly are becoming obsolete.

    It may take a little while longer before IT automation frameworks are the rule than the exception. But as the IT community now approaches the end of the second decade of the 21st century, reliance on manual processes that were developed at the end of the last century is no longer viable in this era of distributed workloads. In fact, regardless of where applications are deployed, the expectation in the DevOps era is that existing IT operations teams will be able to manage them without having to hire additional IT administrators.

    — Mike Vizard

  • Security and Speed: Why DevOps and Security Need to Play Nicely

    Security and Speed: Why DevOps and Security Need to Play Nicely

    It isn’t news that DevOps and IT security teams often struggle to align their departments and maintain a coherent balance between keeping a business secure and developing new applications to maintain customer interest. While security processes are a necessity, they can be deemed by DevOps teams to be manual and cumbersome, blocking the agility that makes them so effective in bringing their solutions to market. IT teams conversely feel their counterparts are prepared to sacrifice security in the name of innovation and revenue.

    Even if both teams do respect the other’s intentions, any conflict could lead to delays in both of their processes. For example, an IT team may need to make crucial updates to the network security and warn different teams they may experience some downtime during this crucial implementation. However, DevOps have typically been given more leeway in how they operate as they are so important in today’s software-driven world, and may ask for the update to be delayed so they can complete tasks or meet deadlines, leaving the IT team waiting and losing time rescheduling their own work.

    This has, unfortunately, led to a myth that DevOps teams choose to ignore security. In reality, developers are keen to know that their apps and the environment they work in are secure—but at the same time, they don’t want security to get in the way of them quickly delivering valuable new products and software features.

    So, is there a way for DevOps teams—one of the most important resources in many modern businesses—to embrace security without impacting agility? Can the integration of DevOps and security be done in a way that alleviates tensions and promotes collaboration, while actually improving both security and agility in the process?

    Yes. The secret is automation.

    Reconciliation Through Automation

    As C-suite executives are now more likely to focus on security, due to the obvious financial and reputational consequences of a breach, DevOps teams should define how they protect and secure their multiple projects and production environments. Automating security as part of the CI/CD process allows DevOps teams to easily follow company security policies because they will be embedded into the automation pipeline.

    This process can remain running with little concern, effectively minimizing stress about security. This still automates policy changes and activities so that there is a significantly reduced chance of error. Although the automation solution remains hidden, it can still be utilized at any point to view data on the vulnerabilities, compliance requirements, security policies and network connectivity, via its continuous scanning abilities.

    Additionally, DevOps teams are already familiar with automated tools in their daily operations and communications—and they are likely to be accepting of switching to a security solution that integrates with their existing processes.

    Automation is the key to creating reliable, effective and connected “DevSecOps” teams, as it makes the secure option the easy option. It combines DevOps’ existing use of automated tools to achieve their ultimate goal of continuous, on-time and on-budget deployments with security’s focus of reducing human error and maintaining continuous visibility into potential vulnerabilities.

    Encouraging Adoption

    A guiding principle of DevOps is collaboration, which is often equated with the idea of shared responsibility. To successfully embed security into the DevOps process, security teams and developers must work together and establish shared responsibility. But how?

    Some organizations may assign a security representative in each development team. This person acts as a pivotal link between the two teams, improving communication and building a balanced process that considers everyone’s mutual interests. A continuous flow of knowledge-sharing among both teams ensures a level of maturity that allows a business to secure applications and services with an automated solution.

    Security teams can begin to define “guardrail policies” that allow development teams to deploy continuously, with the caveat of having to obey security and compliance policies. This is critical for both teams. This new way of working means developers will be able to test their security posture at every step in the CI/CD pipeline and correct things when necessary, and security teams can comprehensively ensure security and compliance throughout the development process.

    Embracing Collaboration

    Any belief that there is common discord between DevOps and IT security teams is unfounded. While it cannot be denied that both teams affect each other, this is not due to conflict—it’s due to business needs. If the two teams work together, they can both achieve their goals and be part of a secure, innovative and profitable organization. The first step is to accept collaboration is a necessity and by embracing security instead of being concerned by it, DevOps teams can stay in control of how their needs work around IT teams’ processes. Then, an automated security solution can be deployed to improve the efficiency and outcomes of both departments—and, in turn, the entire organization. It’s time for DevOps to embrace DevSecOps.

    — Reuven Harrison

  • The 4 Keys to Effective Continuous Testing

    The 4 Keys to Effective Continuous Testing

    In the digital era, products no longer can win on capabilities alone. Speed to market and buyer experience now matter just as much to customers as the extent to which a product meets a pressing need or outperforms competitive offerings. Against this backdrop, software developers find themselves under increasing pressure, left with no choice but to develop apps that provide everything customers want, are delivered on time and are continuously updated, all while ensuring a user experience that’s never short of flawless. There’s simply no margin for error, and that’s why more developers than ever are turning to continuous testing.

    Unlike the traditional approach of testing performed at the end of the development cycle, where it can potentially cause production delays should issues or bugs be uncovered, continuous testing involves running tests throughout the entire software development process. That enables developers to identify and resolve existing issues immediately—and, more importantly, deliver continuously updated and always flawless applications to their customers.

    Fundamentally changing something you’ve always done a certain way is never easy, however, and continuous testing is no exception. There’s no single lever to pull, but rather a series of steps and best practices necessary to effectively implement it within your development team. Let’s walk through the four most critical of those best practices.

    Make a Cultural Commitment to Collaboration and Quality

    As with any transformational change, people come before processes and developing the right mindset comes before developing the right tactics. Effective continuous testing requires a teamwide, cultural commitment to collaboration and quality. The notion that developers own development and QA owns testing inevitably leads to the types of problems—namely, delayed releases and flawed user experiences—that businesses simply can’t afford in the digital era. Instead, developers and test engineers must have a mutual understanding of the baseline level of quality and efficiency they collectively intend to maintain throughout the development process and a shared understanding that testing is everyone’s responsibility.

    Apply a Broad Range of Tests, Anchored by Automation

    As mentioned earlier, continuous testing isn’t about pulling any single lever, but rather about applying a broad range of tests throughout the testing process, including unit testing, integration testing, automated end-to-end and regression testing and production testing. By continuously administering tests at various points throughout the delivery pipeline, you can identify and fix issues spanning functionality, performance and security, and do so before they morph into the kind of roadblocks that delay your release and cost your business money.

    Though you should strive for variety when it comes to the types of tests you implement, you should strive for consistency when it comes to the concept of automation. If you’re serious about succeeding with continuous testing, automation is a necessity. Automated testing, which involves writing a test script (typically in Selenium or Appium) that interacts with the functional elements on a web page without requiring human intervention, not only helps you deliver better scalability and reliability than manual testing, but it also frees your manual testers to focus on more value-add activities such as exploratory testing.

    Leverage Cloud for Breadth and Scalability

    Just setting up and maintaining the infrastructure, physical or virtual, necessary to test across the ever-expanding universe of browsers, operating systems and mobile platforms is both costly and complex. And that’s before you consider the need to scale your test environment to run tests in parallel. It’s a daunting challenge for even the most mature enterprise development team.

    Enter cloud. In many ways, continuous testing and the cloud were made for each other. By leveraging a cloud-based, cross-browser testing platform, development teams can offload all the infrastructure-related headaches they’d otherwise be forced to grapple with on a daily basis, test across virtually any browser/OS combination (leveraging both real devices and cloud-based simulators and emulators) and ensure they have the scale necessary to keep their test times to a minimum.

    Develop a Strategic Approach to Analytics

    Continuous testing, applied optimally, requires a strategic approach to analytics. Developers should actively look to integrate a layer of analytics within the testing process, such that insights can be applied across the product team. Smartly layered analytics can shed light on a number of functional issues, including which browsers and mobile platforms are most often experiencing failure and whether an application’s test results are consistent.

    Beyond just functional testing, teams also can leverage analytics to assess user experience and performance, examining, for example, how the user experience is impacted by heavier traffic loads and how overall application performance is impacted by new updates to the code. Having this added layer of analytic insight enables developers to better understand the entire quality picture and deliver improvements that can be applied both in real-time and to future tests.

    The Digital Age Demands Continuous Testing

    In the digital age, your ability to attract and retain customers is directly tied to your ability to build the best products, deliver them faster than your competitors and keep them continuously updated with all the latest features and function—and do it all within the context of an impeccable digital experience. It’s a tall order, but it’s one made considerably less daunting by the effective application of continuous testing.

    — Lubos Parobek

  • Revolutionize Software Delivery with DevOps and AWS

    Revolutionize Software Delivery with DevOps and AWS

    Technology is constantly evolving—every time we think we’ve got a complete hand-holding experience on some technological aspect, there emerges a better innovation.

    When the DevOps solution emerged, organizations were more than happy to incorporate its principles. But somewhere down the line, there came DevSecOps. Innovation is an upward curve that doesn’t see a decline but rather starts another journey of its own.

    And that is what most DevOps consulting service providers are striving to produce—innovation. Ever thought about getting a cloud service to operate and innovate a DevOps process? Well, that’s exactly what AWS does to innovate the DevOps process and services. Plus there’s more. That’s Innovation + Benefits!

    Benefits the Team ‘DevOps AWS’ Can Offer

    Automate – Automation is vital to the DevOps process and more so with the DevOps processes for cloud instances. It can drastically reduce developers and operators work hours dedicated to creating bug reports, generating scripts for specific functions, performing calculations and testing code. AWS helps automate DevOps approach including software testing, adding innovation, besides accuracy, consistency and repeatability. It’s human to err but it is technology that removes any scope for error.

    Scale – AWS was designed for scaling operations. AWS brings scalability to operations allowing you to do as many things as your businesses require. It facilitates easily availability of resources to fulfill the requirement at hand. The best example of this is handling traffic spikes and downtimes without the need for manual monitoring.

    Collaborate Better – Collaboration is at the very heart of the DevOps process. The unification was essential to make the development process a systematic and organized one. AWS can quickly accelerate this process and make it the way it’s intended to be. It unifies automation and enhanced operations to deliver the promises of DevOps. Gone are the days when different departments would wait on each other for communication; this paves way for improved software delivery.

    Deliver Quality with Performance – DevOps as a working principle and AWS suite of services are individual names of efficiency. So when they come together, it’s a powerful combination of quality, performance and remarkable output. Even before the output is out, developers and operators are rewarded with automated testing, continuous delivery and better infrastructure to work with. The entire process is knitted to facilitate a smooth DevOps process.

    Secure – Just as we discussed earlier, security has definitely become a part of DevOps. With AWS automation and managed cloud computing, it becomes easier to introduce updates and deploy them frequently. In turn, it makes easy to track changes in the code and ensure that security measures are enough and running correctly.

    Some DevOps-AWS combination services that enhance the DevOps process for cloud computing:

    AWS CloudFormation: DevOps-for-cloud professionals need to create and release cloud instances/services faster than general DevOps professionals. AWS CloudFormation increases the pace of creating cloud instances by creating templates of AWS resources such as EC2 instances and ECS containers to let you construct the entire setup without much manual intervention.

    AWS EC2: AWS EC2 brings security and management features to your DevOps process, making it a more the DevOps AWS duo a more desirable combination to work with. It can run containers in EC2 instances.

    AWS CloudWatch: This is an AWS monitoring tool that helps you track every AWS resource. Additionally, it enables DevOps experts to use third-party monitoring tools.

    AWS CodePipeline: This service is a popular AWS feature that eases the management of continuous integration and continuous delivery tools. It allows you to integrate tools such as CodeDeploy, GitHub and Jenkins so that you can visually control the flow of apps from scratch to the end.

    Not a service or tool but the next on the list is more like a functionality feature:

    AWS Instances
    AWS creates and adds new instances to its list. The flexibility to customize along with these instances makes it easy to use AWS DevOps together.

    Conclusion

    If DevOps is a solution, then AWS further enhances that solution to make businesses deliver their best. No wonder our world is being dictated by speed and quality!

    — Veritis

  • Reducing Poverty in the Age of Automation

    Reducing Poverty in the Age of Automation

    In 1966, more than half of the people in the world lived in extreme poverty. I don’t mean public housing, food stamps, subsidized school lunch, free health clinic poverty. I mean the type of poverty in which there’s no electricity, no running water, no toilets and open sewers, and most children never see a doctor or the inside of a classroom. Back then, people living in extreme poverty had a daily diet of gruel; not rice and beans, mind you, but mushy, flavorless, boring gruel. Violence was rampant. Infant mortality was high and living to 65 was more the exception than the rule.

    While it’s true that there were pockets of prosperity to be found in the Third World, for most people, if you didn’t live in Australia, Japan, North America or Europe, the odds were your life was not that different from a peasant living in the Middle Ages, except that maybe you had access to a TV or radio in the village square.

    Cheap Technology Contributes to Declining Rates of Poverty

    But things changed. In 1997, the number of those living in extreme poverty dropped to 42 percent of the people in China and India. By 2017, the number dropped again, this time to an amazing 12 percent in India and .07 percent in China. Now, this doesn’t mean that everybody in India and China rode around in Toyota Camrys and spent their evenings gazing at a 52-inch high-def TV, but it does mean that just about everybody went to school, saw a doctor and had some choice in what they ate for dinner. It might not seem like a lot, but when you consider that their great grandmothers could barely read a newspaper, it’s a big deal.

    This progress didn’t happen by magic, and there’s a good argument to be made that it didn’t happen because of politics. My opinion is that it’s due in good part to the proliferation of cheap technology brought about by automation.

    The Toothbrush as an Agent of Change

    Small pieces of cheap technology can make a big difference. Take the toothbrush, for example. Not that long ago, a personal toothbrush was a luxury item for many. Today there are machines that make them automatically by the dozens, for pennies in material cost. Anybody can buy one and those who can’t many times can get one for free. Think what it must be like for a 7-year-old child in a classroom trying to learn how to read while in the throes of dental pain. There wouldn’t be a lot of learning happening.

    Studies have shown that a leading indicator of good health in the long term is the practice of good dental hygiene on a daily basis. Giving a child a toothbrush and some toothpaste early in life makes a big difference. Children who brush their teeth regularly fare better. A toothbrush might seem like a trivial consumer good, but it is actually a high-impact technology that has lifted many people out of poverty.

    Faster Credit Reporting Means Less Poverty

    Another example of a small piece of automation that’s improved the lives of billions is credit reporting. Thirty years ago it took a few days to get the credit rating of a company, and getting the credit rating of a sovereign state could take longer. Today, getting a credit report takes seconds. Automated credit reporting accelerates financing, which not only makes it easier for a consumer to get a home mortgage or car loan, but also for sovereign states such as India to get large-scale, industrial goods such as bulldozers. More bulldozers translates into more roads; more roads lead to more commerce and more commerce leads to prosperity for more people. It’s a pattern that’s difficult to ignore. Just take a look at the economic growth that accompanied the expansion of the U.S. interstate highway system in the last century. In this century, the same growth pattern is underway as China grows its One Belt, One Road (OBOR) transportation infrastructure on a global scale.

    The Cellphone: A Tool for Prosperity on a Global Scale

    Which brings us to the cellphone. The first mobile phone cost $4,000. Today, you can buy a prepaid, throwaway one with service for around $50. Given the technical infrastructures required to build the phone and networking infrastructure necessary to provide service, selling a fully functional cellphone profitably at $50 is a near miracle. And, its impact goes way beyond a simple, “What time will you be home for dinner?” conversation. There’s a big difference between living in poverty and living in poverty yet having a cellphone. A cellphone puts a person on the grid even when living in a hut in a remote village in Niger. Having a cellphone means being able to solicit paying work. It means having access to simple bank services. It means being able to get medical care in an emergency. And, a cellphone can connect users with online learning resources that can enhance the educational experience for those living in parts of the world where access to secondary education is rare. It’s no wonder the Gates Foundation is focusing on programs that use cellphone technology to address issues in poverty stricken areas.

    Not only has cheap technology reduced the cost of cellphones to nearly free, it also has reduced the cost of dental crowns through 3D printing technology and international trade through automated navigation systems for ocean-going vessels. Cheap technology also has brought about free online college courses and built-in writing assistants for email clients that have had the surprising effect of giving people better command of the written word.

    The Fall of Global Death and Destruction

    Our technological achievements allow us to grow more food, educate more people and reduce the level of violence to an all-time low. Remember, 20 million Russians were killed in World War II. That’s 5 million people a year over from the beginning of hostilities in June of 1941 until the war’s end in August of 1945. We’ve not seen slaughter in such numbers since that time. Yet, news of today’s conflicts is broadcasted on a global scale, so it can seem as though millions are dying everywhere. But the actual number of people killed in armed conflicts today comes nowhere close to the number casualties that were the result of the world wars of the last century.

    As the Numbers Go Down, the Risks Become Greater

    Things are getting better for more people and we technologists have been significant contributors to these improvements. Yet, new risks are on the horizon that could stifle the upward trends. One of the biggest potential dangers is a worldwide pandemic. While cheap air travel has removed the barriers of international travel for more people, it also has made it easier for a highly contagious disease such as Ebola to spread faster.

    And then there’s always the risk of nuclear catastrophe. The number of nuclear weapons has decreased, but technology has made possible nuclear devices that are the size of a suitcase. Whereas it takes the resources of a sophisticated sovereign state to create and deliver missiles capable of a nuclear strike across continents, all that is required for a non-state belligerent to deliver a nuclear blow is a small device hidden in the trunk of a car driven across an unguarded frontier. It’s a remote possibility but a possibility nonetheless, and one that can have a catastrophic outcome of unimaginable dimensions. Fortunately, for better and worse, we in technology have created planetary surveillance systems that make it possible to guard against such an occurrence.

    Considering the Big Picture

    Technology grows in complexity every year, particularly for those of us in DevOps. The rate of innovation can be overwhelming. We’ve got to know more to do more. As such, many of us give most of our attention to the details that make the technology work. Sometimes we lose sight of the big picture. But, here’s something to consider: As weird as it may sound given current perceptions, there’s a good argument to be made that, given the increasing rate of technological innovation and the reduced costs of mainstream technology produced by such innovation, we might eradicate extreme poverty in our lifetime, if not the lifetime of our children. In the past such thinking was considered grandiose. But, as the $50 cellphone demonstrates, it’s a goal we can look forward to achieving.

    Authors note: The source of the data on the reduced rates of poverty worldwide described in the opening of this article comes from the book, “Factfulness” (pages 52-53) by Hans Rosling , Anna Rosling Rönnlund and Ola Rosling.

    — Bob Reselman

  • Toyota Research Institute Discusses DevOps Automation and Rapid Innovation

    Toyota Research Institute Discusses DevOps Automation and Rapid Innovation

    How TRI is using DevOps automation to drive its research and engineering

    Toyota Research Institute (TRI) is working to build a future where everyone has the freedom to move, engage and explore with a focus on reducing vehicle collisions, injuries and fatalities. Working tirelessly behind the scenes is TRI’s Infrastructure Engineering team, responsible for designing, deploying and maintaining the infrastructure that makes this possible.

    I recently sat down with Mike Garrison, the technical lead for Infrastructure Engineering at TRI, to talk about DevOps automation and cloud-based deep learning. (Full disclosure: TRI is a customer of Flux7)

    Note: Mike Garrison and Adrien Gaidon, Machine Learning Lead at Toyota Research Institute, will share more about their experience with Distributed Deep Learning & High Resolution Driving Data at this week’s AWS re:Invent.

    Can you start by giving us a broad idea of the project you’re working on at TRI?

    Our mission at TRI is to improve the quality of human life through advances in artificial intelligence, automated driving and robotics. The Infrastructure Engineering team supports the researchers and engineers at TRI by making it easier for them to utilize the power of the cloud in a secure, automated and reliable fashion while not slowing them down. We see ourselves as partners who can help reduce the time and burden, so they can focus more on their deliverables.

    Have you evolved as a team to enable the rapid innovation that this kind of project needs?

    Yes. The biggest change we’re making is an evolution to embrace more DevOps ideas, methods and processes.  Automation is a core tenet of DevOps. We are applying that in two ways. First, to reduce tactical, manual IT operations activities; second, to make it faster and easier for our researchers to test their ideas.

    For example, we are creating a self-service portal that will allow researchers to easily and quickly provision the AWS assets they need to test new ideas. This helps researchers become more productive as they won’t wait for the infrastructure team to spin up resources. Having a secure cloud sandbox environment enables them to try new ideas, fail fast, destroy the sandbox if needed, and start over. It’ll enable our researchers to innovate at velocity and at scale. Modern cloud infrastructure and DevOps automation is empowering us to quickly remove any barriers that get in the way, allowing the team to do their best work, advance research quickly, push boundaries and transform the industry.

    How are you applying the latest technology advancements to help produce cars that are safer, more accessible and more environmentally friendly?

    The process of developing autonomous vehicles is data intensive. At TRI we log a massive amount of data—terabytes of data per car per day—from our fleet of test vehicles. Researchers analyze the data and continuously refine machine learning algorithms to ensure the most accurate outcomes possible across a myriad of driving scenarios. Researchers train new models and update test cars for new test runs. We’re also using simulation to virtually recreate traffic situations, adding adverse conditions such as bad weather and difficult traffic patterns. It’s a process of continuous improvement that simply couldn’t be achieved without the high compute power of the cloud.

    Supporting this high-powered data analysis is a cloud infrastructure that has been optimized to minimize the amount of time researchers take in this gather, learn, tune and test process. For example, at TRI, we have two driving technology modes—Guardian and Chauffeur. Guardian is designed to intervene only when a driver needs help avoiding an accident, whereas Chauffeur is meant as a truly autonomous vehicle. We developed a single technology stack to support both modes.

    And, using the PyTorch deep learning framework, my teammates on the machine learning side developed a series of computer vision models that monitor and control both modes as well. Working together, we are able to collect information from their deep learning models, spinning up new compute and storage resources on demand. We’ve married this elasticity with advanced management and orchestration services to ensure that collected data can be retrieved and analyzed quickly. Through advanced automation, we are able to shrink model training times so they are measured in hours—or even minutes.

    What lessons have you learned along the way? Any barriers or surprises that others could learn from?

    While DevOps automation requires more upfront work, it pays dividends in time savings, staff being able to focus on strategy rather than tactics, and provides logged history which can be critical for security, problem resolution and much more. If I had any words of advice for others looking to adopt DevOps automation, I would advise them: Don’t try to go back a year or two later and try to reverse engineer what was done. Start on the right foot and automate as much as you possibly can.

    — Aater Suleman

  • Stop the Domino Effect by Joining Build Engineering and DevOps

    Stop the Domino Effect by Joining Build Engineering and DevOps

    As DevOps has become more common, these teams are tasked with uniting everything from coding to deploying and monitoring applications. Build Engineering forms the foundation for all those processes and is essential for building code. Its tools vary according to the operating system you’re building on and by programming language. DevOps needs to work with Build Engineering, but the two don’t usually connect. And that’s a problem.

    Over the last 10 years, there’s been a shift toward DevOps. Previously there was a discrete division between coding, done by developers and engineers, and deployment, done by Ops. However, it became apparent that for companies to optimize benefits such as productivity, development and operations shouldn’t be so disparate. The result of which is DevOps.

    Along with the shift to DevOps have come several trade-offs—namely trading security for speed, certainty for agility and proprietary code for open source:

    • Speed: Getting to market quickly has become critical to success. Speed now takes greater priority over fixing vulnerabilities, which can always be addressed in the next release (hopefully). And retrofitting for security and vulnerabilities after the fact becomes a big blocker for development and engineering teams. At the end of the day, the risk of lost profits is greater than the risk of potential unknown threats.
    • Agility: Agility trumps certainty in the race to market. For instance, iterative pushes to production do away with fixed, known road maps in favor of implementing whatever this week’s most important feature/functionality is. The trend toward immutability is also important here; it’s generally simpler and quicker to throw away something that’s broken or outdated rather than trying to fix or update it.
    • Open source: Due to the greater innovation, reduced costs and transparency it delivers, open source has won over proprietary code. Rather than focusing on creating proprietary code, developers can quickly iterate on a solution by assembling open source code, accelerating time to market. Plus, the ability to take from what has already been created can free up your dev teams to innovate and build IP for your enterprise efforts.

    For most, the tension between fixing something and knowing if it will break something upstream hasn’t been eliminated. You may be able to quickly update and patch a single solution, but how will this impact you once you push to your CI stream? Even for mature DevOps organizations, there’s a hidden opportunity cost associated with two factors that are overlooked: open source programming languages and Build Engineering.

    Overhead is added to your DevOps teams when you retrofit your open source language of choice with new versions, dependencies, security patches and so on adds. Teams should be focused on the things that matter—creating and delivering the latest release.

    On top of that, Build Engineering takes a hit each time the underlying open source language is retrofitted. You need to rebuild all your developer environments, CI/CD systems and production environments. All of this build engineering work is manual, slows down time to market and eats up valuable engineering resources. Plus, you’re not certain your rebuild and update will provide performance improvements that warrant the update.

    Since DevOps already relies on automation, it makes sense to apply this tool to the retrofitting and building of your open source languages and seamlessly connect to the rest of your DevOps cycle. You could ensure build reproducibility, dependency management and compliance with your corporate security and license criteria. For example:

    • Build impact –  What if you could understand the performance improvements (if any) that could be gained by taking the latest build?
    • System update – Consider the benefits of being able to update all of your impacted systems across dev, test and even production environments with the latest build using a single command.
    • Packaging – If each continuous build could be packaged in multiple ways—any form factor for any operating system—how would it change your workflow?
    • Build validation – Imagine that every continuous build could be vetted against your smoke test, inclusive of compliance with your security and license criteria, so you knew whether it was ready to deploy.
    • Continuous builds  – What if every time a new version of a library or patch for a vulnerability was found, a new build was kicked off and you were notified when it was done? Then you could keep up with your continuous deployments.

    The domino effect is a phenomenon that DevOps must deal with on an all-too-often basis. Dev and Ops came together for greater productivity, but unforeseen dependency issues regularly gum up the works. That’s the reason automation is such a useful tool, one that you can used to stop the domino effect as you automate your build pipeline for open source languages. It’s truly possible today to benefit your business by ensuring build reproducibility, dependency management and compliance with your corporate security and license criteria.

    — Bart Copeland

  • Image Management 101: The Web Developer’s Guide

    Image Management 101: The Web Developer’s Guide

    How understanding the user experience can help web developers in managing images effectively

    Online experiences have in many ways supplanted in-person experiences. Today, no one would hesitate to buy a luxury watch online instead of from a jewelry brick-and-mortar store. But as these online experiences become the norm, user expectations for a seamless and “in-person like” customer journey have increased as well. When customers buy that watch, they expect to see every gleam of precious metal and stone that proves its worth before they add it to their shopping cart. This can only happen with the highest quality visual imagery that kindles an emotional connection.

    But getting these high-quality images from the photographer’s camera to the shopper is a bigger challenge than most understand. Web development teams often receive raw photographs in obscure formats, or high-resolution files designed to preserve image quality to nearly microscopic detail; but these files will undoubtedly impact site performance due to sheer size.

    Speed and quality are often at odds in the web development process as companies embrace the need to provide richer and richer online experiences. Web developers need to master the best ways to deliver site performance while balancing marketing needs.

    Optimizations Based on User Experience

    Developers need to understand how image optimization can impact the user experience. No one disputes the goal of delivering images with the best quality possible to end users, but the part that is challenging is how subjective and wide the range of “best quality possible” can be.

    First, it’s important to note that quality can vary depending on whether a content delivery network (CDN) is used to deliver images, the connection speed and device used. For example, an overclocked gaming desktop with a hardwired, high-speed internet connection with images served from local servers is going to provide a drastically different experience than a smartphone overdue for an upgrade on a roaming signal. Also, delivering desktop images to mobile devices can result in site latency and user frustration due to over downloading. On the flip side, delivering images intended for mobile devices to desktops will result in pixelated images that will displease viewers.

    As for devices, different screen sizes mean different viewing experiences. A wide shot on a website viewed on a computer screen will show more detail than a 2-by-5-inch phone. For example, a scenic shot of a hotel with a beach in the background may clearly show the hotel name on a computer screen but be too small to make out on a mobile device. But since more and more people are using mobile devices to access digital content, it will benefit web development teams to ensure images serve their intended purpose with focus points and zoom capabilities.

    Delivering Based on Insight

    Once developers know their objectives in terms of user experience, they need a plan to deliver this experience at the highest possible quality. As mentioned above, the highest quality image does not necessarily translate into the highest quality experience; therefore, it’s important for developers to configure their delivery systems according to varying user needs. Developers can start by collecting accurate information about their users and how they access content. In addition to network conditions and screen size, the user experience can be impacted by what browser, operating system and device visitors are using. Collecting real user data can provide the insight needed to deliver the right quality and size image to the right user.

    To make matters more complex, many browsers and device environments have native image formats that differ from standard formats such as .png or .jpg. For example, Apple computers and devices support JPEG 2000, while Microsoft has a proprietary format called JPEG XR. These formats offer better image compression for delivery, meaning that web teams can use higher quality images without sacrificing faster load time or bandwidth. However, the fact that they are limited to specific environments requires web teams to create all the variant derivatives, store multiple copies and develop the logic to deliver the correct format to the appropriate browser and operating system.

    Automation All Day, Every Day

    Web teams and developers can benefit from automating their image optimization decisions given the challenges they face when delivering images, such as various combination of internet speed, screen size, device performance, operating systems and browsers. Automation to CDN cloud providers can also offload the decision-making logic (which can be CPU heavy) away from the main server, speeding up overall response times.

    Developers also need visibility into how images load and display on user devices. This means their job is not done after delivery. Having this level of insight will help identify bottlenecks and prioritize images for optimization. Automation services that have this option will be best positioned to help web teams.

    Web developers are faced with increasing pressure to meet aggressive marketing priorities. Quality and performance may seem at odds, but by taking stock of all factors that go into image requirements, web teams and developers can build websites that deliver stunning image-driven user experiences without sacrificing quality.

    — Charles Choe