Author: David Geer

  • Q&A: BDO’s Coffman on Change Management, Security and DevOps, Part 2

    Q&A: BDO’s Coffman on Change Management, Security and DevOps, Part 2

    As we covered in part one, DevOps’ best asset and biggest risk factor may be one and the same: swift and precious change. In part two, we continue our discussion of the challenges and solutions with Deena Coffman, managing director of BDO Consulting’s Technology Advisory Services practice.

    David Geer: What specific tools and technologies can help DevOps teams to address change management in these ways?

    Coffman: There are a variety of tools and technologies to address some of the inefficiencies in change management, allowing DevOps teams to automate most of the process. For example, teams can leverage change management software to build pre-authorization processes and workflows to speed up change management approvals. On the security side, infrastructure monitoring tools will alert an organization when a discrepancy is discovered on the network, providing application visibility to the users and monitoring for unauthorized changes. DevOps teams should also consider using a collaboration platform like Slack to improve communication via virtual meeting places.

    Geer: What will these tools accomplish and how should teams be using them?

    Coffman: These tools are only useful if used properly. Deployments that require change management approval can go through the automated workflows. If a change requires approval, developers can use Slack or any other collaboration platform to contact the required party and explain the reasoning behind the modification. DevOps teams must also take the time to identify and automate as many “standard” changes as possible to reduce the reliance on change control meetings to speed up their deployments.

    Additionally, DevOps teams need to consider using an infrastructure monitoring tool to ensure all changes to the production environment are known and authorized. If a change finds its way to production without getting the appropriate authorization, the monitoring tools can detect it and report it to the team so they can rectify the issue.

    Geer: What kinds of things do DevOps teams need to do to address security to the satisfaction of the business?

    Coffman: The development team and security do not need to be at odds. By working in unison, both sides can reach their objectives.

    The InfoSec team would help the development team by making efforts to reduce the latency of security projects. The development team would help not only the InfoSec team—but potentially the entire organization—by preventing an expensive and embarrassing security incident by diligently supporting tools that automate security checks.

    Additionally, the build process needs to run unit testing, integration testing, functional testing, regression testing and information security testing before the code reaches QA. Security testing is often insufficient or not performed at all. DevOps teams need to understand that security testing performs an important function of catching issues early so as to avoid expensive, embarrassing issues downstream.

    Geer: What specific tools and technologies can help DevOps teams to address security in these ways?

    Coffman: Several tools are available to help DevOps teams address security in many different capacities, including setting up continuous integration/continuous delivery processes, automating server configuration, automating UI testing and functional security tests, testing the security requirements of applications and automating the vulnerability scan of applications and infrastructure.

    Geer: What will these tools accomplish and how should teams be using them?

    Coffman: DevOps teams should integrate these tools as part of their continuous delivery process. As soon as a developer checks his code in, the continuous delivery system should run the functional security tests, non-functional security tests and the security scan to ensure that the developer did not introduce a security vulnerability to the application and the infrastructure. These tools are only as powerful as the test scenarios they run. Therefore, teams should write exhaustive test cases to ensure the security of the application.

    Geer: How can teams do all these things and still maintain all the benefits and expectations that people have of DevOps itself?

    Coffman: By deliberately working toward improved collaboration and communication, teams can take advantage of the integration and efficiency benefits of a DevOps structure without sacrificing security or quality. Technical knowledge is important, without a doubt. To be exceptional, though, teams need equally strong teamwork and communication skills to actively support all important functions in the organization.

    — David Geer

  • Q&A: BDO’s Coffman on Change Management, Security and DevOps, Part 1

    Q&A: BDO’s Coffman on Change Management, Security and DevOps, Part 1

    DevOps’s best asset is swift and valuable change, which also may be its biggest risk factor. DevOps certainly challenges change management and security. We recently discussed these challenges and the solutions with Deena Coffman, managing director at BDO Consulting’s Technology Advisory Services practice. Here is part one of that exchange.

    David Geer: What kinds of challenges does DevOps create for change management? Can you illustrate with an example?

    Coffman: By combining the sysadmin and developer functions, DevOps can lend to “shortcutting” the change management process in the interest of expediency. It may also undercut the value of having a critical eye from both a distinct system administrator and a distinct development reviewer. DevOps is designed to move quickly, and while the pace of change is accelerating, speed is often an ingredient for a security incident. Much like having the same individual who approves invoices also issue a check, DevOps makes for a faster, but not always secure, process.

    When the system of checks and balances can be purposefully circumvented, that privilege tends to be invoked more often than necessary. As a result, developers may bypass change management altogether far too frequently. The collaborative benefits from DevOps integrating typically siloed functions such as QA, software development, security and information technology operations are important, so long as they aren’t offset by compromises in quality and security that result from the reduction in the review and approval process.

    The expedient nature of DevOps can also lead to error, given the absence of human experience involved in the evaluation process. Before DevOps, if a developer needed to change a setting on the web server, they would submit a change request and wait for approval. Under a DevOps structure, though, the developer simply scripts the change and commits it to the repository—automatically kicking off the build process. The process runs unit testing, integration testing, functional testing and regression testing against the new code, and then pushes the new code to the staging environment and finally, to production.

    Geer: What kinds of challenges does DevOps create for security? Can you illustrate with an anecdote?

    Coffman: The biggest security challenge in a DevOps environment is that speed becomes paramount, which reduces critical evaluation on security risks. Developers incentivized to quickly deploy new capabilities are naturally pushed toward productivity and efficiency, which often run counter to security. DevOps can skew that balance so that security is only lightly or sometimes insufficiently considered. Automated testing will test for known exposures, but a trained, experienced human may be able to anticipate a risk before it materializes.

    Geer: How can flaws in change management in DevOps approaches lead to security issues?

    Coffman: Flaws or inefficiencies in the change management process reinforce the views of those who are prone to bypassing what they perceive as tedious process requirements. If a change management process is too burdensome or seen as lacking value, people will avoid it at any opportunity—not just when an exception permits a bypass. Since technology professionals tend to have a bias against waste and inefficiency, introducing a process requirement that consumes time and effort without returning value has a high probability of triggering a wholesale negative reaction.

    Geer: What kinds of things do DevOps shops and teams need to be doing to address change management so that they continue to meet the expectations of the business?

    Coffman: When it comes to ensuring that change management meets business expectations, it really comes down to several key ingredients: flexibility, speed, efficiency and risk management.

    DevOps teams should design and manage the change management system to be efficient, responsive, and flexible enough to accommodate cursory reviews for low risk scenarios and careful, in-depth reviews for other changes. Leadership needs to also set a tone of support for the change control process so it is not dismissed or marginalized. Additionally, engaging outside expertise and/or including additional quality and security measures for higher risk projects are recommended.

    A risk review that identifies projects with security and privacy risk and distinguishing types of changes supports the automation decisions and builds a DevOps program that balances its benefits with risk management.

    Stay tuned for part two.

  • Sounding the Death Knell for Agile: Not so Fast!

    Sounding the Death Knell for Agile: Not so Fast!

    Some say DevOps is the next step in Agile’s evolution.

    Software development is changing rapidly; if you don’t change with it, modern development may leave you behind. DevOps connects information technology operations with development and vice versa; this is a big change for the art of software creation. Agile was not designed with this new level of focus on information technology operations in mind, and must change to include this focus to thrive.

    According to Tony Higgins, CTO of Blueprint, the hot new trends that are supposed to replace the methodology actually are extensions of it. DevOps, for example, aims to extend the collaboration happening between development and QA teams to the operations people who are running the production infrastructure. “It’s the next step in the Agile process,” he says.

    Survey data correlates with assertions it is growing in large enterprises even as DevOps is expanding. According to VersionOne’s 2016 State of Agile Report, more than 24 percent of respondents who use the methodology work for organizations with more than 20,000 employees, compared to 21 percent last year.

    Addressing the State of Agile

    Some enterprise software development stakeholders are shifting their sentiments by moving into enterprise Agile from software project-based/development side-only Agile.

    To some, the original creed should be unwavering, even though it addresses only software development and not information technology operations. “In an ISV context, which is where Agile ‘grew up,’ this understanding did really well. But now with [it] making rapid inroads into information technology, the context is very different and, in a nutshell, the manifesto is insufficient,” says Higgins. The needs of information technology operations such as those that DevOps is meeting are driving new frameworks so it also can morph to include a Dev and Ops methodology.

    “Rather than dying, Agile is transitioning from a competitive differentiator to a core competency,” says Higgins. Whether as part of Agile, what the industry calls DevOps or some new methodology, development is accelerating by building new features and innovations in small, rapid, incremental trips through the pipeline; that approach is originally a core concept of Agile. “Industry folks declaring it’s dead don’t realize that businesses have not given up on it; they’re just looking for better ways to do it,” he adds.

    Agile is Growing

    According to Higgins, the methodology isn’t something that just “happened” in 2001 when its manifesto came to into existence. “Software development organizations were practicing and evolving the various principles of Agile all along and the manifesto was really a culminating event that codified it,” Higgins explains.

    The methodology is spreading much like DevOps, appearing in executive groups such as marketing, sales and information technology. “In marketing, they’re time-boxing their work into iterations with tangible outputs/deliverables. You’re seeing regular ‘stand-ups’ happening with the inside sales group. Agile is making fast and strong inroads into information technology, driving tighter collaboration with operations, and enabling faster delivery,” says Higgins. It is proving to be a repeatable business model that applies to every part of the organization that cares to use it.

    Proof it is a core competency lies in the fact that technology must now be able to support it. “Analysis tools that help determine requirements must be able to do so in an Agile environment,” says Higgins.

    Enterprises are approaching the methodology in better ways as organizations are seeing that requirements are keys to its effectiveness. “Requirements often have been overlooked in Agile processes, but recently we have seen research firms (Gartner, Forrester) and the market shift their focus to requirements to improve their deployments,” says Higgins.

    Both Agile and DevOps are useful in extending the collaboration of development and QA not only to operations people but ultimately also to other departments. “Instituting DevOps, continuous integration tools or any type of automated deployments will inherently require collaboration between departments; this must actually extend well beyond just Dev, QA and Ops and to the rest of the business to ensure that end-to-end visibility is achieved and decisions are not made in isolation,” says Higgins.

  • Vivint Meets Smart Home IoT Challenges Using DevOps

    Vivint Meets Smart Home IoT Challenges Using DevOps

    Smart home technology vendor Vivint  serves more than 1 million U.S. and Canadian consumers with services including broadband internet, cloud storage, home automation products, home security services and energy automation controls.

    The company, which got its start in home security, leverages door and window sensors, motion detectors and glass-break sensors, as well as those that detect flood, heat, freezing, smoke and carbon monoxide in the home, said Nick Brown, director of Platform Architecture at Vivint. “We have a bunch of indoor and outdoor cameras such as doorbell cameras and the ping camera that is an interior camera supporting two-way audio.” For the smart home, Vivint extends automation devices, intelligent thermostats and Amazon Echo into the household.

    Vivint used DevOps to get the software platform behind these sensors and smart home devices up and running within a year. Vivint continues to use DevOps to meet the needs of its growing platform.

    Building a Software Platform for the Smart Home

    Vivint started its expansion from home security company to the largest smart home provider with a challenge of growing a platform that could serve a million customers. This was necessary to fold in the 700,000 home security customers it already had in with the 200,000 new smart home customers its leadership knew were coming in a matter of months.

    “We were building this new infrastructure from absolutely nothing and we needed to support 200,000 new installations in the first few months of operation,” Brown said. Vivint had its existing home security customers on a legacy platform supported by a third-party company; that company had a full 10 years to build out its infrastructure. Vivint had one year to bring those 700,000 existing customers into a new system and less time than that to prepare for the influx of new customers.

    Vivint accomplished its goal, serving more than 900,000 homes in the first 12 months.

    What DevOps Now Accomplishes for Vivint

    DevOps enables Vivint to manage stateful and historical data at scale, Brown said, which allows the company to power the biggest smart home platform in the world. Customers see everything that is going on in their house, including current and 45-day history of what happened, through a 7-inch control display.

    The real value the Vivint platform provides is in the integrated in-home experiences, which the company virtualizes and makes available anytime, anywhere around the world, giving customers a real-time view into their home data, he noted.

    To build, use and evolve this platform, Vivint uses MongoDB, RabbitMQ, Nginx, ELK (once the ELK stack, now the Elastic Stack), Salt and homegrown tools. Vivint applies its homegrown code to the task of guiding a communication protocol that enables the smart homes and the Vivint cloud to talk to each other.

    The company uses RabbitMQ, an Erlang language-based open-source message brokering tool that supports Advanced Message Queuing Protocol (AMQP) to disperse those messages in and out of its software platforms, driving them to and from Vivint’s services in its internal infrastructure. Vivint uses Nginx, a multifunction web server/reverse proxy server to serve web applications and services.

    Vivint runs an ELK stack, which consists of Elasticsearch, Logstash and Kibana for monitoring. Its ops team uses Salt, a Python-based configuration manager to manage everything. “We’re looking forward to adding Kubernetes as a new infrastructure solution to make this a little more modular,” Brown said.

    The DevOps Path to Modularity

    Vivint needs its software platforms to be as modular as possible. “Each piece of the infrastructure must support a modular paradigm. Because we are following a high-risk, urgent schedule, we need to make sure that any mistakes we make along the way are easy to tear out and replace,” Brown noted. Vivint uses virtually countless services to make this possible while pushing out weekly software releases.

    “From a DevOps perspective, we have a lot of things to manage here,” he said. Vivint uses AWS to support its infrastructure in the cloud and the company has its own private cloud as well. Vivint needs to move safely and quickly while retaining its agility and orchestrating all these different elements.

    To bring all the smart home technologies into the new system, Vivint chose MongoDB because its data models and code tend to naturally mesh with the data models in that database. “We took advantage of the flexible schema, and the document-oriented architecture of Mongo really allowed us to write very little data access code,” Brown said. This significantly cut back on implementation time.

  • Case Study: Stytch Before, After DevOps Culture, Tools

    Case Study: Stytch Before, After DevOps Culture, Tools

    Cloud analytics provider Stytch leverages its cloud-based analytics for customers who want to centralize, fuse and model their data. “Our mission is to make it easier and faster for business users to get insights and make decisions,” said Taylor Frost, cloud operations engineer at Stytch.

    Before the company implemented a DevOps approach, its primary software platform and some services were large monoliths maintained by disconnected teams. Stytch tested new features in isolated Jenkins jobs and performed quality testing and product releases manually in siloed realms that were out of sight and out of mind, Frost said.

    Stytch’s Dev, Ops and QA were walled  from each other, isolating the knowledge base of activities in any one group from members of the other two groups. Stytch teams did not share this information, and sister teams could not benefit from the insights this knowledge held.

    What’s more, developers did not have transparency into the products and tools that manipulated the cloud-based infrastructure or the build pipelines. “This fostered an attitude of ‘it’s an (Ops/Dev/QA) problem’ and discouraged a sense of ownership of or responsibility for success, which led to an increasingly lengthy feedback cycle and slower, less frequent builds,” Frost noted.

    Stytch lacked the orchestration and the down-in-the-weeds level of control it needed to enable real-time changes to labyrinthine back-end technologies without repercussions that could cause the customer base to suffer, and to speed and simplify deployment, she said. Standards such as Terraform or CloudFormation did not provide the fine-grained control of deployment that Stytch needed.

    The company’s Ops was in charge of Python software libraries, but other groups including Dev and QA needed access. Stytch also has a performance team that the company was eager to include in Python library development.

    Stytch After DevOps

    Stytch is using DevOps to unite teams and divide the various software monoliths into microservices for continuous integration (CI) and continuous delivery. The company has already fed its core software platform through the new build pipeline a number of times, and continues to create even more automated and efficient deployment processes.

    As it works to create homegrown Python orchestration libraries, Stytch is establishing a stronger grip on deployment. All teams connected to the pipeline including Dev, QA and Performance are actively expanding these libraries.

    Stytch increasingly intermarries its Dev, QA, Ops and Performance teams and enables and rewards collaboration across the stretch of the DevOps pipeline. “Our goal is transparent, scripted change integration, from check-in to testing and deployment. We empower our developers to take responsibility for their work, from check-in to production deployment,” Frost said.

    Stytch uses frequent meetings of a continuous delivery committee with members from all four groups to update the status of the pipeline. The company is swapping out Jenkins for Travis CI for builds and tests, making Dev more hands-on and able to gauge its work from start to finish.

    “Developers are iterating on their pipelines in the same way that they iterate on their projects,” Frost said. Stytch is able to find and fix problematic code earlier in the pipeline while accelerating development cycles. Customers are pleased with the faster rate of innovation on the software platform they use.

    New Infrastructure, Training = More Results

    Stytch has made cross-pollination of expertise common for pipeline teams. “We have an annual training allowance and our engineer-led ‘Ministry of Propaganda” holds regular afternoon sessions to introduce new technology and practices,” Frost said.

    As CI and continuous delivery processes grow, absorb and automate manual steps, related activities across the pipeline become more translucent for everyone. Both the Travis CI platform and the use only of scripted product releases and scripted tests for performance are increasing that transparency.

    The metamorphosis that has spread throughout the teams with direct access to the DevOps pipeline now extends to the rest of the company, as needed, such as with marketing. Frost provided an example:

    “Our marketing team was struggling to make good progress with regular releases of our WordPress-based website. Our engineers were fielding regular requests for basic tasks—starting and stopping servers, updating configuration files and retrieving logs,” she said. “We brought our engineers and marketing together on this. The result was a simple command-line tool that enabled our nontechnical marketing staff to perform complex actions on their own, including promoting their changes to the production environment.”

  • xMatters Sees Value in Continuous Testing

    xMatters Sees Value in Continuous Testing

    xMatters builds an intelligent communications platform that alerts and connects company stakeholders for IT incident response and in manufacturing line workflow support and DevOps systems scenarios. Customers who count on xMatters also count on the effectiveness of the vendor’s continuous testing mechanisms: A failure of continuous testing early on at xMatters could lead to a magnified impact on the customers’ crises response, resulting in costly repercussions.

    XMatters uses continuous testing to promote continuous delivery of its intelligent communications platform, delivering changes to a limited number of features in an accelerated fashion. “This reduces our risks of failures and makes it easier to resolve defects,” said Deepa Guna, QA Architect at xMatters.

    For xMatters, the proper use and placement of continuous testing in the DevOps pipeline is critical to ensure the intelligent communications solution does not introduce errors during customer crises.

    xMatters Software, Languages and Tools

    The xMatters platform consists of a service stack architecture with front-end technologies accessible via the web and back-end technologies that support it. According to Guna, the platform takes advantage of Java, RESTful JSON, Node.js and Postgres, which run on data centers that are geographically dispersed in North America, Europe and the Asia Pacific regions.

    “Our data centers are built on infrastructure as a service, which includes technologies such as OpenStack, Puppet, Fabric and Consul,” Guna said. XMatters also has a mobile application for iOS and Android platforms as part of its communication service.

    xMatters Testing Approaches, Before and After DevOps

    The company’s testing before the advent of the continuous delivery pipeline included manual test execution of regression tests in the midst of a slow process of releasing one large set of features per month and deploying customer-reported defects, Guna said.

    Today, xMatters uses automated and manual regression tests and unit test code coverage monitoring, releasing small sets of features and resolved defects weekly while enabling customers to provide feedback early in the feature development process.

    XMatters uses several tests during the development, test and release phases of the DevOps pipeline and when delivering the platform to the customer. The development phase includes the following testing tools and their associated types of testing:

    • SonarQube analyzes the quality of the code.
    • Jacoco measures the code coverage of the developer unit/integrations tests.
    • Git + Jenkins run the unit and integration tests before xMatters commits the changes.

    The testing phase, Guna said, runs tests as follows:

    • Jenkins deploys artifacts in test environments and runs automated tests.
    • An automation framework that includes Jmeter, Gradle, Java, TestNG and Selenium runs its tests.

    The release phase tests software after this manner:

    • Jenkins deploys artifacts in test environments and runs automated tests.
    • An automation framework that includes Jmeter, Gradle, Java, TestNG and Selenium runs its tests.
    • Splunk and Takipi do exception monitoring.

    When xMatters delivers the platform to its customers, Guna said, it performs the following tests:

    • Runscope monitors the health of the system to ensure critical services are running.
    • Pingdom monitors the status of the customer websites.

    Drilling Down into Release Testing

    “During the release testing phase, we put the application through 1,000 fully automated and manually initiated tests that we execute with every release cycle. We also have tools that are monitoring for exception during the release testing,” Guna said. Passing this phase successfully, the new software platform release is ready to be pushed into the production environment.

    XMatters uses the continuous delivery pipeline to deploy the new platform release first to the customer staging environment and next to its production environment, Guna said. “All production environments have ongoing smoke tests that we execute every five, 10 or 15 minutes, based on customer tier, to monitor the health of the services.”

    Results Metrics

    xMatters’ method of continuous testing seems to be working for the vendor. “We have reduced the number of major incidents in production due to application defects by 40 percent this year compared to last year,” Guna noted.

  • DevOps Q&A: SUMO Heavy’s Brodie on the Hybrid Cloud

    DevOps Q&A: SUMO Heavy’s Brodie on the Hybrid Cloud

    Not every expert says DevOps will enable the hybrid cloud. One says it never will. Find out why in this Q&A with Bob Brodie, partner and CTO of SUMO Heavy, a digital commerce design and development firm.

    David Geer: How will DevOps enable the hybrid cloud?

    Brodie: I do not feel that DevOps enables the hybrid cloud. Hybrid clouds are very complex, and DevOps would need to be supported by a team of network and system engineers. The beauty of being on the cloud is that it’s API-driven. You can provision servers and configure and deploy with tools that are easy to set up.

    Bringing dedicated servers into that environment which do not have tooling to match the cloud counterpart means that it can’t be solely maintained by a DevOps team, but rather needs much more support. We have worked in hybrid cloud environments, and have experienced firsthand the amount of effort and resources required to create and maintain a hybrid cloud environment.

    Geer: What makes hybrid clouds more complex than being either fully in the cloud or fully dedicated?

    Brodie: When mixing dedicated servers with cloud servers on the same network, you will likely need parallel VPNs set up so you are not sending your data unencrypted over the internet and so that if there is a networking issue there is a fallback. Setting up full-duplex, parallel VPNs can require both expensive hardware and specialized resources to configure and maintain that hardware.

    Geer: Why have tools such as some of the ones AWS is building (OpsWorks On-Premises) not hit the inflection point where these are making it easier to enable the hybrid cloud? What other tools are examples of this?

    Brodie: OpsWorks On-Premises is just one tool that AWS has. They also have VPNs to connect your dedicated hardware to your virtual private cloud. These tools aren’t necessarily to make it easier to run a hybrid cloud, but they allow you to utilize existing AWS provisioning and deployment tools to set up and deploy to your existing infrastructure.

    Geer: Why do you think these tools never will make it easier to enable the hybrid cloud?

    Brodie: Hybrid cloud infrastructure is inherently more complex than being fully dedicated or fully cloud-based. They are two different types of ecosystems, and are managed differently. They require different skill sets and levels of experience within those ecosystems.

    Geer: Please expand on your following statement: “The problem is that the methodologies behind the two schools of thought (Separate sysops and dev versus DevOps) are vastly different in thought and practice, and to really bring dedicated hardware into a cloud-based architecture is very cumbersome, especially if the dedicated environment is not utilizing complex virtualization.” What does the difference in schools of thought have to do with it? What does bringing dedicated hardware in have to do with it?

    Brodie: My point here is that “DevOps” is often misused. It isn’t a tool, it is a mindset. It’s a combination of development and sysops. A “hybrid cloud” is a mix of dedicated hardware and cloud infrastructure. My main point here is that DevOps practices do not make managing enterprise developments easier across hybrid cloud environments. Hybrid cloud environments are inherently more complex than the alternatives (dedicated hardware or fully cloud-based).

    The reason for this is that development is becoming more and more intertwined with sysops. For example, five years ago you could download MAMP onto your Mac and be fully set up with a PHP & MySQL development environment. Modern PHP development requires a developer to understand specific versions of PHP and MySQL, how to manage them, how various applications communicate with each other and, often times, many other applications (Redis, MongoDB, Queueing systems, etc.). These can be complex to maintain, and even if they are distributed to developers from the ops team, it will often be in the form of Vagrant boxes or Docker files, which the developer will be required to understand.

    This is the nature of the “DevOps” school of thought, versus the traditional “SysOps separate from development” school of thought. The target environment needs to be considered when developing an application. For example, if the application will reside on AWS then the developer will need to have a solid understanding of the tools available on AWS (RDS, EC2, ElastiCache, etc.). The maintenance of these becomes part of the developer’s job, which was not present in previous times. Adding the extra complexity of a hybrid cloud means that the developer needs to understand the underlying network architecture of that system.

    Geer: In what way is the hybrid cloud a settling point, mostly built on marketing?

    Brodie: Hybrid cloud environments are not ideal. They are costly to maintain and provide more points of failure because of the networking between them. What it can do is provide a halfway point for CIOs that are reluctant to move to the cloud, so they can see some of the benefits on the cloud side and hopefully move their organization to the cloud.

    Geer: What are organizations afraid to let go of?

    Brodie: Many organizations are scared to move to the cloud. They may have a CIO/CTO that is afraid of “losing control.” I have seen many environments such as this.

  • Health Care’s Shift to the Cloud via the Hybrid Cloud, Part 2

    Health Care’s Shift to the Cloud via the Hybrid Cloud, Part 2

    In part one of this series, we looked at how the healthcare industry has changed, requiring integrations and interoperability that were not previously needed.

    Having established that the cloud is the way to go and that the hybrid cloud is the stepping stone hospital CIOs are comfortable with, we must ask how DevOps can help.

    DevOps in Health Care

    The legacy HL7 connection technology healthcare facilities use is a great example of how DevOps brings interoperability to health care and the hybrid cloud. “We have a cloud-hosted HL7 interface that can connect to any number of different hospitals and to our system as well. That’s something that would be very difficult to do without a cloud-based HL7 interface,” said Mayank Thanawala, CTO of HealthLoop, a provider of a cloud-based platform to automate follow-up medical care.

    Another challenge hybrid cloud can address concerns the registries that track patient episodes and the bundled payment initiatives that health insurance companies use to pay the hospital systems, Thanawala noted. “The challenge is that hospitals have to get the patient data into the registries in order to receive payment.”

    A hybrid cloud makes it easier to submit data to the registries because it provides an opportunity to connect to the registries using a system that already connects with the EMR system also and contains other data useful for registries and payments.

    Thanawala illustrated registry solutions in the hybrid cloud. The Comprehensive Care for Joint Replacement Program is a large registry hosted by Medicare CMS. Hospitals need data such as diagnosis histories from medical records and patient-reported survey information to submit to this registry. “So, in an example of a hybrid cloud, HealthLoop is a cloud system that uses a web interface to ask the patients the questions and collect the data. Then we use this hybrid cloud-based HL7 interface to query the on-premises EHR (electronic health records) system for all the rest of the data, fill in the blanks, create this file for this registry, and basically solve the hospital’s problem,” he said.

    How DevOps Speeds the to the Hybrid Cloud

    DevOps and the move to containers have helped accelerate health care’s move to the hybrid cloud. “When you deploy a Ruby on Rails API in, say, a Docker container, because of how the container is configured it is going to take care of logging for all of the applications and transactions and putting those in a store that will keep the transactions for the seven-year period that HIPAA requires,” Thanawala said. This automation saves the hospital CIO the step of having the logging set up.

    In the same way, if encryption of data both in flight and at rest is set up automatically in the container default configurations, that’s another matter the hospital CIO doesn’t have to worry about, he said.

    If applications are set up as microservices in containers so that they already perform all these functions when containers are created or scaled up, the hospital and HIPAA requirements surrounding these activities automatically are met. “This lets startups like HealthLoop focus on building app functionality so we don’t have to worry about these other more involved matters,” Thanawala noted.

    Even with health care’s complex move to the hybrid cloud, the modularity, completeness and singularity of applications in container technology as microservices make it possible to configure the containers, use them as building blocks and to build out efficient, scalable solutions in the cloud, he said. “It allows companies such as Catalyze, a medical systems integration and compliance organization, which we use to offer its products in a PaaS (platform-as-a-service) model using compliant containers so that we can simply deploy the containers and not have to worry about building compliance.”

    Saving Time, Effort, Worry

    HealthLoop’s story is a great example of how the automation and modularity of DevOps components such as containers and microservices can make or break new business opportunities, such as taking health care to the cloud and solving interoperation challenges.

  • Health Care’s Shift to the Cloud via the Hybrid Cloud, Part 1

    Health Care’s Shift to the Cloud via the Hybrid Cloud, Part 1

    The changing healthcare scene is dragging hospital systems into the cloud to enable the interoperability necessary for the industry to send electronic medial records (EMRs) and other data to outside systems such as registries that enable bundled payments.

    Because health insurance companies and Medicare only pay for patient episodes rather than for each subsequent treatment regardless of whether they’re medically necessary, hospitals must follow each treatment and take responsibility for them all. This gives hospitals incentive to get treatments right the first time.

    devopsy.kinsta.cloud paints the landscape behind the healthcare industry’s DevOps-enabled move to the hybrid cloud together with illustrations of how DevOps speeds that transition.

    The Old World Order in Health Care

    The Old World order for health care dictated that each hospital maintain its own isolated, protected, siloed and closed systems and patient health records, which in the not-too-distant past were maintained entirely in paper documents and charts. “That closed system was a file cabinet,” said Mayank Thanawala, CTO of HealthLoop, a provider of a cloud-based platform to automate follow-up medical care.

    Patients likely would have outpatient visits with their doctors at the local clinic adjacent to the hospital. Interoperability between the hospital systems and the clinic systems amounted to hospital staff faxing over the patient record or chart or sending it over via Sneakernet.

    A transition still occurring to move all that into EMRs/EHRs (electronic health records), although in a similar model as paper, with closed systems tied to the hospital and local clinics. “These are mostly not cloud systems but are rather on premises with largely client-server topologies,” Thanawala said.

    Pain Points Driving Health Care’s Shift to the Cloud

    In the old billing model, the hospital or physician billed the patient’s insurance or Medicare for each individual visit and procedure, treating them as separate, disconnected events even when the patient had the same treatment twice for a single need. “Insurance companies and specifically Medicare are now saying, ‘Hospitals, we don’t really care about all these complications, we care about the overall cost. If your patient had to go to the emergency room and had to have you perform the same surgery over again, that’s on you. We’re only paying for that initial surgery,’” Thanawala said.

    The result is that the hospital must know whether the patient had additional care or the same care twice for the same need, where they had the procedure or treatment and what that was. A closed system, electronic or paper cannot address these requirements. “That is driving the need for interoperability between systems across different organizations, which is driving the shift to the cloud to handle that need,” Thanawala noted.

    Delays in the Industry’s Shift to the Cloud

    There is a breadth and complexity to health care that requires a great deal of cloud infrastructure, which is creating delays in the industry’s move to the cloud. Health care and HIPAA regulations demand tight, auditable security, which is easier to achieve with the current on-premises systems. “There’s not a lot of confidence in cloud security. It’s getting better, but hospital CIOs have great responsibility and want to be the kings of their domains where security and control are concerned,” Thanawala said.

    At the same time, and because it can be problematic for hospitals to update workflow technologies, the cloud must support legacy connections such as HL7 that are not going away. “HL7 is designed to work within a network. Once you talk about cloud systems accessing HL7, you are talking about site-to-site VPNs for security and developing to embedded older technologies such as versions 8 or 9 of Internet Explorer,” he said.

    The fact that the data so heavily affects people’s lives, including life-or-death situations, makes the proper scalability, security, stability, uptime and privacy, as well as all the other goals of the cloud and DevOps, doubly important. “It adds another level of scrutiny and pressure to the move to the cloud,” Thanawala said.

    Take, for example, auditing, which touches every piece of healthcare data with such intensity. “From a systems perspective, you have to start thinking about auditing systems and tracking all transactions seven years back, which puts a lot of pressure on logging systems and databases,” he noted.

    As for uptime, if a healthcare system goes down, how does it affect life-or-death situations in that moment?

    How Health Care’s Transition to the Hybrid Cloud is Unique

    “The hybrid cloud gives the hospital CIO the ability to maintain control over what they need to control and still look to the cloud to enable the integration and interoperability with outside systems,” Thanawala said. This is important because of the many factors that make transitioning to the cloud unique in the healthcare industry, including:

    • Users in hospitals blame their systems—systems the CIO is responsible for—for any number of challenges and errors.
    • Healthcare facilities don’t really have to scale in the cloud quickly, because in health care there already is a defined patient population and facilities won’t have a large number of new users signing up that it can’t handle.
    • Service providers must convince hospital CIOs that collaborating with cloud technologies, including opening up VPNs and ports where necessary, is worth the risk.

    Digging into DevOps

    In part two of this series, we’ll explore how DevOps helps to enable the hybrid cloud in health care.

  • Case Study: Laserfiche Builds Hybrid Cloud for ECM

    Case Study: Laserfiche Builds Hybrid Cloud for ECM

    Laserfiche is a privately owned software development company that creates enterprise content management (ECM), business process automation, workflow, records management, document imaging and webform software. The company provides secure document management, image capture, electronic forms, workflow automation, digital signatures and DoD 5015.2-certified records management, “Automating and streamlining business processes with prepackaged solutions for contract management, HR onboarding and new account openings,” says Thomas Phelps, vice president of Corporate Strategy & CIO.

    Laserfiche also maintains a business process library including templates that enable its customers to hit the ground running on new ECM deployments.

    The business counts Fortune 500 companies among its customers, serving industries including government, education, health care and finance. “We’re an enterprise content management software leader in state and local government, with a majority of the Center for Digital Government’s top digital cities and counties as our customers,” Phelps says.

    This is the story of the company’s addition of a cloud infrastructure and its plans for a hybrid cloud.

    Embracing DevOps and its Culture in Preparing for the Cloud

    Laserfiche embraced DevOps and its methodologies and tools simultaneously as it began development on its cloud infrastructure, the Laserfiche Cloud, which the company announced in January 2014 at the company’s worldwide Empower user conference.

    With that, Laserfiche took a sharp detour from its usual scrum development practices, adopting DevOps for its ability to encompass the development needs and demands of a cloud project.

    To use DevOps to move up the timetable on bringing the Laserfiche Cloud to life, the company had to swallow the DevOps culture, tools and processes whole. “We created a separate, dedicated cloud infrastructure team responsible for rapid deployments of test and production environments. We assigned a member of each software product team as a designated Laserfiche Cloud liaison to represent each product team including Laserfiche Forms, Laserfiche Mobile and Laserfiche Connector teams with respect to any feature updates, testing or releases for Laserfiche Cloud,” Phelps says.

    Though the Laserfiche Cloud infrastructure development team works in unison, each team member has the opportunity and responsibility to isolate, program and interweave software and services. Proper governance and controls ensure this means progress rather than setbacks for the ongoing cloud project.

    Development adopted PowerShell scripts, open-source and homegrown tools and Chef to promote advancement of a DevOps pipeline for its cloud infrastructure.

    Software Development Before DevOps and the Cloud

    Before moving to DevOps, the company developers relied on scrum-based agile development methods. The process followed from planning two-week sprints, through developer programming on desktop machines, unit testing, committing daily code changes, testing using daily automated builds and ultimately ending with retrospectives. Software products were tested individually by their own teams on separate VM clusters assigned to each team.

    That process allowed for one big release per year with smaller releases before and after. “Before DevOps, we manually uploaded release packages to our support site for customers and resellers to download, install and configure the Laserfiche ECM,” Phelps notes.

    After DevOps, with the Cloud

    The Laserfiche Cloud, a software-as-a-service (SaaS)-based ECM subscription service, offers mobile synchronization, e-forms and workflow automation in addition to its previous services, once offered only on premises. “The Laserfiche Cloud provides robust granular system security, encryption and audit reporting to bring centralized control and transparency to business operations,” Phelps says.

    With the cloud infrastructure in place, the company now is using the same DevOps culture, structure, tools and people to develop a hybrid cloud version. “We’re developing a hybrid solution, using Laserfiche Web Access, to provide access to hosted content and enable collaboration with third parties, business partners and customers,” Phelps notes.

    Results, Benefits

    “Instead of taking days or weeks for information technology or our software testers to deploy new VMs and infrastructure, the cloud infrastructure team can spin up new test environments within hours,” Phelps says.

    The company progressed from one major product release a year to updates every one to three months by automating the process of configuring new environments and deploying software. “Many software engineering teams have begun adopting a cloud-first development system where they develop and test binary packages and use internal tools to deploy the packages for testing,” he notes.

    Laserfiche can now satisfy customers with larger, swifter feature enhancements to its Laserfiche Cloud. “The cloud infrastructure team works closely with our software engineering teams, using accelerated release management processes, to perform upgrades in hours within an approved maintenance window,” Phelps says.