Tag: QA

  • Enterprise Cassandra Deployments

    Enterprise Cassandra Deployments

    Successful Cassandra deployment in the cloud takes a certain amount of skill and knowledge. In this blog I will share best practices for a successful Cassandra deployment in the cloud such as Amazon Web Services (AWS) and Google Cloud Platform (GCP). I will also discuss how you can protect and recover the data in your Cassandra cluster when deployed in the public cloud.

    Cassandra Deployments in the Public Cloud

    There are two main ways to have successful enterprise Cassandra deployments in AWS:

    • Use a ‘managed’ Cassandra deployment. This is a reasonable approach; the only downside is the inability to customize data placement beyond what is typically provided by the vendor.
    • Manage the cluster yourself, if you want to have full control over your Cassandra deployment and customize it to scale and perform according your business needs.

    Please refer documentation from Datastax or Apache Cassandra for detailed best practices for Cassandra deployments. The following best practices are reiterated here to simplify deployment of a scalable Cassandra cluster in AWS.

    Storage

    Both AWS and GCP offer two types of local storage options for their compute instances. For AWS, the options are Elastic Block Storage (EBS) and Ephemeral Storage. In GCP, the options are Persistent Disks and Local SSD. Both options have their own trade-offs. Pick the one that best suits your deployment.

    Persistent Storage

    EBS volumes have performance characteristics similar to network storage such as a NAS or SAN. Because storage from EBS volumes is multitenant in nature, it is not suitable for distributed, log structured databases such as Cassandra. If you want to ensure reliable performance through EBS volumes, use “EBS optimized instances” with “provisioned IOPS.” Since the IO traffic to EBS volumes goes over the instance network, EBS optimized will provide reliable performance, but does not guarantee any increased performance. To get the most performance out of EBS volumes, use instances with multiple disks and separate the volumes used for “Data” and “Commitlog.”

    If you have to use EBS Volumes, go with “EBS Optimized Instances,” with “Provisioned IOPS” and split data and Commit Log on different volumes.

    Ephemeral Storage

    Ephemeral storage is a great option to use for downstream Cassandra environments such as test/dev and staging. Since ephemeral volumes are typically SSDs, there is no need to separate the data and commitlog into separate drives. To optimize performance, choose EC2 instances with multiple ephemeral storage drives and stripe them together into one logical volume. Send data and commitlog to the logical volume. Remember that when you stop the EC2 instance, the data in Ephemeral storage will be deleted.

    Stripe all Ephemeral drives into a single volume to get maximum performance.

    Data Distribution

    Cassandra uses Snitch to determine the topology of the cluster. The Snitch is also used to determine the data center and rack for each node in the cluster. Replication strategy is used to determine the placement of data across the different nodes in the cluster.

    AWS

    1. EC2Snitch works well for single region deployments.
    1. Ec2MultiRegionSnitch works well for cluster distributed across multiple AWS regions. Please note, Multi-DC, Active-Active configuration makes your cluster resilient to region failures and for multi-dc cluster, use “local quorum” to improve performance.

    Google Cloud Platform (GCP)

    1. Use GoogleCloudSnitch for Cassandra deployments in GCP.
    1. GoogleCloudSnitch treats regions as Cassandra data centers and availability zones as racks.

    Other Features

    Both AWS and GCP offer a series of operational capabilities that seem to make it easier to maintain high uptime on Apache Cassandra cluster. These capabilities have to be chosen carefully depending your deployment needs.

    1. Load Balancing: Network-level load balancing service is offered by both AWS (Elastic Load Balancer) and GCP. It is not recommended to put a load balancer in front of Cassandra. It is better to use a client connector that can handle this, consider open source utilities such Presto and Astyanax
    1. Autoscaling: Both AWS and GCP offer Autoscaling facility for compute instances. However, although this functionality looks very enticing, putting all C* nodes in an autoscaling group will not produce the desired result. Instead, create a separate autoscaling group of ‘1 node’ for every node in your cluster. This will ensure that anytime a node goes down, it is automatically brought back up. This will ensure the cluster data placement is not disturbed when nodes go down and nodes coming up do not cause excessive data movement.

    Scalable Backups and Reliable Point-in-Time Recovery

    Backing up your data is a must-have for enterprise organizations. The impact of data loss is so high that no enterprise wants to risk relying alone on replication. Protecting data in a public cloud environment may seem daunting, but there are several options available that can make this easier compared to an on-premise deployment. A simple option may look like one of the native snapshots:

    Storage Snapshots

    I have seen few customers use snapshots of EBS volumes as a way to protect the data in their cluster against logical and/or operational errors. This solution will work but with some severe limitations. On top of the overhead of maintaining the scripts to take snapshots, there is a large overhead in storage costs. Taking a snapshot of volumes does not allow for incremental backups of your data. This causes an exponential bloat in the amount of data stored in snapshots, leading to high cost of the backup solution.

    Amazon S3 or Google Cloud Storage

    Using an S3 bucket as the secondary storage enables enterprises to use low-cost, highthroughput storage, while reducing overall operating costs for storing backup data. Here are a few recommendations when using S3:

    1. S3 Region: Amazon S3 buckets are region-specific. A bucket created in one region will provide better performance characteristics to other infrastructure in the same region. Subsequently, it is important to ensure the S3 bucket used as the secondary storage is in the same region as the production database being protected and the Datos IO cluster.
    1. Storage Class and Lifecycle Policy: Amazon S3 allows users to create lifecycle policies on buckets. These policies dictate the performance and response time characteristics of objects in the bucket. STANDARD_1A is more suitable for longer lived and less frequently access data, and is not suitable as secondary storage. Additionally, do not turn on any lifecycle policies on the buckets used for the secondary storage. All buckets should be in the standard storage class.

    Conclusion

    As enterprises leverage the agility provided by public cloud environments, it’s important to understand the various facets of Cassandra deployments to ensure the efficient use of your money and time.

  • Data Protection for Operational Reporting Platforms

    Data Protection for Operational Reporting Platforms

    Making real-time business decisions based on data from social, mobile and cloud platforms is a requirement in today’s world. That’s why an ecosystem of next-generation operational reporting platforms such as Spark, Apache Cassandra, Kafka, Docker, Mesos, Marathon and more have come to life. Enterprises are using these technologies to re-architect their data storage and processing frameworks to achieve lower costs, scalability and faster response times. However, as enterprises onboard business-critical applications on this next-generation technology stack, they need to prepare for failures or errors that may disrupt customer-facing applications.

    One of the key benefits that operational reporting platforms provide is real-time visibility, which allows businesses to make strategic decisions based on actionable insights. However, the new applications generate a large amount of data and archaic operational reporting platforms (those based on traditional data warehouses, data marts and ETL logic) are unable to catch up and, as a result:

    • Business decisions are based on stale data
    • Fragmented data increases the risk of errors
    • Multiple copies of data increase storage costs

    These issues have driven many progressive, data-centric enterprises to re-architect their operational reporting platforms. In this new architecture, Kafka serves as a persistent message bus that ingests data from relational stores and provides failure resiliency and message buffering. Spark or Spark Streaming may be used for transformation, aggregation and other lightweight stream processing. These processing nodes may be deployed in Docker containers. Finally, Apache Cassandra serves as a distributed data storage layer that provides linear scalability and high-availability. Kafka also enables the ability to add multiple data consumers such as Hadoop/HBase through Flafka.

    Datos IO - operational reporting platforms

    However, as enterprises adopt operational reporting platforms, they want to ensure that bad data feeds and human error do not result in data loss. Apache Cassandra natively replicates data but that does not provide the capability to go back in time to recover data efficiently and at scale.

    As enterprises onboard their critical applications to achieve development agility, scalability and lower operating costs, they must also explore data protection solutions that are specifically dedicated for scale-out databases.

  • Q&A: Sematext’s Thies on Docker Monitoring, Part 2

    Q&A: Sematext’s Thies on Docker Monitoring, Part 2

    In part one of my conversation with Stefan Thies, Sematext DevOps expert. we spoke about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Let’s continue the conversation.

    David Geer: What makes Logsene centralized? Where is it centralized a typical deployment? What are the advantages of it being centralized?

    Thies: Traditionally, systems store logs from multiple applications on every server. This makes it difficult to search logs for troubleshooting or auditing purposes. To find a relevant log entry, operators must log in to each server, find the right file on this server and search in this file for a relevant message.

    This can easily take several minutes. With systems storing logs on multiple servers, it is difficult to correlate data, so minutes can turn into frustrating multi-hour root cause-hunting sessions, often involving multiple team members. Logsene provides a central full text index for all log data coming from multiple servers and applications. There is no need to log in to multiple servers and search/grep multiple log files. All team members see all data at the same time and can collaborate much more efficiently.

    Having a central place for search makes it much easier to find logs quickly, run analytic queries or centralize alert rules in a single place.

    Geer: What is it about what Logsene does for Docker logging?

    Thies: Sematext Agent for Docker interacts with the Docker API to automate the collection, parsing and processing of logs, metrics and events. It detects containers automatically (no setup or configuration for a new container!) and supports many log formats for applications on Docker Hub such as Nginx, Apache, Elasticsearch, Solr, MongoDB, MySQL and many more.

    Geer: What does your statement mean, “Some DevOps engineers even think of Logsene as ‘ELK Stack on steroids’”?

    Thies: The ELK Stack is a combination of open source tools: Elasticsearch, Logstash and Kibana. These tools are very popular for log shipping (Logstash), indexing and search (Elasticsearch) and visualization (Kibana dashboards). However, these tools are missing many enterprise features like multiuser user interface, role-based access controls, secure transport of logs or out-of-the-box configurations and an easy query language to search logs. This is where Sematext Logsene adds the relevant enterprise features to these open source tools to make them easy to use and securely integrated into the enterprise information technology infrastructure.

    Maintaining Elasticsearch clusters is another problem for many people. It requires a lot of expertise. Using Logsene does not require this sort of expertise. The Logsene SaaS provides a managed solution where users don’t need to know anything about maintaining Elasticsearch clusters or the required infrastructure. Sematext takes care of that for them. Sematext customers can benefit from these managed services and pay a fraction of the cost compared to self-hosted ELK stacks, where storage, computing and networking infrastructure and manpower for highly skilled workers counts.

    Geer: What is Tutum? What is so important about ease of launching SPM and Logsene Docker on Tutum?

    Thies: Docker acquired Tutum in 2015. Tutum provides management of Docker containers as a cloud service. Tutum is a platform made to operate “Dockerized” application stacks. In the case of Sematext Agent for Docker, the Tutum integration deploys Sematext Docker Agent automatically to all cluster nodes in a few seconds. These nodes could run on different cloud platforms such as Amazon, Digitalocean, Rackspace or on premises. Again, a big time-saver for people running larger infrastructures or hybrid clouds. Supporting Docker products and having excellent integration with Docker products is important for Sematext.

    The Sematext Docker Agent runs on all solutions offered by Docker e.g., Docker Swarm and others such as Google Kubernetes, Google Container Engine, RancherOS, CoreOS, Amazon ECS or Hashicorp Nomad.

    Geer: Please discuss the Sematext Docker and Docker image?

    Thies: Sematext Docker image is available on Docker Hub, and we shared the Tutum Stackfile for Sematext Docker on Stackfiles.io. The easiest way to get it is to use Sematext, which generates the stackfiles for you, including tokens.

    Docker Hub is the central registry for public application images including related installation instructions. Docker users can find out quickly how to run a “Dockerized” application with the help of the instructions in Docker Hub. Typically, this requires a little configuration work before a container can be started with the right settings.

    Stackfile.io is the place where people share preconfigured application stacks (e.g., a web service together with the required database). Sematext provides a preconfigured setup for Sematext Docker Agent to monitor all containers in Tutum Cloud. This means one can launch Sematext Docker Agent and monitor all containers running in Tutum Cloud with just a single click for the chosen stack file and by entering a unique identifier (Token) for the application. That is really all it takes.

    When you log in to Sematext SPM or Logsene the general installation instructions have all required configurations displayed. Sematext users can simply copy/paste the relevant commands including unique identifiers (tokens) for their application to get the Sematext Docker Agent deployed. This, too, is a big time-saver for our users and it gets them up and running in just a few minutes.

  • Trust, You Must: 8 Tips for Adopting CI/CD

    Trust, You Must: 8 Tips for Adopting CI/CD

    We have done continuous deployment (CD) for four years, and enable many of our customers to do continuous deployment with quality assurance (QA). We’ve helped tons of companies adopt and optimize their CD setups. We’ve found that continuous deployment is all about trust — augmented with processes and infrastructure.

    To that point, here are some common tips for adopting continuous integration (CI)/CD that I find myself sharing frequently:

    Use a CI Tool

    This sounds obvious, but you’d be surprised at how many companies try and roll their own. Don’t. There’s no upside for your company—and infinite downside. We love Circle, but there are plenty of other great options. Most are at least somewhat prescriptive when it comes to workflow, which is important when you’re getting started.

    Do Unit Testing

    It’s always worth reminding everyone that continuous deployment is simply continuous integration applied to deployments. To do continuous integration, you need good unit test coverage. If you haven’t got a solid base of unit testing and continuous integration already, you’re not ready for continuous deployment.

    Have Solid Production Monitoring

    Bugs and regressions will get shipped; it’s inevitable. Put a system in place, by looking at the data in production, to tell you when a regression or bug was shipped. Tie this into automated rollbacks, so that if a significant performance or key performance indicator (KPI) regression (i.e., signups per minute) appears, your CD system rolls back to the most recent stable version while your team investigates.

    Trust Your Team

    We talked about the infrastructure requirements, but if you don’t have trust, then continuous deployment isn’t going to work for you. Trusting your team is about giving your developers the freedom to make mistakes, deploy as they see fit and check each others’ code. It is possible to combine continuous deployment with a culture of hierarchical permissions, but we have yet to see that truly work.

    Remove Red Tape from the Code Review Process

    Your team should review code changes, typically through a pull request, which goes hand in hand with trusting your team. Developers should be trusted to choose the most qualified and insightful person to review their code. One of the most common mistakes among teams trying to adopt continuous deployment is that they only let one or two “senior” developers review the rest of the team’s code, “bottlenecking” all releases. Don’t make this mistake.

    Involve Developers Closely in Production Operations

    The most common mistake we notice among companies that don’t make a successful transition to continuous deployment is that their development organizations are siloed. Dev and Ops should be joined at the hip. Have your developers deeply involved in the setup and definition of your CD infrastructure, and half the battle is already won.

    Shift Left: Test Early and Often

    Your team needs constant feedback. Think of your QA objective as getting the right feedback in the right hands at the right time. When it comes to deployments, this means knowing what’s working and what’s broken as the breaking change is made. The sooner you uncover the bug after it’s created, the easier it is to fix. The best companies doing continuous deployment have comprehensive unit test and integration test coverage that runs before any code hits the staging server.

    Treat Staging as Your Holy Place

    The final pre-production environment is usually known as ‘staging.’ A common mistake is to have developers ship new code direct to staging, while figuring out if the new code is ready to go live. This is a terrible idea. As soon as more than one developer is trying to ship their changes, your Slack channel will blow up with cussing and recriminations. Isolate the pre-production environments to each project, and enforce the “staging as ready to go live” philosophy. This way, integration testing can happen at a self-defined pace, without anyone else’s release being blocked.

    About the Author / Fred Stevens-Smith

    fredstevenssmithFred Stevens-Smith is Co-founder and CEO of Rainforest QA, a quality assurance-as-a-service startup backed by Storm Ventures, Rincon, Andreessen Horowitz, Morado and Crosslink.

  • DevOps by Osmosis — Is That Even Possible?

    DevOps by Osmosis — Is That Even Possible?

    “We don’t do DevOps. We can’t, as our development plans are agreed so far in advance and our priority from a production point of view is to keep our systems available 24/7.”

    I heard this comment from a pal of mine as we had a quiet beer over the weekend, but I’m sure this comment, or a variation on it, has been uttered many times in many places. Here’s the irony: I had met my pal, who works as a developer for a major financial services company, after he had had the week from hell. His group tried—and succeeded—to fix a major system issue that had come out of the blue. He had certainly earned the beer that week.

    So, I asked him: “How did you get the problem sorted in the end?”

    “Well, we had to form a Tiger Team,” he replied. “Pulled together some of the DBA team, the key developers and the Ops guys, worked throughout the night and managed to keep the systems up and running the next day, while we used that time to address the root cause.”

    “How did you get around the Governance guys?” I asked.

    “Well, we brought one of them into the team for a few hours, and she was able to get us past some of the key bottlenecks,” he said.

    “And how about QA?” I asked.

    “Oh, they knew this was a major issue and so we cut out a lot of the QA and just went ahead,” he replied. “It was a bit risky, but we had all the right guys in the room to fix anything that cropped up.”

    “So, you manged to get a major change through in a couple days, kept your systems up and running, kept your customers happy, and earned that beer,” I said.

    “Yep, that’s about it, alright,” he said.

    “So, that’s pretty much what DevOps is all about—minus the emergency element,” I pointed out. “Seems to me like you guys are already on the DevOps road.”

    It was about then that the penny dropped with my pal. He suddenly realized that yes, they could do changes quickly, yes, they could do them in an orderly manner and yes, they could keep their systems up and running while all that happened. But, as quickly as the penny dropped and the realization dawned on him, the light went out again.

    “There’s still no way we can do DevOps in our place,” he said. “There’s too many hurdles to overcome, too many guys used to doing things the existing way, which to be fair, has worked well for us. To be honest, there’s too many guys who would feel under threat if we suddenly changed how we do things.”

    And that, in a nutshell, sums up the major challenge facing people promoting a DevOps approach. I have spent many years in the IT industry; I go so far back that I wrote my first code using punch cards (look them up) so I’ve seen change come and go, and change has been the one constant. I think many people in the IT industry have become too focused, whether it’s on the latest tools in their area of expertise, or on the newest piece of kit that has been bought, or indeed on the latest problem the business has put in front of them. The industry has become mature—if we compare the IT world to a field such as medicine, you pay passing attention—if you even notice at all—to things outside your specialty field.

    Now that’s right and proper. I don’t want an orthopedic surgeon trying to fix the problem with my eye, and I don’t want a DBA to write me some code. But when something comes along that affects the whole industry, I want them to pay attention and not ignore it.

    Daysha ran a lunchtime seminar last week, and we titled it “BizDevOps – DevOps isn’t compulsory but neither is survival.” We wanted to use this seminar to engage business people who, let’s face it, are the people we support and who need to understand how IT can make things better. Somewhat disappointingly, while we attracted a great crowd, they were all IT people. No one from the business attended. Now maybe we advertised this event in the wrong places, or maybe we held it in the wrong place, but here’s the thing: We attracted IT people from major companies, and none of those IT guys said to their business people, “You need to hear about this, come along for lunch.” I hope that after our seminar they went back to their business people and said just that, and maybe the next lunch will have the business people in the majority.

    When PRINCE came out into the project management world, it was not bought into by a majority of project managers. The same happened with ITIL. And the same will happen with DevOps. It is fair to say that PRINCE and ITIL are pretty much standard on a worldwide basis. I am old enough to remember the time I first heard of them, and the time I did my first course and passed my first tests, but nowadays, they’re part of the furniture—taken for granted, and rightly so.

    I think people are a little bit afraid of DevOps, and I can see why this is the case. Think about it for a second: Someone is now saying that the way we have done things for years has to change. That we can do things better, faster, with higher quality, but we just can’t do the massive 12-month long project in the same way. We might need some new tools, primarily on the Ops side. We might need some new skills—again, primarily on the Ops side. But, by the way, we’re not going to do away with the Ops guys (so don’t expect to make major budget savings there). But here’s the thing: Our end customers will notice the difference, and so will not be tempted by the new kid on the block—who, as it happens, is probably using a DevOps approach to develop his application, which is threatening our business.

    Change is scary, it’s not easy, and making a major change while you’re still having to churn out the new business requirements, and keep the current systems running … now that’s really scary. But, like my pal whom I shared the beer with last weekend, maybe you’re not as far away from that change as you think you are, maybe it’s not as scary as you think it is, maybe it will affect people positively, maybe you just have to do it.

    About the Author/John Brophy

    eJohn Brophy is currently Head of Professional Services at Daysha Consulting in Ireland, where he leads a team of 25 project managers and business analysts working on projects in blue-chip clients. He is a 35-year veteran of the IT industry, starting out as a database systems programmer, moving onto leading a team of systems programmers looking after what were then called midrange systems. He then became a senior project manager with a global IT company and followed that up by running his own IT infrastructure support company for 6 years. His background is Ops – and he is proud of that!

  • Q&A: Sematext’s Thies on Docker Monitoring, Part 1

    Q&A: Sematext’s Thies on Docker Monitoring, Part 1

    Stefan Thies is DevOps evangelist Sematext Group. I recently spoke with him about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Read on for part one of our conversation.

    David Geer: Technically speaking, for the purposes of this article, what is logging?

    Thies: Logging means to store the output of programs (applications, servers) running on a computer. Typically, log messages that software applications produce are stored in log files. However, software running in Docker containers writes its log messages to a virtual terminal (console). The Docker Daemon (the server process managing the containers) has several facilities to forward log messages via log drivers or the Docker API to external applications, which are responsible for storing the log messages. This is not the traditional way of working with log data. This changes the log collection and storage mechanisms and forces many organizations to change their workflows for log management by introducing new tools.

    Typically, you would use a service called syslog to receive and store log messages on servers, but this requires special configuration for all Docker containers. While Docker made it easier to deploy applications, the new technology made logging and monitoring much harder as you could not use the traditional tools apart from integration with Docker.

    This is where companies like Sematext provide added value, by enabling:

    • Integration into the Docker Ecosystem
    • Cloud storage for log files
    • Indexing for full text search, analytics, visualization and alerting on log data

    Geer: What has Sematext Logsene contributed to logging, technically?

    Thies: Logsene Log Management includes all required workflows and security procedures to handle log data. Logsene provides role-based access controls for access to log data. Logs are collected and shipped to Logsene servers (running in the cloud or on premises) where the data gets stored and analyzed.

    Other important features are:

    • Full text search
    • Analytics
    • Alerts
    • Anomaly detection

    Geer: In what way is Logsene/Sematext available to users and organizations that seek solutions to capture logging for monitoring their Dockerized distributed applications?

    Thies: Logsene is available on premises and as SaaS in the cloud. It fits in hybrid environments as well such as running the Dockerized application in-house on bare metal while shipping logs to Logsene SaaS. In this case, only Sematext Agent for Docker runs on premises.

    Sematext Docker Agent automatically detects running Docker containers and automatically collects logs from the containers. The Sematext Docker Agent can transform unstructured log messages into structured data for analytics. We do this with integrated parsers and a pattern library, which detects log formats to structure logs. The final task of Sematext Docker Agent is to ship the structured logs from all containers securely to Logsene, where indexing for full text search and analytics queries takes place. The Logsene user interface provides all tools for search, visualization, anomaly detection and alerts on log data.

    Geer: What is SPM for Docker? What role does it play?

    Thies: Both Logsene and SPM collect data via the Sematext Docker Agent. SPM is a performance monitoring, anomaly detection and alerting solution for many server applications like Web servers (e.g., Apache, Nginx), databases (e.g., Cassandra, MongoDB, HBase, MySQL, etc.), big data engines (e.g., Spark, Storm, Hadoop, etc.) and full text search engines (e.g., Elasticsearch, Solr). Sematext Agent for Docker collects not only container logs but also performance metrics of all containers at the same time, plus Docker events.

    SPM for Docker provides operational insights into application performance on the Docker platform. It is not limited to performance monitoring—it integrates with Logsene for log management in a single user interface. DevOps teams can find log messages in correlation with potential performance bottlenecks. Most organizations use different tools for performance monitoring and logging. Sematext has recognized that metrics and logs must be available in a single platform to provide detailed operational insights without having to switch between different tools and without overhead for things like integration of multiple tools, multiple user interfaces, logins managing access permissions for multiple tools, and billing or licensing from multiple vendors such as NewRelic or Splunk.

    Watch this space for part two of my conversation with Stefan Thies of Sematext.

  • Continuous Testing Live Network DevOps

    Continuous Testing Live Network DevOps

    My earlier blog, “From Laboratories to DevOps: Factories for Continuous Testing,” discussed the importance of and suggested practices for a fully automated, orchestrated software DevOps laboratory infrastructure, akin to a modern factory staffed by robots churning out products at a dizzy pace. The blog applies to development of software for networks, and the practices increasingly are important as the network becomes more software-based thanks to software-defined networking (SDN),  network functions virtualization infrastructure (NFVI) and cloud-based networking. The edges of mobile networks, even the radio access networks, are becoming more software-dependent due to the need to position computing resources closer to mobile customers to resolve latency, bandwidth and spectrum concerns.

    However, the need for highly automated software processes does not stop within the development lab.

    At a recent DevOps Networking event in Santa Clara, California, Google Network Operations Manager Anees Shaikh indicated that Google’s live network has many thousands of nodes and management elements, supported by millions of lines of software-based configuration files that are updated tens of thousands of times per month. The sheer volume and speed of software updates for large networks is not feasible using manual methods. Automation tools for live network software updates are vital.

    Typical live network DevOps environments use DevOps continuous tool chains such as Git for software version management, YAML for defining abstract configuration definitions, Jinja2 to translate the configurations into concrete templates that can be consumed by real devices and Ansible for automating the distribution and activation of config files to the network nodes. But what about testing? How can a network operator be confident that the many updates will perform as expected once deployed across the live network? This requires much more than simple syntax and semantics checking of the config file entries.

    Recommended Practices for Continuous Testing of Live Network DevOps Environments

    For continuous testing of live network DevOps environments to work, here are some best practices to consider:

    1. Lab-to-live transition: Prior to deployment of any updates to the live network software updates should be tested in a lab environment using production-equivalent equipment and topologies.
    2. What to test? The actual tests to be executed depend on the risk of changes that are being deployed. If a small, low-risk configuration change is all that is being deployed, then a simple configuration check may be sufficient. On the other hand, if a change has the potential to disrupt the network performance significantly, then a much more comprehensive performance test is needed.
    3. Test orchestration tools: The test environment prior to deployment will vary depending on the tests that are selected. Test environment orchestration tools are needed to set up network test topologies that match the tests rapidly—and release them just as rapidly so the test resources can be used for the next tests.
    4. Elastic on-demand topologies: The ability to scale test resources horizontally and vertically, on-demand, is essential to be able to match the speed of testing required for short duration deployment cycles.
    5. Automate live network tests during deployment: To verify software updates do not interrupt or degrade network performance, functional and performance tests between pairs of nodes across the network mesh are required as the updates are deployed in stages across the live network nodes, clusters, centers and regions.
    6. Automate live network testing after deployment: After verifying that the deployments are successful, network tests need to be scheduled periodically to ensure latent problems with the new software do not impact the network.

    This is just a partial list of suggestions for continuous testing essentials for live network DevOps.

  • Are You a Transformational CIO?

    Are You a Transformational CIO?

    George V. Hulme discusses the state of CIO Relevancy and Transformational IT with Tim Crawford

    Facing increased consumerization of technology, more autonomy by business units in choosing and deploying IT, citizen developers and increasingly affordable cloud software services, IT departments and the office of the CIO are under more pressure than ever before. To get some answers about how smart IT leaders are ensuring their relevancy, we turned to Tim Crawford, an internationally recognized expert on enterprise IT transformation, cloud computing, IT relevancy and innovation.

    Crawford has held CIO and other senior IT roles with global organizations including Konica Minolta/All Covered, Stanford University, Knight-Ridder, Philips Electronics and National Semiconductor. He regularly speaks at industry conferences on the topics of strategy, applications, services, infrastructure and operations. Crawford also serves as an adviser to global enterprises, government agencies, venture capital firms and startups across numerous industries.

    Hulme: There is a lot of conversation around the relevancy of IT, and it escalated as rogue apps, IT and even citizen developers have taken root in many enterprises. And many are questioning the relevance of the IT organization in the years ahead. What is your take on the current state of IT relevancy?

    Crawford: I think we have to separate the types of IT organizations. There are those that are very technology-focused. They are focused on the latest, greatest whiz-bang, bright, shiny object. These are the folks that look at IT for IT’s sake. They look at IT as the linchpin in the organization and work under a command-and-control management style. Those organizations are really going to struggle. We’re already seeing this happen. We’re already seeing CIOs of that caliber getting removed from their posts. We’re seeing that type of IT organization get marginalized significantly in terms of its strategic value to the organization.

    A good way to test this is to ask someone from elsewhere in that business, “What role does IT play? Are they the ones that you call when the Internet is broken or you need to get a projector fixed or an application doesn’t work? Or, are they the folks that you call when you’re trying to figure out a new strategic initiative for the company as a whole and you want IT at the table? Or, is IT coming to you with different business objectives that actually are substantial and are not about technology?”

    I would differentiate between that—what I call the traditional—IT organization and the transformational.

    The transformational IT organization, on the other hand, is very much in demand and there are very few IT leaders that understand it. There also are very few business folks, so folks outside of IT, that have that level of expectation of IT; that transformational expectation, that business-centric expectation. And the best way to think about a transformational IT org is they are a business organization first that happens to have responsibility for IT. So, a CIO that is very business-oriented as opposed to tech-oriented, that also happens to have responsibility for IT.

    Back to your fundamental question, I would argue that the traditional, not the transformational, but the traditional CIO and the traditional IT organization is very much in decline. The transformational IT organization and the transformational CIO is very much in demand and that trend is something that we are just starting to scratch the surface on.

    Hulme: With that in mind, how do you move your organization to a successful transformational IT organization?

    Crawford: That is the $64,000 question and the No. 1 issue is in understanding culture, understanding people, understanding organization. And notice, technology was not one of those key attributes.

    The problem is not a tech problem; the problem to transform is not a tech problem. Quite often you have leaders that are somewhat sedentary. That’s one category of folks. Another category of folks is they just don’t know; they don’t know how to get from here to there and they need some help. But then you also have those folks that are very much of the mindset this has worked well for us for the past 10, 20 years and that’s what we’ll continue to do.

    But, for those folks that truly want to transform, there are a couple of challenges that they need to be able to get around. One is inertia. There’s a phenomenal amount of inertia within an IT organization that heads you down the status quo path; the traditional status quo path. On the other hand, if you understand the organizational shift and you understand that culture needs to change, then you will take steps in order to do that. And some of that may be the language that gets used, it may be how the IT organization is organized; just the fundamentals of organization. We no longer have an apps org and an ops org or an infrastructure org. They’re all one organization and they’re centered around other functions. And that gets to be a little more specific to the nature of the business and what works for that team.

    There’s a lot of personalization that comes into play. For example, pick you and I. We may have very similar perspectives on many different things, but we’re also two very different individuals. And so the way that you would handle George vs. Tim, and the way you would engage them might be very different. And so there’s a lot of personal work that needs to be done. But it starts with a vision, it starts with a direction and it starts with a game plan or road map to be able to achieve that.

    Hulme: It would strike me that enterprises outside of the IT organization—enterprises that have transformational IT—are going to be a lot better off in their ability to compete than those that don’t have good IT leadership because they may end up floundering.

    Crawford: Absolutely. Without a doubt, that is absolutely true. In order to compete moving forward, you absolutely need a transformational CIO and an IT organization that is of similar mindset.

    Now, to get from here to there, because, remember most enterprises said, “Hey we’re not starting fresh. We’re not an Uber, we’re not an Airbnb where we’re just creating this brand new IT organization where we can pick and choose the people. We have legacy culture, we have legacy systems, we have legacy thinking. We’re not starting out fresh; we’re not starting out new.” Enterprises don’t have that luxury.

    And, this is where we often get into this conversation around bimodal IT. Well, to do that, do you create a startup within the IT organization that focuses more on the transformational aspect and then have a different group that handles the legacy aspect? From experience from the organizations that I’ve seen that have made an attempt, a solid attempt, not just a half-assed attempt but a solid attempt at bimodal IT, they have struggled or failed miserably.

    Hulme: At doing bimodal? Why do you think that is?

    Crawford: At doing bimodal. The reason for that is because a lot of that has to do with the fact that IT is an incredibly complex and integrated system; you know if I can use system in a more generic term as opposed to technology. And so, think about it. You’ve got your systems of record that might be in the legacy camp, but you’ve also got systems of engagement that might be in the startup camp. The systems of engagement still have to connect to the systems of record.

    So how do you start to make those connections and forget about technology for a minute. Now you’ve got legacy folks that are looking at the startup folks going, “How come they get to do all the cool stuff and we get stuff with this legacy?”

    We do have to be a little more mindful that, yes, there are different needs within an organization, but to create more silos, which is what many organizations have tried to do when they’ve tried bimodal, just creates an incredibly more complex problem and takes the organization in the direction away from where we want to go. Especially if you believe in the value of DevOps, for example.

    Now, I’m not saying that bimodal is completely bad and should be thrown out, but there needs to be a very mindful way in which to leverage it. And, at some point, as part of your vision, as part of your road map, you need a means to bring that all back together again.

    Hulme: So, what is an answer to that? Legacy IT, because some of it’s just not going to go away overnight.

    Crawford: You have to divide and conquer. I mean if you try and take the entire IT portfolio in one big bite, that’s the equivalent of the old adage of eating the elephant all at once.

    You can’t do that. I mean, you didn’t build that IT system—the people, the org structure, the technology, the applications—overnight. It took time to put that together. And much of how it’s built is institutional knowledge. It’s not documented anywhere. It sits in George’s head. It sits in Tim’s head. And then what happens when Tim leaves the organization and then something breaks or something doesn’t work? Oh no! Well, we don’t know how to fix it. We now have to re-solve that problem all over again because it was in Tim’s head and Tim’s not here anymore.

    So, you have to divide and conquer. I use the methodology that I’ve developed over the years to do that. To bring some clarity to the areas that require more focus. To get to a position where you can take advantage of newer methodologies and newer technologies like cloud, like containers and some of the new AppDev methodologies, you have to understand what you have and then you have to have a means in which to pick apart the pieces. So you might say, you might take the ripe, low-hanging fruit and say, “You know what? Okay, just because we have file sharing on-prem today in our own data center doesn’t necessarily mean we need to do that moving forward.” Yes, we’re a regulated industry. But there are mature solutions that allow us to audit documents and understand where documents go today on cloud systems. So, let’s go leverage this cloud storage service. Let’s move that out. Great, we just lopped off a huge amount of storage and file servers.

    After that is completed successfully, you move on to a new phase, perhaps you don’t have to run an Exchange server in your environment anymore. There’s nothing unique about it. Yes, you configured the hell out of it. Yes, you couldn’t replicate exactly all those configurations, but do you really need to? And, this kind of thinking is actually a very mature way of thinking compared to where a lot of organizations are today.

    Hulme: You may be bringing old thinking forward that is not necessary anymore.

    Crawford: Exactly, and that’s part of this theme. A lot of that thinking is what hinders us moving forward.

    A lot of that legacy thinking gets pulled forward and this is one of the reasons why I’ve said this before and I’ll say it again: Moving forward, it would almost help everyone if we forget everything we’ve learned up until this point. I know that’s incredibly counterintuitive, but in many ways legacy thinking is what is holding us back—that thinking that we’ve always done things this way, so we’ll do it this way moving forward.

    Note that a common question that gets asked by IT leaders when you want to bring something new and innovative to the table, and this used to drive me batty as I was coming up through rank and file of IT. It was when the CIO would ask, “Tell me who else has done this in our industry at our size.” Now, if you think about it, when the CIO says that, what they’re looking for is stability. They want some validation to know that things are going to be okay.

    Now, let’s change the CIO with the CEO. If you go to a CEO with a new innovative thing, is the CEO going to ask you who’s done it in our industry at our size before they can seriously consider it? Or they going to be looking at it and go, “Ooh, this could be a differentiation opportunity.”

    I don’t care who’s done it before. Doesn’t matter. I want to know what people want and how we can deliver something that’s of value to them. And if it’s a different mindset that is about embracing the opportunity.

    Hulme: What are some indicators organizations should look for to see if they are thinking transformational, or with a legacy mindset?

    Crawford: One key indicator that I look for when I get asked to go in and assess organizations is experimentation. Is experimentation not just allowed, but encouraged? Now, experimentation is a good indicator, but it’s no more than an indicator, but it shows a mindset and an openness for new thinking. So, when you start talking about how do you determine if an organization is innovative? OK. Are they trying things? Are they failing? If they’re failing, great, because that indicates that they’re at least trying something and they’re not afraid to fail. Most times in IT organizations, failure is a big no-no; you want stability, you want an outcome that you’re expecting. Obviously, you have to kind of take that statement with the right perspective? You can’t just fail, fail, fail. The business still has to be able to operate. But you can’t be afraid to fail, either.

  • DevOps Q&A: Datalink’s Dennis Vickers on MobileApps

    DevOps Q&A: Datalink’s Dennis Vickers on MobileApps

    Companies are custom-building mobile enterprise applications to enhance employee productivity on smartphones. devopsy.kinsta.cloud recently spoke with Dennis Vickers, director of Software Solutions at data center services and solutions provider Datalink about this trend and where DevOps fits in to the process.

    devopsy.kinsta.cloud: Why does applying smartphones to business initiatives require so much additional custom-built software?

    Vickers: Companies have developed and designed legacy software to group things that need to get done together into applications. With mobile application development the focus is more granular in terms of functionality. It’s not that the applications that we designed for the legacy environment don’t fit on the smaller screen, though they don’t. It’s really that the whole concept of having a group of functionalities in a single application doesn’t really work in a mobile environment. You need to rethink the way that not only your applications work but really the way that you do work. You need to think more in terms of granular work than accumulation of work.

    In finance, for example, you might have an application that combines accounts receivable, accounts payable, even expense management. You would have an application that would have all of that functionality in it. For mobile apps, what would make sense is to break out the expense management part of that and make that a mobile app that allows users to be able to use their smartphones to actually capture expenses as they’re incurred, as opposed to having to save a receipt or remember what your expenses were and then go back to an application and fill that in later at the office.

    devopsy.kinsta.cloud: How do you apply DevOps development approaches to provide usability on mobile devices for the most needed features and functions of in-house applications?

    Vickers: Mobile applications tend to be single functions focused on getting something done. Users are much less tolerant of anomalies in the software, whether that anomaly is a bug, a poor user interface or anything that is undesirable about the application. They’re not nearly as willing to look for workarounds with a mobile app as they are with legacy applications.

    You have to have a continuous delivery approach with mobile applications because the time frame between deliveries has to be accelerated. You have to automate that process. We’ve made good use of some of the automation tools, specifically Puppet, around doing continuous delivery. We use tools for source control and QA (quality assurance). Some type of automated delivery process is critical because the turn time for release has to be shortened. There’s just not a willingness for users to wait long periods between releases.

    devopsy.kinsta.cloud: Can you discuss the custom-built software project where you developed a custom equipment maintenance app that feeds into the customer’s proprietary maintenance application?

    Vickers: We developed that for an aerospace company that had a requirement to provide field service for one of the Department of Defense systems. They had to be able to maintain large pieces of equipment. We had to build an application that enabled them to be at the equipment and do the data collection that they needed at that time. It walked them through the data collection process right then.

    Before that, they had a paper process where they would print out instructions, go out and write down the information that they needed to collect, and then go back and enter that into an application. This allowed them to be able to do that closer to the actual event, which cut transcription errors and data collection errors and made the whole process much more efficient. That was an extension of a legacy app that added quite a bit of value.

    devopsy.kinsta.cloud: For that Department of Defense project, were there particular development tools or approaches in DevOps for the software?

    Vickers: We used an agile methodology where we iteratively released the software and did test and review and then essentially layered the application. We provide a layering process as we build and add features to the application. We delivered to the customer through an automated process using Chef. We use Jenkins to do some release management, but we often revert back to Chef or Puppet.

    What we haven’t done a lot of yet, which I think is important and I see growing in the mobile platform, is monitoring. I think that monitoring is an important part of DevOps. I haven’t seen that move into the mobile environment nearly as much as I think it needs to. I think it will happen in the short term. I think there’s some good monitoring tools out there that are coming online or that are maturing and are going to be adopted for mobile app dev.

    devopsy.kinsta.cloud: There was another project where you custom-built a jobs app that enables union members to apply for posted jobs without going into the union hall.

    Vickers: We developed a mobile app so union members can review jobs that are available and request the job through the app. They can receive job assignments and electronic tickets to go out and do that job. It saves traveling down to the union hall on a regular basis and makes the whole process much more efficient.

    The only difference in that project is that we needed to integrate with the back end through an API. The tool we used to help us interface with that API and manage that API integration was Postman. We use it for testing, managing and documenting APIs we build or work with.

  • QA in the DevOps Era

    QA in the DevOps Era

    In software approaches that produce monolithic applications, it is not uncommon to find siloed teams, with development, testing and operations each an island unto themselves. The responsibilities of test and development are distinct from each other and are not meshed into a hybrid team as you might see in advanced DevOps shops.

    “QA (quality assurance) always comes late in the game to verify or test the software developed by the development team against the predefined specifications know as software requirements,” says Venkata Vinay Pandella, team lead at Nisum Technologies Inc.

    That’s the view inside and among these teams in which legacy development approaches still flourish. The rest of the enterprise sees these teams altogether differently from how they see themselves. “Even in a traditional software engineering organization, folks outside the engineering team generally consider Development, QA and Operations teams as a single group with a single responsibility to deliver a quality software product,” says Pandella.

    Even when Dev and Ops do embrace each other, the rest of the enterprise often maintains the same POV it had when Dev, Ops and QA were three entities and not one. How do DevOps and QA see QA after the shift?

    DevOps as a Foundation for the New QA

    Enterprises, the business units, Dev and Ops all expect DevOps to propel software using continuous integration (CI), continuous delivery (CD) and ultimately continuous deployment techniques that are fueled by and gain traction using automation. The more processes that Dev and Ops teams are able to automate, the more that DevOps accelerates the process of software delivery and deployment to the delight of software consumers and the business.

    DevOps enables teams to push software out weekly, daily and hourly through continuous software improvement. This negates the many obvious downsides of applying the traditional software development approach, which requires excessive planning, cross-team coordination, staging, testing and releases of monolithic blobs of software all at once. “Given that companies organically adopt DevOps together with agile software development practices, business managers are able to fulfill the objective to get high-quality software out to market faster,” says Pandella.

    To meet the goals of DevOps, QA must feed into the automated development process and help teams to achieve increasing software quality and faster time to market.

    QA: The Quality Icing on the DevOps Cake

    According to Pandella, QA specialists and teams have a number of roles and responsibilities in DevOps:

    • QA should prevent software defects at the earliest point during the software development process rather than find and report on software bugs after DevOps has produced the software. This will help to ensure that customers experience as few defects as possible.

    • QA takes ownership of and responsibility for continuous software improvement and software quality tracking throughout the software development life cycle.

    • Every QA team member is responsible for identifying problems not just in the product but also in the process, and recommending changes wherever they can to improve the continuous delivery of quality software.

    • Tests must be in code, based on the designed release cadence to publish a new release every week, every day or every hour. There is no room for manual testing—QA must develop automation systems through code that can ensure quality standards are maintained.

    • Automation rules: QA should automate anything that can be automated.

    • QA testers are the quality advocates, influencing both developmental and operational processes. They don’t just find bugs; they look for any opportunity to improve the quality of software through the software development process.

    • QA must move beyond functional testing, because even a small, nonfunctional defect introduced into a critical transaction process on an e-commerce website can bring a business to its knees. Many organizations have mature processes for automating functional testing; they are only beginning to apply these practices to other areas of testing such as security and stress testing. In particular, load testing and stress testing are critical disciplines for DevOps organizations that move at high velocity.