devopsy.kinsta.cloud

Tag: collaboration

  • Chef Boards the DevOps Express with CloudBees, Others

    Chef Boards the DevOps Express with CloudBees, Others

    You often may have more questions than answers as you progress along your DevOps journey. Which tools should I choose? Do these tools work well together? What have other people done in my situation? You’re often left Googling to find the right integrations and piece together an architectural solution.

    Today, I’m excited to announce that Chef is joining DevOps Express, a new industry initiative comprised of 14 leading DevOps vendors and service providers. Led by CloudBees and Sonatype, and including everyone from Atlassian to GitHub, this initiative is focused on creating solutions that will help you adopt DevOps practices. Specifically, by defining reference architectures and highlighting best practices, supported integrations and solutions, we believe that DevOps Express will simplify and accelerate DevOps adoption for organizations of all kinds.

    DevOps Express uses popular technologies spanning multiple solution components to provide a framework for building reference architectures that are better integrated and supported. Creating reliable and actionable reference architectures for organizations will ease DevOps adoption and minimize risk for organizations.

    DevOps is collaborative by its very nature—it’s all about bringing teams together. This includes cultural transformation, breaking down silos between disparate teams and working together around the common goal of adding value for customers. These principles are just as true for us, the vendors providing tools to help you on your DevOps journey. We, too, must break down silos and work together to provide more value for you, our common customer. I’m looking forward to what we’ll build.

    About the Author / James Casey

    james_caseyJames Casey is vice president of Partner Integration at Chef. His career spans more than 15 years in companies ranging from mobile and finance startups to CERN, the birthplace of the World Wide Web. He specializes in building tools for operating large computing infrastructures. He previously created monitoring tools at CERN for the Large Hadron Collider.

  • Can Culture Detract from Success with DevOps?

    Can Culture Detract from Success with DevOps?

    Collaborative culture is considered to be an essential best practice for enterprises. DevOps culture correlates strongly with enterprises that achieve the highest level of business performance. However, is it possible that an organization can get overly fixated on organizational culture improvements? Can culture actually detract from achieving success with an enterprise DevOps transformation?

    An article by Curtis Franklin Jr.,”8 ways to fail at DevOps,” indicated that there are “plenty of places in the DevOps process where an organization can run aground on rocky shoals of woe.” My blog, “7 Pillars of DevOps,” identified seven essential best practices areas that make up successful DevOps. One of the seven pillars is collaborative culture. These pillars are not silos, but rather foundations that need to be kept in balance.

    I bet most of us have experienced situations in which cultural improvement missteps hindered an enterprise DevOps transformation.

    Culture Missteps

    I’ve listed some examples l below which I have personally seen. Have you experienced these? I would love to hear examples of your experiences and solutions you have observed, too.

    1. Unrealistic expectations from leadership: Business expectations from leadership get too far ahead of the DevOps implementation reality—then teams reporting to leadership end up feeling unappreciated and experience frustration, resulting in organizational friction.
    2. Ops-side transformations fall behind Dev-side transformations: The Dev team accelerates ahead of the Ops team and implements “its part” of a DevOps transformation before the Ops team has a chance to implement complementary changes. The Dev-side is disappointed that Ops has not kept up with the complementary Ops-side planned changes. Dev changes cannot be put into effect, which deepens the mistrust between Dev and Ops.
    3. Dev-side transformations fall behind Ops-side transformations: This is the reverse of the second example, with similar effects.
    4. QA tester roles are confusing: When new continuous test strategies are implemented, the lines of test responsibilities between Dev, QA and Ops team members became blurred. Communication between cross-functional team members become confused, resulting in misunderstandings, dissatisfaction and organizational friction, which obstructs progress.
    5. Infrastructure and tools teams are siloed: Dev, QA and Ops teams maintain separate, siloed tools and infrastructure staff and do not cooperate to support end-to-end infrastructure and tools improvements. The transformation needed for a smooth DevOps pipeline stalls.
    6. Rewards and measures are not consistent with DevOps goals: The organization makes changes toward a cooperative culture but fails to change the way people are rewarded and measured. Conflict results between the teams.

    Solutions

    Below are solutions that were applied to the situations listed above. Do you agree with these solutions, or have others to suggest?

    1. Leaders were coached to be more aware that DevOps transformation is a journey of continuous improvement. Leaders then understood that there is really no “end state” to DevOps—there are specific business improvements that can be expected along the way. Each DevOps improvement initiative was refocused on realistic specific use cases and goals. With improved clarity, leaders were more realistic in their expectations for each improvement. This drove alignment between leaders and their teams, improved satisfaction and productivity.
    2. Project goals and team meetings were realigned and organized. Dev and Ops sides could monitor end-to-end progress together. When situations occurred in which one team got ahead of the other, the end-to-end progress visibility helped overall group communication. Empathy across the Dev, Ops and QA teams improved. With improved communication, team members celebrated the success of each other’s improvements. Teams shared a common end-to-end goal.
    3. Same as 2.
    4. QA testing responsibilities changed significantly during the DevOps transformation. Separate independent test team member responsibilities changed toward QA testers being more integrated with Dev teams. New skills requirements for the QA engineers emphasized test automation more than in the past. Training for the cross-functional team helped mitigate concerns related to the new QA responsibilities. QA testers were uplifted with new skills and became more valuable and respected by their peers. Cross-functional productivity improved.
    5. The tools and infrastructure teams from the separate Dev, QA and Ops teams were combined into a common tools and infrastructure team. Project goals were realigned. The improved clarity around knowing they are working toward tools and infrastructures for a common end-to-end pipeline accelerated implementation of DevOps.
    6. Rewards and measures that had been in place for separate Dev, QA and Ops roles were changed to rewards and measures that encouraged end-to-end cross-functional team activities. This improved overall morale and job satisfaction and reduced organizational friction that was previously impacting progress.

    Summary

    From the above real-life examples, it is clear to me that culture and organizational practices must evolve in concert with progress on other DevOps pillars. And as the pace of change increases with each transformation, the cross-functional teams must be even more cohesive. Continuous acceleration of interactions between team members drives the need for tighter relationships, more end-to-end responsibilities, improved customer awareness, continuous skills training and end-goal focused leadership.

    The is no need to create the “perfect culture” before starting a DevOps transformation journey. It can be counterproductive for sections of the culture to get too far ahead of itself, as indicated in the examples above. Perhaps it is best to evolve the culture carefully. Changes most important for each transformation project should be the focus. Continuously communicate goals and progress, and evolve the culture together with evolution of other pillars. What do you think? Do you agree is it possible that an organization can get overly fixated on overall organizational culture improvements?

    While DevOps implemented with all seven pillars provides a strong foundation for long-term enterprise business improvements, it is important to understand DevOps is not an island. Enterprises implementing DevOps should be aware that DevOps interoperates with other IT systems and practices. Enterprises are well-advised to choose tool-agnostic IT partners that can provide solutions that best suit the needs of each unique enterprise and can integrate and evolve DevOps together with all of their IT systems.

  • REP SPOTLIGHT: Glenfis AG

    REP SPOTLIGHT: Glenfis AG

    1. Tell us a little about your organization: Based in Zurich, we offer consulting and training services since almost 20 years. We see ourselves as an independent enabler for the digital transformation, helping our customers getting fit for the digital age. To this end, we use the ‘Glenfis Principle’ – a holistic approach that, alongside theoretical knowledge, attaches particular importance to the mediation of practical skills (competence), which can be used to successfully and lastingly deploy this knowledge in the company or organisation (Implementation).
    2. What other topics do you teach or consult on? Topics include:
      – Cloud & Sourcing
      – Lean, Agile & DevOps
      – Value Chain Service Models
      – Training and Awareness (Cloud, ISO, ITIL, Cobit, Security, Agile, DevOps)
      – Security & Cyber Resilience
      – Change Enablement
    3. What type of interest in DevOps are you seeing in your region? As everywhere the interest of DevOps is rapidly growing, in Switzerland this in not different. For many it’s really to understand in a first step what DevOps is about and why it is relevant for them. In Switzerland we deal a lot with financial institutes with a high level of legacy and regulations, the question is often how to apply DevOps practices in such an environment.
    4. How do you think that DevOps contributes to the business’ value stream? DevOps is all about getting faster and better and this is how it positively influences the value stream. And DevOps does it in a holistic way by looking at the value stream end-to-end, from idealization to $$$.
    5. What topic area in DevOps excites you the most? Dev’s and Ops’ finally talking to each other!
    6. Why should enterprises and individuals consider formal DevOps Training? Before starting any change initiative, it is paramount that everybody gets a common understanding. This is especially true for DevOps, as it includes a wide range of Agile, Lean and ITSM practices. A training or special awareness sessions definitely facilitates this.
  • Iterative Indicators: Measuring DevOps ROI

    Iterative Indicators: Measuring DevOps ROI

    It’s well-established that application of ROI measurements to IT investment remains an imperfect science, and the same goes for ongoing adoption of DevOps methodologies.

    The fuzzy math of vendor-driven ROI calculators aside, even the most respected industry watchers have long struggled to create concrete parameters for assessment of technology spending. This, in large part, has been related to the fact that the requirement for continued investment has outweighed the value of short-term metrics.

    At the same time, it’s hard to put a price on the influence that paradigms including mobility, virtualization and the cloud are having on today’s business world. The scope of the so-called Applications Economy is only truly beginning to be realized.

    For in-depth discussion of DevOps ROI, register today for the upcoming webcast “Maturity Metrics – Analyzing the 2016 State of DevOps Report” with noted industry expert Gene Kim.

    With DevOps, the challenge of ROI measurement evokes similar characteristics as with these other bellwether movements. The affect that leveraging DevOps to become more agile and respond adeptly to changing customer expectations will have, long-term, ultimately will be realized as much in organizational viability as in any immediate cost-savings or other factors.

    That said, the recently published 2016 State of DevOps Report, assembled by the researchers at DevOps Research and Assessment (DORA), goes further down the path toward creating meaningful ROI metrics than any other such previous effort.

    For starters, the 2016 report builds on previous year’s conclusions that so-called “high-performing organizations”—those already deeply engaged in DevOps practices—are “decisively outperforming their lower-performing peers in terms of throughput,” and this gap continues to widen.

    Moreover, DORA contends that high performers deploy applications code 200 times more frequently than slow performers and achieve 2,555 times faster lead times. Perhaps even more importantly, in particular as it relates to the ability to improve applications performance, the report estimates that leading DevOps practitioners lay claim to 24 times faster recovery times and three times lower change failure rates.

    The State of DevOps Survey, completed by roughly 4,600 technology practitioners, also found that high-performing teams spend far less time on so-called “unplanned work and rework”—most often required to address existing problems—allowing them to spend nearly 50 percent more of their time on new work. The latter brand of effort is typically aimed at adding or improving feature functionality—or, in simple terms, innovation.

    As increased software velocity and reduction of MTTR represent the most critical differentiators in today’s environment, where an end user may wait only seconds for an app to load before abandoning your brand altogether, these conclusions obviously carry impressive weight.

    Finally, based on a framework established by research analysts at IDC which suggests that hourly application downtime costs can range from $1.25 to $2.5 billion for a Fortune 1000 firm, and that the average cost of a critical application failure is $500,000 to $1 million per hour, the State of DevOps Report allows for the conclusion that high-performing organizations save an average of more than $91 million per year.

    Much of this advantage is found, the researchers note, in the form of lower applications change failure rates, or those updates to software code that result in subsequent performance issues or outages. While high-performing DevOps practitioners experience an average change failure rate of 7.5 percent, medium-performing organizations see that figure rise dramatically, to a whopping 38 percent.

    If the undeniable adage that “time is money” applies in today’s Applications Economy as much—if not more than ever—than these ROI calculations would appear to infer substantial meaning.

    Later this month, Gene Kim, one of DORA’s founders and a key contributor to the State of DevOps Report, will host a webcast with CA Technologies expert Aruna Ravichandran to discuss DevOps ROI and many other related topics. Register for that event today.

  • REP SPOTLIGHT: getITright

    REP SPOTLIGHT: getITright

    DevOps Institute: Tell us a little about your organization.

    getITright: With offices in Australia, South Africa, The Netherlands, Nigeria and Malta and a network of partners operating in 31 countries, we offer a bouquet of services to help organizations to:

    • Manage better
    • Lower risk
    • Perform better
    • Make better business and IT decisions

    Our service offering includes:

    • Training courses – Learning best-practice through our Partner Academy courses or network of training providers
    • Accredited IT Best practice course material – License our training material
    • Consulting – We use an agile approach for projects and a lean approach for operations when consulting and coaching.

    DevOps Institute: What other topics do you teach or consult on?

    getITright:

    • Service Management: ITIL®, ISO/IEC20000, Lean IT, Service Desk Institute
    • Governance of Enterprise IT: COBIT® 5
    • Project Management: PRINCE2®, PRINCE2® Agile, DSDM AgilePM®, Scrum
    • Information & Cyber Security Management: ISO/IEC27001, NIST Cybersecurity using COBIT5
    • Quality Management & Business Improvement: ISO9001, LEAN SIX-SIGMA
    • Risk & Continuity Management: ISO31000, ISO/IEC27005, ISO22301
    • Cloud Computing

    DevOps Institute: What type of interest in DevOps are you seeing in your region?

    getITright: DevOps awareness in the region is still low, but we see a steady increase in the level of interest in the topic.

    DevOps Institute: How do you think that DevOps contributes to the business’ value stream?

    getITright: It makes projects and business as usual faster, more stable, with less risk.

    DevOps Institute: What topic area in DevOps excites you the most?

    getITright: DevOps enables organizations to improve productivity and respond to change with less risk. It’s easy to adopt by organizations with mature ITSM practices in place.

    DevOps Institute: Why should enterprises and individuals consider formal DevOps training?

    getITright: Training ensures that all resources in the organization are in sync when implementing new practices like DevOps in an organization and ensures a cohesive approach that facilitates proper organizational buy-in and adoption.

  • Wake Wants to Help Streamline Your Collaboration

    Wake Wants to Help Streamline Your Collaboration

    Effective collaboration is imperative to DevOps success. DevOps has many moving pieces working simultaneously and things happen fast, making it crucial that teams and individuals have an easy way to communicate and keep everyone on the same page. Wake is a collaboration tool that may help you do just that.

    “Collaboration across the enterprise is critical for DevOps success,” I wrote recently in an article on the importance of effective collaboration. “Great code and development needs to make it over the finish line to production to benefit customers. The challenge organizations face is how to do that seamlessly and with as much speed and automation as possible without sacrificing quality or performance.”

    “Just as open source is associated with cost savings and flexibility, DevOps is associated with speed and responsiveness,” stresses Jay Lyman, enterprise software analyst for 451 Research. “Enterprises are increasingly interested in initiating or growing their DevOps thinking and implementations, but they must be aware of the dramatic change it brings not only for technology and tools, but for people and process as well.”

    Wake is a collaboration platform used by teams and developers at Facebook, Instacart, Zendesk and other organizations to communicate effectively and share work with others. Wake was created as an internal project at digital agency Bakken & Baeck by Chris Kalani, a former Facebook designer. Silicon Valley companies Medium and Airbnb loved it and begged to be able to use the product themselves, so it was spun off into its own company. When Wake launched last fall, 350 companies signed up in the first month, including Cisco Systems, OpenTable and Uber.

    “Adjacent trends such as cloud computing, DevOps and containers are all speeding up the time it takes to iterate and deploy software and the movement of applications and workloads from test and dev to production, but the reality for enterprises is they cannot simply start fresh with a DevOps approach,” Lyman says. “Existing infrastructure, applications, people and process must be considered and integrated so that the transition to DevOps is not only less painful, but more realistic as well.”

    Wake ostensibly helps facilitate the increased speed and agility by simplifying communication and collaboration. It has a bit of a Pinterest look and feel to it, but Wake makes it easy to share your progress on what you’re working on without leaving the application you’re working in. It connects with messaging systems such as Slack and HipChat, and you can find what you’re looking for quickly by searching for specific people or tags.

    A Wake spokesperson tells me that Wake will soon launch significant updates to its enterprise product as it builds momentum with bigger brands including Capital One, Netflix and Mozilla. You can check out some use cases of Wake at Instacart, GoPro, Intercom, OpenTable and other places in this post on Medium.

    Pricing starts at $125 per month for five contributors and 10 spectators. There is a Plus subscription for $350 per month that includes 15 contributors and 30 spectators. Beyond that, pricing is negotiated for larger organizations. You can try Wake for free for 30 days.

  • State of DevOps Report: Reaping Results

    State of DevOps Report: Reaping Results

    The newly released 2016 State of DevOps Report concludes that DevOps is “no longer a mere fad or buzzword,” but rather an established set of practices and cultural patterns. Yet, a closer look at this year’s findings reveals progress is far more substantial than any matters of perception.

    The report was conducted and analyzed by the experts of DevOps Research and Assessment (DORA) and sponsored by CA Technologies. Among its findings is proof that high-performing organizations that have embraced agile and DevOps methodologies deeply are outperforming massively those that have not. And the gap is widening.

    The State of DevOps Report also shows leading DevOps practitioners are appreciating benefits that span nearly every area of their business. If I know my colleague Gene Kim, he will assert this point exuberantly on our upcoming State of DevOps Report Webcast.

    It’s noteworthy that high performers are surpassing incredible milestones related to core elements of the applications life cycle. Examples include deploying code 200 times more frequently than low performers, driving 2,555 times faster lead times, 24 times faster recovery times and producing three times lower change failure rates. This is evidence that DevOps is having the desired outcome on applications delivery, software quality and organizational velocity. It also shows organizations truly committed to wholesale transformation of culture and process are actively succeeding.

    At the same time, the State of DevOps Report shows employees working at organizations enjoying this success are more satisfied with their jobs and more closely engaged with their companies. Chalk this up to the halo effect of operating in a fundamentally more sensible, innovative fashion. Cultural change is undoubtedly the greatest hurdle to DevOps advancement. Here’s proof that if you can navigate the process, your workers will be happier and more productive.

    Using Employee Net Promoter Score (eNPS) as a barometer, the report finds that employees in high-performing organizations were 2.2 times more likely to recommend their organization as a great place to work. They also were 1.8 times more likely to recommend their team as a great working environment. Previous studies have found that such an air of positivity can be directly correlated with improved business outcomes, researchers noted.

    Shifting the manner in which you build applications to better adapt to changing customer expectations is a necessity. That’s what began driving this movement years ago. That said, proof it also makes it easier to keep valued software engineering and IT operations employees should not be overlooked.

    In addition to positive conclusions on the impact of continuous delivery and improvement of test data management, the report offers some important metrics related to return on investment. This is nothing short of landmark, as for decades we have struggled to make a hard case for IT investment beyond implied competitive advantage. It actually was impressive that last year’s report highlighted such a “predictive relationship between IT performance and organizational performance.”

    One such new measurement of note is that high-performing teams occupy the least amount of time responding to unplanned work and rework—efforts widely recognized to be results of flawed processes. On a related note, DevOps leaders spend 49 percent of their time on new work‚ surely, in many cases, the next “must-have” feature which impacts competitive standing and, ultimately, the bottom line.

    This report is fantastic. It reaffirms powerfully that so many of the promises made about DevOps over the last few years are neither hype, nor overstated. The gap between the leaders and laggards is growing. Industry landscapes and business models are being redefined.

    This is critical transformational work that every organization must adopt to survive.

    Are you on board? Download the full State of DevOps Report here.

  • DevOps Matchmaking: APM + CI = A Perfect Marriage

    DevOps Matchmaking: APM + CI = A Perfect Marriage

    As anyone who has ever been set up on blind dates by friends can attest, finding the right match in today’s world is a tricky feat.

    There’s a lot of evidence suggesting that the frantic pace of life makes such matchmaking more difficult than ever. That said, when the right set of complimentary traits exist and there’s a willingness to find common ground, things can work out.

    Spike Jonzian-notions of artificial intelligence aside, certain technological partnerships are increasingly being perceived as “a perfect marriage” within the context of DevOps automation.

    Register today for the upcoming webcast: “Using APM to Turbocharge Continuous Integration”

    These technological matrimonies closely support the underlying tenets of DevOps methodology—collaboration, experimentation and increased velocity. In fact, it’s easy to wonder why no one ever conjoined them before. But we were using traditional waterfall development and operations and so couldn’t conceive of such a thing. On the other hand, emerging availability of tooling supporting new integrated workflows represents a significant catalyst.

    Finding the Perfect Fit

    One such DevOps confederation getting a lot of attention is the matchmaking of continuous integration (CI) and application performance management (APM) technologies. Tying together these once-disparate systems makes a lot of sense, despite the fact that few tools would seemingly appeal to more (traditionally) polarized constituencies.

    Continuous integration has been a staple of the development community for many years. It emerged as teams evolved the practice of long build times into more frequent check-ins to deliver code and look for potential errors more quickly. Today, with the rise of CI tools such as the Cloudbees Jenkins platform, some development teams are touting tens of thousands of builds per day.

    On the other end of the spectrum, squarely on the turf of IT operations, APM has been evolving rapidly over the past few years as organizations have sought to improve use experience and applications reliability via dedicated monitoring. Enabling this ability to identify and address emerging issues has resulted in APM becoming the fastest-growing segment in the IT ops management market, according to Gartner.

    DevOps matchmaking factors in when you bring these tools together to create an automated workflow that spans their respective use cases. A cross-functional team can leverage APM-level monitoring and KPIs within the domain of CI and testing to identify potential performance obstacles introduced in new code. Doing so then rather than waiting to respond later in the applications life cycle benefits everyone.

    Reaping the Benefits

    For example, say an e-commerce applications developer has implemented a new inventory search functionality—something considered a key differentiator by business management. The developer knows this feature could rely heavily on some back-end database system and wants to ensure it’s tested properly, even as the code is being introduced.

    Integrating APM and CI allows the entire team to monitor the process and test various conditions related to the update, tracking multiple builds and looking out for related performance issues. Such integration also allows for centralized tracking of other key attributes such as build number, status and build date, to further ease management.

    Most importantly, by tying these two critical aspects of the applications delivery life cycle together, the collaboration and feedback loops required to truly embrace DevOps methodology become ever more valuable as these core Dev and Ops functions become intrinsically linked.

    If this type of DevOps advancement gets your heart pumping, you’re not alone. Later this month, we’ll host a webcast here on devopsy.kinsta.cloud featuring experts from CA Technologies and Cloudbees. They’ll be discussing the benefits of matchmaking with APM + CI integration. They’ll also detail real-world benefits organizations leveraging integration of our respective technologies (CA APM and Cloudbees Jenkins) enjoy.

    We obviously feel that this is a DevOps tooling marriage that will mature, change and stand the test of time. We’d love to garner your questions and feedback on the topic.

    Which is all good, because thankfully there’s no such thing as Tinder for DevOps … yet.

  • MTTR: The Magic Metric

    MTTR: The Magic Metric

    How do you improve with one metric quality, cost, performance, employee and, most importantly, customer satisfaction?  I think MTTR is one to consider, but with a DevOps twist (agile+lean+ITSM). The thing I love about DevOps is that it allows for people to think differently. DevOps is the movement driving organizations to find new ways of collaborating, communicating, cooperating and improving the way they use technology to create and support services.

    While many agree that MTT stands for “Mean Time To,” there is no common agreement on the R.  I like using 5 Rs, as they cover the life cycle of idea to realization (in the hands of the customers in a way they like).

    So what are the 5 Rs? Well, let’s work backward (shift-left thinking):

    • Review: What happened? How can we stop this in the future? How can we shorten the cycle?
    • Return: Do customers agree that all is well? How satisfied are they with not only our performance but the fact the event occurred?
    • Restore: The time and effort to get servers, network, applications, data, passwords back as they should be.
    • Repair: The time and effort to resolve the issue.
    • Respond: The time to notice, decide if we can fix or get others involved, and their reactions to begin getting things sorted.

    MTTR is a cycle of events. Breaking them down and understanding the value of each step enables you to reduce time, effort, waste and cost. How often do things break and why? Are they self-inflicted because we test poorly? How can you apply automation or self-healing? How good is your monitoring and alerting and response mechanisms? How great is your documentation? Do you rely on a certain person to fix things or can anyone? Knowing the answers to these questions can make issues become less frequent—or, at least, reduce the effort to address them when they do occur, as things will break.

    Knowing these answers also helps clarify priority of issues. Working back from the customer drives the flow of work to highlight faults. Why treat all issues as P1 all month? For example, payroll is only a P1 when you are getting paid, but the rest of the month it can be a P2. Working with customers will help ensure you are fixing the important things, not something someone wrote in a service level agreement that is rarely updated.

    MTTR is also a fabulous learning measure. DevOps is all about collaboration and that begins when people talk to each other, repair things or introduce a better test or monitor or alert to capture issues and customers are minimally impacted. Let your service desk own MTTR and see how issues get resolved better, faster and more safely.

    MTTR helps drive movement to virtual or cloud. MTTR can also help you improve your A/B use of infrastructure or services.  Have an issue? We’ll flip a switch.

    But the overall goal of any metric is to drive actions, decisions and improvement. MTTR is such a measure. Be brave, throw away your SLAs, get people involved for issues and use MTTR to make things better across the life cycle. Then celebrate the success of your journey!

    Let us know. Do you use MTTR? Can you measure each step and perform a retrospective?

    About the Author / Daniel Breston

    dDaniel Breston is the Chief of DevOps Transformation for Ranger4. Daniel has 30+years leading IT Service, Operations, Applications, Data Center and service partners in management of technology enabling business processes and functions. He has also spent 15 years of consulting and coaching leadership teams across the EU and UK in developing their people, processes and use of technology to deliver technology services better, faster, safer. Daniel has qualifications in IT Service Management (ITIL, COBIT,etc.), Lean and DevOPS.

  • Enterprise Cassandra Deployments

    Enterprise Cassandra Deployments

    Successful Cassandra deployment in the cloud takes a certain amount of skill and knowledge. In this blog I will share best practices for a successful Cassandra deployment in the cloud such as Amazon Web Services (AWS) and Google Cloud Platform (GCP). I will also discuss how you can protect and recover the data in your Cassandra cluster when deployed in the public cloud.

    Cassandra Deployments in the Public Cloud

    There are two main ways to have successful enterprise Cassandra deployments in AWS:

    • Use a ‘managed’ Cassandra deployment. This is a reasonable approach; the only downside is the inability to customize data placement beyond what is typically provided by the vendor.
    • Manage the cluster yourself, if you want to have full control over your Cassandra deployment and customize it to scale and perform according your business needs.

    Please refer documentation from Datastax or Apache Cassandra for detailed best practices for Cassandra deployments. The following best practices are reiterated here to simplify deployment of a scalable Cassandra cluster in AWS.

    Storage

    Both AWS and GCP offer two types of local storage options for their compute instances. For AWS, the options are Elastic Block Storage (EBS) and Ephemeral Storage. In GCP, the options are Persistent Disks and Local SSD. Both options have their own trade-offs. Pick the one that best suits your deployment.

    Persistent Storage

    EBS volumes have performance characteristics similar to network storage such as a NAS or SAN. Because storage from EBS volumes is multitenant in nature, it is not suitable for distributed, log structured databases such as Cassandra. If you want to ensure reliable performance through EBS volumes, use “EBS optimized instances” with “provisioned IOPS.” Since the IO traffic to EBS volumes goes over the instance network, EBS optimized will provide reliable performance, but does not guarantee any increased performance. To get the most performance out of EBS volumes, use instances with multiple disks and separate the volumes used for “Data” and “Commitlog.”

    If you have to use EBS Volumes, go with “EBS Optimized Instances,” with “Provisioned IOPS” and split data and Commit Log on different volumes.

    Ephemeral Storage

    Ephemeral storage is a great option to use for downstream Cassandra environments such as test/dev and staging. Since ephemeral volumes are typically SSDs, there is no need to separate the data and commitlog into separate drives. To optimize performance, choose EC2 instances with multiple ephemeral storage drives and stripe them together into one logical volume. Send data and commitlog to the logical volume. Remember that when you stop the EC2 instance, the data in Ephemeral storage will be deleted.

    Stripe all Ephemeral drives into a single volume to get maximum performance.

    Data Distribution

    Cassandra uses Snitch to determine the topology of the cluster. The Snitch is also used to determine the data center and rack for each node in the cluster. Replication strategy is used to determine the placement of data across the different nodes in the cluster.

    AWS

    1. EC2Snitch works well for single region deployments.
    1. Ec2MultiRegionSnitch works well for cluster distributed across multiple AWS regions. Please note, Multi-DC, Active-Active configuration makes your cluster resilient to region failures and for multi-dc cluster, use “local quorum” to improve performance.

    Google Cloud Platform (GCP)

    1. Use GoogleCloudSnitch for Cassandra deployments in GCP.
    1. GoogleCloudSnitch treats regions as Cassandra data centers and availability zones as racks.

    Other Features

    Both AWS and GCP offer a series of operational capabilities that seem to make it easier to maintain high uptime on Apache Cassandra cluster. These capabilities have to be chosen carefully depending your deployment needs.

    1. Load Balancing: Network-level load balancing service is offered by both AWS (Elastic Load Balancer) and GCP. It is not recommended to put a load balancer in front of Cassandra. It is better to use a client connector that can handle this, consider open source utilities such Presto and Astyanax
    1. Autoscaling: Both AWS and GCP offer Autoscaling facility for compute instances. However, although this functionality looks very enticing, putting all C* nodes in an autoscaling group will not produce the desired result. Instead, create a separate autoscaling group of ‘1 node’ for every node in your cluster. This will ensure that anytime a node goes down, it is automatically brought back up. This will ensure the cluster data placement is not disturbed when nodes go down and nodes coming up do not cause excessive data movement.

    Scalable Backups and Reliable Point-in-Time Recovery

    Backing up your data is a must-have for enterprise organizations. The impact of data loss is so high that no enterprise wants to risk relying alone on replication. Protecting data in a public cloud environment may seem daunting, but there are several options available that can make this easier compared to an on-premise deployment. A simple option may look like one of the native snapshots:

    Storage Snapshots

    I have seen few customers use snapshots of EBS volumes as a way to protect the data in their cluster against logical and/or operational errors. This solution will work but with some severe limitations. On top of the overhead of maintaining the scripts to take snapshots, there is a large overhead in storage costs. Taking a snapshot of volumes does not allow for incremental backups of your data. This causes an exponential bloat in the amount of data stored in snapshots, leading to high cost of the backup solution.

    Amazon S3 or Google Cloud Storage

    Using an S3 bucket as the secondary storage enables enterprises to use low-cost, highthroughput storage, while reducing overall operating costs for storing backup data. Here are a few recommendations when using S3:

    1. S3 Region: Amazon S3 buckets are region-specific. A bucket created in one region will provide better performance characteristics to other infrastructure in the same region. Subsequently, it is important to ensure the S3 bucket used as the secondary storage is in the same region as the production database being protected and the Datos IO cluster.
    1. Storage Class and Lifecycle Policy: Amazon S3 allows users to create lifecycle policies on buckets. These policies dictate the performance and response time characteristics of objects in the bucket. STANDARD_1A is more suitable for longer lived and less frequently access data, and is not suitable as secondary storage. Additionally, do not turn on any lifecycle policies on the buckets used for the secondary storage. All buckets should be in the standard storage class.

    Conclusion

    As enterprises leverage the agility provided by public cloud environments, it’s important to understand the various facets of Cassandra deployments to ensure the efficient use of your money and time.