devopsy.kinsta.cloud

Author: contributor

  • Swarms: Adapting Guilds To Scale Agility

    Swarms: Adapting Guilds To Scale Agility

    In my first article in this series, “Using Guilds to Combat Information Silos,” I reviewed the adoption of guilds at xMatters. As our company has grown, we’ve used guilds to keep critical information from languishing in silos—as often happens when an organization scales. Guilds have helped our engineers apply their creativity and critical problem-solving skills to other channels and contribute to areas outside their normal scope. It’s a win-win situation for everyone who participates—and especially for the company.

    Implementing guilds began as a grand experiment. We quickly learned a lot about what enhances or hinders a guild’s success, making adjustments and adding a necessary dash of protocol along the way. But, as is the case with any growing organization, new needs and pain points challenged us to adapt and tackle them. One attempt to solve a critical problem led to an entirely new team model: a body we call the Swarm.

    The Swarm arose as a result of key changes in our release protocol. We used to be very focused on getting new features out the door as fast as we could. As it turned out, many of our customers prefer a more predictable schedule. So we gave our customers  flexible options by maintaining continuous delivery of code to production, with flags to stage larger features or more visible changes. Now, customers who wanted more predictable updates could opt for a quarterly release model while others could continue to access features as they became available and were being iterated on.

    While the quarterly release model improved our rhythm for customers, it also created stricter deadlines. A project that just missed a quarterly release could get delayed by almost another full quarter before our customers could start using it. Throwing more engineers at a problem doesn’t necessarily solve it faster—but, being the engineers we are, we thought about ways to parallelize the problem.

    We knew that guilds were very good at collaborating around a shared problem, so we adopted temporary guilds, which “swarmed” around a shared problem, completed work at an accelerated pace and disbanded. Centered around a specific initiative with a definitive start and end, multiple teams from Dev to Ops collaborated to complete a defined project. For instance, we might form swarms to create new product features, and their lifetime would span a finite number of sprints across multiple teams until completion. Due to its nature of focusing on new product features, swarms differ fundamentally from Guilds in that they are typically not self-forming; each one is an entity intentionally designed for a specific purpose.

    We began our swarms experiment with a large architectural change to roll out a new format for our customer hostnames, which involved six Engineering and Ops teams. We realized immediately that this would be an incredibly challenging undertaking on our tight deadline. A slew of teams was working on different stacks in different programming languages, so there were many competing visions and different contextual understandings of the end goal. It became very apparent that the swarm could not be left to operate in a completely flat hierarchy, as a guild usually does. We needed a leader to help ensure there was no duplication of work, things weren’t falling through the cracks and necessary dependencies were completed in the scrum storywriting between teams.

    Having someone take care of all the logistical problems is one thing, but we realized that in-person communication between all the teams in the swarm is absolutely necessary. We tried staying in the loop with each other by defining a process of posting updates from each team in our swarm’s Slack channel, but this was not enough to keep the project on track. Mattius, one of our very talented operations leads, initiated regular in-person meetings across all the teams, which led to far greater clarity. It was absolutely crucial to have all of the teams together and face to face on a regular basis.

    In the end, swarming on this massive endeavor allowed us to get the project delivered a week before the deadline. It was a challenging and imperfect first run, but it absolutely paid off.

    I leave you with these three recommendations:

    1. Create each swarm with a clear purpose and deadline;
    2. Have a leader to enforce consistent communication; and
    3. Have regular face-to-face interactions.

    The lessons we learned were invaluable for refining (and continuing to refine) our approach to tackling projects for faster release, and we’re continually working to improve and enhance this model. Good luck with all your future swarms.

    About the Author / Nick Fletcher

    Nick Fletcher, Engineering Manager at xMatters, started designing and developing websites back in 1998, when the World Wide Web was the Next Big Thing. In the time since that first website, he’s mastered all the aspects involved in creating great products: design, development and strong leadership. Connect with him on LinkedIn.

  • Elasticsearch Ransomware Attacks Highlight Need for Better Security

    Elasticsearch Ransomware Attacks Highlight Need for Better Security

    Recently, reports surfaced that a large number of Elasticsearch servers fell victim to potential ransomware attacks. Ransomware is the type of malware a company doesn’t want on its systems or network. It takes systems hostage, most commonly by encrypting or stealing data, and exposes the owners to blackmail attempts. According to a report by the Herjavec Group, the cost of damages from ransomware was projected to reach $1 billion by the end of 2016.

    A new wave of ransom attacks observed over the last several weeks targets unsecured MongoDB databases. Security researchers Victor Gevers and Niall Merrigan call these attacks a “ransack,” and Merrigan estimates that more than 40,000 databases were impacted in the first two weeks alone.

    Now research shows that Elasticsearch servers, which are configured to be insecure so they can be accessed over the public internet, are being subjected to similar ransom attacks. Victor Gevers tweeted that within the first three days, 2,515 Elasticsearch servers were eradicated and ransomed and 34,298 vulnerable Elasticsearch instances are still open. In the following days, the number of affected servers has risen to more than 5,000. John Matherly, founder of Shodan, tweeted that the vast majority of vulnerable Elasticsearch servers are open on Amazon Web Services (AWS).

    If an Elasticsearch server is hacked, users will find data indices gone and a message that reads: “SEND 0.2 BTC TO THIS WALLET: 1DAsGY4Kt1a4LCTPMH5vm5PqX32eZmot4r IF YOU WANT RECOVER YOUR DATABASE! SEND TO THIS EMAIL YOUR SERVER IP AFTER SENDING THE BITCOINS…”

    The FBI stresses that victims should refuse to pay Bitcoin ransoms, so users might or might not get their data back depending on the security processes they had in place in case of an attack. At this point, it is unclear who is behind the attacks.

    Ironically, what makes these attacks possible is not that Elasticsearch in itself is insecure, because it isn’t. Ransom attacks are possible because these instances have been configured in a way that makes them vulnerable. It’s like leaving the front door open.

    Technology journalist Steven Vaughan-Nichols of ZDNet gave an excellent summary, explaining that, when used by amateurs without any security skills, Elasticsearch is simple to crack. The people deploying instances on AWS clouds are under the impression that AWS is protecting them, but that’s not the case. While AWS tells users how to protect their AWS Elasticsearch instances, users still need to do the work themselves.

    He notes: “The worst thing about this? Just like the MongoDB attacks, none of this would have happened if its programmers had protected its instances with basic, well-known security measures.”

    Elasticsearch is often used in log management, typically as part of the Elastic Stack or ELK, which stands for its main open-source ingredients of Elasticsearch, Lucene and Kibana. Since it’s free, open-source software, ELK is an easy first choice for many. It’s a great, powerful piece of software. These open-source projects are highly active, with thousands of code contributions every month and a growing combined code base of about 2.5 million lines of code.

    Users need expertise to deploy and run it efficiently and safely, though, and that means needing people in an organization with the skills and time to maintain ELK clusters. If these people leave, companies need to have a backup. If they aren’t willing or able to invest in these resources, they are likely to get into trouble, like this latest ransom attack situation illustrates.

    ELK is free software, but keep in mind what Richard Stallman, founder of the Free Software Foundation and the GNU Project, has to say about free: “’Free software’ is a matter of liberty, not price. To understand the concept, you should think of ‘free’ as in ‘free speech,’ not as in ‘free beer.’”

    Whether a company pays for a log management service or runs ELK, one approach isn’t necessarily better than the other. For some companies, it makes a lot of sense to run an in-house-built log management solution based on ELK, or even one built from scratch. For others, a delivered solution may be best.

    Regardless, companies and teams need to carefully evaluate if open source makes business sense and if they are realistically able to properly support the deployment without imposing risks. If these factors aren’t weighed or completed correctly, the number of Elasticsearch ransomware attacks will continue to grow and be a profitable endeavor for hackers.

    About the Author / Sven Dummer

    Sven Dummer is the senior director of product marketing at Loggly. Previously, Sven worked with Yahoo, Wind River (acquired by Intel), SUSE, and Microsoft in product development and management. At Intel, Sven also helped launch (and named) the collaborative Yocto Project, an open-source initiative that enables users to create custom Linux-based systems for embedded products regardless of the hardware architecture. Connect with him on LinkedIn and Twitter.

  • Strategies to Build Trust in DevOps

    Strategies to Build Trust in DevOps

    I’ve been a member of the DevOps community a long time (literally since before it was ever even called “DevOps”). I was a pre-publication reviewer for both “The Phoenix Project” and the recently published “The DevOps Handbook,” and I’m always interested in seeing the outcomes successes software teams have and the failures they overcome, including trust.

    While DevOps has proven to be effective in organizations of all sizes—startups and enterprises alike—there are still many who have not yet begun their DevOps journey, so I thought I might share some of my own experiences while introducing DevOps into an enterprise IT group.

    When I was the VP of Operations at Big Fish Games, I had the responsibility for production operations at Big Fish’s three production data centers in Seattle, Virginia and Luxembourg. I also had responsibility for internal corporate IT and information security. Our production operations supported around a million game downloads per day. Games were localized into 11 different languages. We processed essentially every major currency, and right before I left Big Fish, we started taking Bitcoin as well. From an IT perspective, Big Fish Games was a global eCommerce company selling digital widgets on the internet. We had around 800 employees in our five offices in Seattle, Ireland, Oakland, Luxembourg and Vancouver, BC.

    My first attempt to introduce DevOps at Big Fish failed miserably due to cultural misunderstandings and a fundamental lack of trust between the development and operations teams.

    You absolutely need some minimum baseline level of trust in your organization if you are to have any chance at implementing DevOps. Trust is important because you are blurring traditional boundaries between development and operations responsibilities. Including development in production operations can be scary for operations teams. It just feels wrong from a historical ITIL-type perspective, and operations teams may know development as “the people who always break things.” At the same time, it can be scary for development to involve operations in the development process because they may know operations as “the team that always says ‘No.’”

    Building Trust

    Successfully merging the priorities and goals of development and operations teams to create one cohesive DevOps effort can fail due to cultural misunderstandings and, oftentimes, a fundamental lack of trust. I used three specific strategies to overcome these challenges.

    Lose the Term ‘DevOps’

    At Big Fish Games, application development teams equated DevOps with developers carrying pagers: an idea that they really did not like. They were highly resistant to this. Every time I brought up DevOps, they thought I was engaged in a secret plot to make the developers carry pagers. What I found interesting about this was that I had no intention of asking developers to carry pagers.

    The term “DevOps” covers a large range of continuous improvement techniques. Some organizations have developers carry pagers, and it works for them, but this is certainly not a prerequisite for DevOps. What was worse, I had no idea they were thinking this because they were afraid to bring it up for discussion. We failed in our first attempts to implement DevOps because the trust level between development and operations teams was so low that we couldn’t even tell each other what we were thinking.

    I realized two things from this experience:

    • Getting rid of a term that scared both development and operations teams would eliminate an unnecessary distraction preventing these teams from coming together to focus on common goals.
    • We needed to do something about the trust issues between development and operations before we could succeed with DevOps.

    Once I realized this, I immediately stopped using the word “DevOps.” Instead, I talked about continuous improvement and specific continuous improvement methodologies that we wanted to implement, and how these new adoptions would result in shared, organizational-wide growth.

    Focus on Methods That Benefit Other Teams

    When deciding which of these DevOps, a.k.a. “continuous improvement,” techniques we wanted to implement, I specifically chose ones that would benefit development teams as opposed to ones benefiting operations or other teams. The development teams noticed this over time. They could see I was doing this and that I (the person who ran operations) was going out of my way to help them. This helped build trust. This wasn’t a one-time thing. We did this consistently.

    Here’s another simple failure example: I noticed while managing release efforts that release schedules for the various development teams ranged from a couple weeks to a couple months—too long for a hyper-competitive industry like gaming. I asked the development teams if they would like our release engineering team to support daily code releases or even multiple code releases per day. I thought this was an innocuous question. In fact, I thought I was being a little clever by playing dumb here. I assumed I would get an enthusiastic high-five along with a, “Yes, we’d love that! How soon can you make that happen?!”

    I was surprised when I received a violent “No! We don’t want that!” This confused me for a while until I realized that the development team thought I was in cahoots with the product management team to put the squeeze on them. In my mind, I was offering to make their release process more lightweight so they could break up their work into smaller, more frequent releases with less stress for everyone, but I didn’t understand their fears and problems. I wasn’t looking at it from their perspective and here was the trust issue rearing up again. They misinterpreted my intentions as nefarious.

    I pictured them being able to release code whenever they want, which seemed like something they would like. In their minds, they pictured a product manager constantly randomizing them with demands for new features to be released right there while the product manager was leaning over their shoulder. This was not the kind of problem you could resolve with logic. The solutions and working procedures I was proposing were so far outside the development team’s experience that they couldn’t extrapolate through simple logic from their current state to the world I was describing.

    Maximize Daily Interactions Between Development and Operations Teams

    It’s easy to distrust someone you don’t know personally or you don’t have to see every day. It’s a lot more difficult to hate on someone you know and work with on a daily basis.

    One of the DevOps techniques that I implemented early on was embedding a dedicated operations liaison into each development team. This liaison acted as an advocate for that development team back into operations and they helped smaller development teams navigate their work through a large, complex, centralized operations organization. The operations liaison attended daily development team standups, group meetings and project meetings. This program was a huge success. Development teams saw how their operations liaison helped expedite their work through the operations organization and advocated for their development team’s interests inside operations.

    After the development teams got used to working with their operations liaison on a daily basis, we pushed a bit farther and got the development teams to agree to include operations in architectural discussions at the beginning of application design. At first, our development teams were skeptical that operations could provide useful input here, but the important thing is we had developed enough mutual trust that they were not fearful of including operations in the discussions. They were skeptical that we would add any value, but they weren’t afraid that we would cause them problems. This was a big win.

    Although it felt a little weird to our development teams to include operations in design discussions, they quickly saw how obtaining input from operations early on, before a product was built, resulted in better quality and fewer headaches for everyone when we launched into production. I also made sure that we put our most senior and diplomatic operations team members into this role.

    In conclusion, it is worth repeating that trust is a prerequisite for any organization to succeed with DevOps. This is due not only to problems arising from fuzzy definitions of what DevOps is and is not, but also the seemingly dangerous idea of blurring the boundaries between development and operations organizations. Whatever your definition of DevOps is, fostering an increased level of trust between development and operations teams as a prerequisite for your DevOps journey will ensure that even greater trust—and the positive business impact that comes with it—are your outcomes as well.

    Good luck!

    About the Author / Paul Farrall

    Paul Farrall is the VP of Operations for Skytap. He was most recently vice president of Operations at Big Fish Games where he led the rollout of global infrastructure for the Big Fish cloud gaming platform and implementation of the Big Fish DevOps program. Prior to Big Fish, Paul was the Director of Operations and Chief Information Security Officer at Intelius, a Bellevue e-commerce company. Previous companies also include AT&T Wireless in Bothell, Washington, and GetThere.com in Palo Alto, California (now Sabre). Connect with him on LinkedIn and Twitter.

     

  • Increasing Your ‘Bionic Footprint’ with DevOps

    Increasing Your ‘Bionic Footprint’ with DevOps

    “The Six Million Dollar Man” was a 1970s TV series. In the show, NASA astronaut Colonel Steve Austin is severely injured in a crash and is “rebuilt” with bionic features that enhance his strength, speed and vision beyond normal human capabilities. The main tagline was, “We can rebuild him, we have the technology.”

    I recently was watching an old episode and it got me thinking about how the show is analogous to the genesis of DevOps—specifically, how individuals with an operations background have to “rebuild themselves” to keep up with the speed and direction of the technology industry.

    Over the years, there have been several decrees of “Unix SysAdmins are going away.” I have always firmly disagreed with that assertion. Rather, I believe that SysAdmins have to continually evolve their engineering skills to stay relevant and, more importantly, gainfully employed. It was also becoming increasingly apparent that the manual tasks of deploying infrastructure, keeping it patched, monitoring and the myriad other things a SysAdmin performs every day could not keep up with the demands of the increasing pace of the digital world and the onset of cloud computing. Infrastructure is moving rapidly to being automated and programmatically deployed and the talent needs to adjust accordingly to stay relevant.

    With many enterprises adopting the agile methodology and moving toward a continuous integration/continuous delivery (CI/CD) pipeline, the culture of DevOps became a reality. Due to the increased velocity, there became a clear need for the core tenets of DevOps culture, which include culture, automation, measurement and sharing (CAMS). In Six Million Dollar Man terms, this was simply “Better … Stronger … Faster.”

    Embracing DevOps through CAMS is similar to the superhuman enhancements that composed the rebuild of Steve Austin, which took him from man to bionic man. Here are the four ways that DevOps can increase your enterprise’s “bionic footprint”:

    Improve Culture

    Even Austin’s bionic features only got him so far; the bionic man had to rely on his team to complete his dangerous missions unscathed. The same is true in the enterprise: Building a collaborative culture is crucial for success. Culture is paramount in maintaining overall velocity without adversely affecting quality. Developers and operations teams need to function in concert to ensure a resilient application and infrastructure.

    Increase Velocity via Automation

    Automation greatly increases velocity, akin to how Austin’s bionic legs allowed him to run at high speeds and make great leaps. Automating processes saves time, money and lets your team focus on more complex tasks—a win-win for productivity gains.

    Measurement

    Steve Austin had a bionic left eye that enabled him to see in the dark as well as detect heat. This provided him precise measurements of his surroundings in any situation, enabling him to gain the upper hand against adversaries. Similarly, measurement gives insight into all aspects of the software development life cycle, application performance and other key metrics. Without insight and transparency into measurement, your team is essentially working blind.

    Sharing

    Austin had a bionic right arm that had the strength equivalent of a bulldozer. Strength and resiliency are increased by teams that work and communicate effectively made possible by DevOps.

    To summarize, I strongly believe that the mantra of “Better … Stronger … Faster” should be that of every DevOps team moving forward. DevOps’ parallel of CAMS upgrades your team’s capabilities to perform more productively. There is no end game; the objective is about continuous improvement, velocity and sustainable growth. Enterprises that want to undergo digital transformation need to adopt this cultural approach to increase their “bionic footprint” to succeed.

    About the Author / Mike D. Kail

    Mike D. Kail is chief innovation officer at Cybric. Prior to CYBRIC, Mike was Yahoo’s chief information officer and senior vice president of Infrastructure, where he led the IT and global data center functions for the company. Prior to joining Yahoo, Mike served as vice president of IT operations at Netflix, where he was responsible for employee technology and various cloud engineering components. He has been widely recognized for his insightful industry commentary on social media Twitter, and was recently named by the Huffington Post as one of the “Top 100 Most Social CIOs on Twitter.” Connect with him on LinkedIn and Twitter.

  • Defining DevOps: Not Theory, nor Implementation

    Defining DevOps: Not Theory, nor Implementation

    Ask a dozen engineers, developers, testers or their managers what their definition of DevOps is and you will get likely very different answers from each of them. Some may say it’s primarily about culture, others may say it’s about accelerating or automating the release of software and infrastructure changes and some may use the mantra, “People, over process over tools.”

    What no ones seems to be able to agree on though is how to define the boundaries and constituent parts of what DevOps is really all about. It is all-encompassing, it seems to cover topics as varied as continuous improvement, collaboration, a whole raft of processes from agile, through testing to operations procedures, audit, compliance and customer feedback to collaboration. It seems to be attempting to cover every facet of life in this small part of development and IT operations that we live in—and because it’s so broad no one can really agree on what it means.

    My suggestion is that we should try to narrow the definition of what DevOps is in an attempt to create some boundaries so that it can be considered more of a discipline than a utopian goal. For this reason, let’s consider some of the core concepts—culture, communication, collaboration, continuous improvement—as things that are important to its implementation, but not the thing itself. These are aspects of high-performing teams and should use a different lexicon outside of the DevOps world.

    So culture should not be considered part of DevOps?

    Put simply, any initiative that fundamentally reshapes an industry through a fundamental change in its objectives, process and delivery mechanism implicitly needs the support, buy-in and leadership of those involved. Much of the cultural aspects of DevOps seems to be around these areas—but if you are attempting to radically change the way any industry works, then these are common goals that use management and leadership theory that are not restricted to change within a narrow segment of software development and IT operations. These leadership challenges exist not only within the teams responsible for the delivery of software and IT operations—but in many industries that undergo disruptive change.

    Should culture be a primary factor in implementing a DevOps initiative?

    Now, this may sounds like a contradiction to what’s been said above but the answer is, of course, yes. However, the culture of any organization should be at its core. The culture within a law firm is important, as it is in a research lab or within a retail organization. If we take retail as an example, when a retailer implements just-in-time (JIT) delivery, which is analogous to what certain aspects of DevOps is trying to achieve—the faster delivery of goods, automation, small batch, feedback loops, metrics, audit, collaboration of disperse part of the supply chain—how much priority is based on whether the employee’s need to be bought into the idea before the initiatives were started? I would suggest not much; it was a management decision, based on shareholder value and the competitive advantage it would provide over competitors. Now that DevOps has become much more mainstream, I think we should think about it in a similar fashion. Retailers that have not implemented JIT are either out of business or very niche. We should no longer be thinking we need to get staff buy-in; the world has moved on, it’s happening, it’s mainstream and if your not on board, then get a job elsewhere. If you work in a shop and put some goods in the store for sales that have not been delivered by JIT, then you get fired. Simple as that.

    Is DevOps really the same as JIT Delivery?

    No, of course it’s not. JIT was a top-down, strategic management decision to implement on its workforce. It did not come from the grass roots, it did not come from a philosophy and belief from those at the coal face that there is a better way to deliver goods to customers, it was not a revolution by checkout staff that demanded that no longer should their managers buy from suppliers on a golf course and return to the store and tell them to make it work. The DevOps culture, by which I mean the culture within the DevOps community, is so inspirational because they shunned the top-down approach to implement bloatware and chose to go off piste—collaborate with their piers and change the face of an industry in a revolutionary way. They demanded that they know best.

    Too many Brents

    For all the talk of culture however, this industry we work in has far too many Brents (as in the indispensable engineer in the Phoenix Project, who is the only person who can fix the problem, but is also the root cause of all the issues). I am often baffled at the irony of the no hero’s mantra that is all too often exposed in the DevOps world, but many of the most talented engineers I believe do crave a hero status. They derive a lot of their self worth from the work they produce and consider themselves a snowflake, even if their servers aren’t. They write complex scripting frameworks in Ruby or Python and are attracted toward complex and powerful tools that they enjoy using, engineering themselves into a position where they derive brent-like satisfaction and create some job security. I’m not necessarily being to critical of such an approach, but I do find it guiling that at the same time believe they are part of a cultural revolution.

    But do the engineers know best?

    I think it’s hard to argue that, certainly if compared to 10 years ago, that the broader influence of DevOps has not only made software delivery and IT operations a much more exciting and dynamic environment to be working in—but is also delivering huge business value and competitive advantage for those organizations that have embraced it. Was culture and philosophy a key driver to achieving this? Absolutely. But much as Ben Horowitz espouses in the “Hard Thing About Hard Things,” you need a team to take a business to a certain level, and once it has reached a certain level of success it needs to be taken forward by specialists, not generalists who brought it to that point. I believe we have reached that point.

    DevOps is now becoming mainstream and may benefit from being subdivided into different areas of specialization. As a startup, there is fun, excitement and a belief that you can change the world. Individuals that thrive on that often enjoy the variety in the work, performing a variety of functions within the organization—maybe some software development, support, pre-sales and marketing. But as an organization grows, specialization is required, there is a marketing team, support team, pre-sales and engineering team. This is a reality of working in a growth organization. There is no longer one team that does everything, but they are split out into specialization, collectively bringing the whole organization to a stronger place. DevOps has grown up—and needs to demarcate specializations to take it to the next level.

    So how should the boundaries of DevOps be defined?

    I would propose one of two things. DevOps remains the overarching philosophy of how we make the world a better place, underneath it sitting some more defined disciplines which are less related to the philosophy. So we don’t have the concept of a DevOps engineer, but instead they are roles such as automation specialist. Or perhaps, if I’m honest the way I felt when I started writing this blog is that we (and I consider myself part of the DevOps engineering community), now it has become much more mainstream, allow it to be taken forward by specialists disciplines, of which engineering is one, but the cultural, philosophical and management practices are no longer in the startup realm but need to be taken forward by the “grown ups.”

    The philosophy and implementation should be separated

    Ultimately, I think there needs to be some separation from philosophy of what DevOps is and the constituents parts of it. My primary reason for writing this blog, if I’m honest, was I am becoming bored of the DevOps bandwagon. The whole DevOps concept is is meant to conjure exciting and positive ideas how we can make the world a better place … but spring turns to summer, turns to fall, and successful startups turn from dreamers with passion to having VC backers or being owned by private equity.

    This is not meant to be a negative point though—anyone who creates a start up ultimately wants it to go on to become a grown up. And Patrick Debois may be the founder of the startup, but DevOps is now a publicly listed company.

    About the Author / Dave Sayers

    Dave Sayers is a co-founder and technical services director of MidVision, a provider of Release Automation and Continuous Delivery Software. He works closely with customers and partners on DevOps and Automation initiaitves throughout Europe and North America and is responsible for devising and managing the partner and channel strategy. Connect with him on LinkedIn.

  • Automation, ChatBots Can Improve DevOps

    Automation, ChatBots Can Improve DevOps

    At its heart, the goal of DevOps is to foster better communication between the development team and operations team at an organization. When these two teams are systematically connected, they can easily stay in sync on most issues and there is strong coordination in building and delivering the product to customers.

    Automation and ChatBots can play a big role in the connecting development and operations. They open the door for companies to go beyond simply being alerted when something happens in a chat application, allowing an employee to take action after receiving the alert either in an automated manner or by providing guidance to the employee on which actions to take.

    Automation: Creating Smart Alerts That Don’t Require Follow-Up

    Monitoring applications, systems, virtual machines and networks are a key part of any operations team. There are a number of tools that are used for this purpose, including CloudWatch, Splunk, Rollbar, Pingdom and Newrelic. These tools typically alert the operations teams when there are issues. Engineering teams, on the other hand, use tools such as JIRA and Github to track tickets and manage their agile development process. Alerts from the tools used by operations are helpful, but they still require some manual follow-up. For example, imagine a critical alert is reported by one of the tools because of a product issue. Follow up on this alert would require:

    • Creating a ticket in JIRA/Github with all the relevant information to help diagnose the issue.
    • Escalating that ticket so that it gets looked into right away.
    • Keeping the required people in sync as the status of the issue changes.

    This entire process can be automated. When automated, the workflow operates like this:

    • When the monitoring tool senses there’s something abnormal, like error rates above certain thresholds or a critical failure, alerts are sent and a ticket is created in JIRA or Github automatically with all the relevant information about the issue.
    • Taking this a step further, based on the type of issue, it can be automatically assigned to the right person and even post a message on the appropriate Slack channel.
    • As engineering works through the issue, everyone is automatically informed in Slack when the ticket is updated.

    The key to getting this scenario automated is to leverage an integration and automation platform that is powerful enough to handle complex workflows, yet easy to use so that these automations can be built in minutes. Users don’t want to work for your automations, they want their automations to work for them. With that in mind, the platform should also be flexible so changes can be made to the workflows without any engineering effort (i.e., an automation platform where integrations and automations can be built and changed without any coding).

    ChatOps: Monitor and Orchestrate via Chat Platforms

    With all the apps being used by DevOps, it is difficult for any team to stay on top of monitoring and managing issues while also taking actions across all these tools. Studies show that app hopping is not efficient and results in constant context switching, which negatively impacts productivity.

    ChatOps allows users to stay on top of everything including orchestrating workflows right from chat platforms such as Slack. To make ChatOps work, users need a bot that works in tandem with an integration and automation platform. The bot works with all business apps (even the tough ones with custom objects such as Github and Salesforce), is customized to meet the specific needs of the business and can extend to address all operational needs.

    Here are three ways users can leverage bots for better DevOps:

    Smart, actionable alerts: Rather than have a firehose of alerts coming in, users can define various thresholds, monitor velocity of alerts and, based on these settings, post an alert to the relevant channel or person on Slack. Bots can also prompt the person to take action based on the notification.

    Stop services, virtual machines: Bots can connect with any app, or service that has a HTTP/REST API. Not only can they monitor various apps and environments, they also can start and stop them.

    Workflow orchestration: Various workflows such as managing incidents, escalations and deploying code are all triggered and managed through Slack.

    Automating DevOps has significant benefits in overall increased engineering and operations productivity. The key to automation success is to select an integration and automation platform that is secure, resilient, can handle complex workflows, can be easily customized to meet the specific needs of the business and can be managed through Slack. It should be easy to use so automations can be built fast and changed easily as workflows change. Automation and ChatOps will streamline business workflow, keep the important players from both development and operations informed and increase productivity for everyone involved.

    About the Author / Gautham Viswanathan

    Gautham Viswanathan is the founder/head of products at Workato, www.workato.com. He also leads the team in creating Workbot For Slack, a bot designed to be customizable and allow you to do work in your apps right from Slack. Gautham has a deep background in integration platforms and was previously the head of products and engineering at TIBCO. Connect with him on LinkedIn.

     

  • Puppet Launches 2017 State of DevOps Survey

    Puppet Launches 2017 State of DevOps Survey

    It’s that time of year again!

    We at Puppet have kicked off the 2017 annual State of DevOps. survey. Now in its sixth year, we have once again teamed up with DevOps Research and Assessment (DORA), founded by our dear friends Gene Kim, Jez Humble and Nicole Forsgren, to survey technology professionals worldwide in an effort to better understand the health and habits of organizations adopting DevOps. We’re pleased to see that the report now serves as the de facto gauge for the DevOps community to find out how IT organizations are doing and how they can improve.

    One of the most critical aspects of the survey is that it manages to translate the processes between technical practices and choices into actual business outcomes and helps organizations understand the real benefits of DevOps adoption. Through analysis of each year’s survey responses, we’ve been able to quantify IT performance. From last year’s report, we know that high-performing IT teams are pulling away from the pack in terms of throughput. They deploy 200 times more frequently, have 2,555 times faster lead times, have a three times lower change failure rate and recover from failures 24 times faster.

    Survey Says …

    This year, we’re interested in learning more about infrastructure architecture patterns within organizations. We’re not just asking how the services are designed, but we also are curious to see if teams are free to move at their own pace and if they are largely self-sufficient. Can they easily make large-scale changes to their architectural design?

    We’re asking questions about team experimentation and whether teams are able to evolve the design of their architecture as they work and if they are required to interact with a lot of people across the organization. One of the things to emerge from the agile and DevOps movements is that the more we can enable teams to work quickly and self-sufficiently, and increase speed without reducing quality, the better it is for everyone.

    This year, we expect to see proof that high-performing IT teams are those who have been enabled to operate with a much higher degree of autonomy, and that their quality metrics remain stable or even exceed those of high-performing IT organizations. We also expect the findings to confirm that early DevOps adopters are still seeing exponential benefits compared to their colleagues who haven’t yet embraced DevOps. The longer your organization has been adopting DevOps practices, the more it becomes muscle memory and reinforces positive behavior within them, which has unparalleled benefits. As any IT professional working in a large company knows, there’s an awful lot of waste and inefficiency and we’re not anywhere near peak DevOps penetration. With each passing year, survey results show DevOps adoption is a key contributor to enhanced productivity and efficiency. We expect this year’s results to be no different.

    We’ll be publishing the results and our analysis in June, and have no doubt it’ll be worth the wait. Stay tuned!

    About the Author / Nigel Kersten

    Nigel Kersten is CIO of Puppet. He came to Puppet from Google, where he was responsible for the design and implementation of one of the largest Puppet deployments in the world. At Puppet, Nigel was responsible for development of the initial versions of Puppet Enterprise, and has since served in a variety of roles, including head of product, CTO and CIO. He has been deeply involved in Puppet’s DevOps initiatives, and regularly speaks around the world about adoption of DevOps in the enterprise and IT organizational transformation.

  • 3 Approaches to End-User Experience Monitoring

    3 Approaches to End-User Experience Monitoring

    The volume of transactions running through websites and mobile apps make customer-facing applications crucial to online businesses. If these applications perform well for their users, they generate revenue for the business. If they don’t, they affect the credibility of the business, which in turn affects the overall revenue. It is therefore imperative that businesses using monitoring to understand how well their revenue-critical applications are behaving for their end users.

    From an IT team’s point of view, understanding the user experience of their applications is becoming more challenging as technology evolves. Newer and more complex applications are being written using an assortment of languages. These applications are being deployed on a wide variety of infrastructure components. To add to that, today’s users access these modern applications on a variety of devices such as the web, smartphones, tablets and smart watches.

    Fortunately, there are a few means available through which businesses can determine the user experience of their web applications. Let’s take a look at three common approaches below.

    Real User Monitoring (RUM)

    Real user monitoring is a passive monitoring approach that involves collecting metrics at the browser level to accurately determine the application performance as perceived by the end users. Monitoring at the browser level is achieved by injecting JavaScript snippets into the header and footer of the HTML code of the web application. This code will ascertain the full page-load experience—including downloading the assets from the content delivery network (CDN), rendering the page and executing the JavaScript from the browser’s perspective. Additional instrumentation can be used to collect more metrics by injecting additional JavaScript code.

    The data gathered through RUM provides answers to questions about user experience such as:

    • How long did it take to load the full page?
    • What is the response time from a network perspective (redirection time, DNS resolution time, connection time)?
    • What is the time interval between sending the request and receiving the first byte of response?
    • What is the time taken by the browser to receive the response and render the page?
    • Are there any problems on the page? If yes, what caused the problem?
    • How is the performance when the application is accessed from different countries?
    • What is the response time across different browsers? Do new application updates affect the performance in a specific version of the browser?
    • How does the application perform on different platforms such as desktop, web and mobile?

    The biggest advantage of monitoring real user data is that it relies on actual traffic to take measurements. There is no need to script the important use cases, which can save a lot of time and resources.

    Real user monitoring captures everything as a user goes through the application, so performance data will be available irrespective of what pages the user sees. This is particularly useful for complex apps in which the functionality or content is dynamic.

    Server-Side Monitoring

    Although user experience is best tracked at the browser level, application performance monitoring at the server side also provides insight into end-user performance. Server-side monitoring is mostly used in conjunction with real user monitoring. This is because problems originating on the server side can only be efficiently detected using server-side monitoring.

    Monitoring performance on the server side involves agent-based instrumentation technology for acquiring and transmitting data. This monitoring approach is used to watch user transactions in real time and troubleshoot issues such as slowness or application bugs.

    Developers have to install agents on the application server to help capture and visualize transactions end to end, with performance statistics across all components, from the URL down to the SQL level. This visual breakdown reveals the flow of all the user transactions being executed in each layer of the application infrastructure.

    Server-side monitoring helps track response time and throughput taken by each application component, with the option to trace transactions end to end via code analysis. This helps the IT operations/DevOps teams identify slow web transactions and then isolate performance issues down to the level of the specific application code that caused them. The underlying database is also monitored most of the time to determine slow database calls, database usage and overall database performance. With server-side monitoring, users will be able to identify the SQL queries executed during a transaction and thus identify the worst-performing queries.

    Synthetic Transaction Monitoring

    Synthetic transaction monitoring is an active monitoring technique based on the concept of simulating the actions of an end user on a web application. This method involves the use of external agents executing prerecorded scripts that mimic end-user behavior at regular time intervals. The monitoring agents are usually very light and do not create any additional load on network traffic.

    Most application performance monitoring solutions provide recorder tools to capture the actions or paths a typical end user might take in an application, such as log in, view product, search and check out. These recordings are saved as scripts, which are then executed by the agents from different geographical locations.

    Technically, there are two different approaches to generating requests. Some solutions replay recorded HTTP traffic patterns, while others drive real browser instances. The second approach is more useful for modern applications that make a lot of JavaScript, CSS and Ajax calls.

    Since synthetic transaction monitoring involves sending requests across the network, it can measure the response time of application servers and network infrastructure. This type of monitoring does not require actual web traffic, so you can use this approach to test your web applications prior to launch—or anytime you like. Many companies use synthetic monitoring before entering production in the form of automated integration tests with Selenium.

    Synthetic monitoring does have its limitations, though. Since it is based on predefined transactions, it does not monitor the perception of real end users. Transactions have to be “read-only” because they would otherwise set off real purchase processes. This limits the usage to a certain subset of your business-critical transactions.

    The best approach is to use synthetic transaction monitoring as a reference measurement that will help identify performance degradation, detect network problems and notify in case of errors.

    Every business is different and has its own requirements that can help to choose which type of monitoring to implement. An ideal strategy would be to use active and passive monitoring techniques side by side so that no stone is left unturned in the pursuit to monitor end-user experience.

    About the Author / Sridhar Iyengar

    Sridhar Iyengar is vice president, product management, at ManageEngine, a division of Zoho Corp. For more information on ManageEngine, the real-time IT management company, please visit www.manageengine.com; follow the company blog at http://blogs.manageengine.com, on Facebook at http://www.facebook.com/ManageEngine and on Twitter @ManageEngine. Follow Sridhar on Twitter @iSridhar.

     

  • Forecasting Culture: Sunny with Accumulating Collaboration

    Forecasting Culture: Sunny with Accumulating Collaboration

    Bryson Koehler believes the most important enabling factor for DevOps success is culture.

    Koehler is CTO of IBM’s newly formed Watson and Cloud Platform group. Prior to this role, he served as executive vice president, distinguished engineer and CIO/CTO of The Weather Company (TWC), an IBM business since 2016. He has a record of successfully leading significant business transformations and underlying cultural transformations, which he maintains in tech companies must be incited first at the developer level. As such, while at TWC Koehler built a DevOps culture and practice that encourages transparency, accountability and collaboration, and it is his mission to replicate the successful model in the transformation of IBM’s culture.

    Since I experienced some of the TWC culture firsthand, I asked if he would be willing to share his thoughts on enabling a high-performing culture as well as maintaining that culture during periods of rapid growth or change.

    He started the interview with a theme clearly central to his core ideology as a transformational leader, and one of TWC’s key factors for success:

    Culturally, the goal is to find people who really love what they do. I want people to wake up in the morning and love what they do—not tolerate it, not like it, but love it. We work very hard to find people who love what we do and love the mission we are on. If you work very hard to assemble a team that loves what they do, the culture will take care of itself as long as you enable it … it’s about enabling them, with as much freedom and flexibility as possible, while still maintaining some level of consistency and control. It’s a balancing act and at TWC we worked very hard to give the right tools that enable people to work on any device, anywhere, at any time. The tools were an enabler; it was the culture that really invented itself based on the fact that we hired people who love what they do.

    Koehler’s commitment to finding the right people with the skills and passion consistent with his company’s mission put the base focus of enabling DevOps practices where it needs to be—with the people.

    For those that have read some of my other blogs, one of the things I’ve written about before is the mistake of choosing tools without first addressing the critical underlying aspects of DevOps including, a solid agile foundation and the cultural changes required to facilitate DevOps practices. Needless to say, Koehler’s comments about tools being an enabler really resonated with me because it was clear he still views the tooling as very important but puts the real emphasis on having the right people. The other key message that really resonated with me during the interview was his commitment to accountability and transparency:

    … from a leadership perspective, it’s around making sure there is clear accountability. There is single-person accountability for work. If you walk into a meeting and you don’t know who the single decision maker is in the meeting, my recommendation to the team is to leave. Don’t have the meeting, because it’s a waste of time. There should be one person that everyone knows is the decision maker, whatever the topic. If we don’t know that, then it’s my job to make sure I make that crystal clear. You can’t walk into a meeting with five people all thinking they own the decision when also working to create a culture with high velocity, high throughput and high passion towards meeting objectives and getting stuff done. You hire people who love what they do, you enable them with great tools, you have crystal-clear accountability and decision-making so that things can get decided quickly and that we know who is accountable for making sure that decision was successful, and then constantly work against the goals of the business. You also have to be very willing to adjust your tools, very willing to adjust your people, and very willing to adjust your decision-making. We must be constantly willing to look at ourselves and figure out how we improve.

    Koehler’s willingness to continuously look inward and adjust what is not working demonstrates another key success factor for high-performing DevOps teams. Having an executive driving that culture of continuous improvement is critical for DevOps success.

    He also is dedicated to core agile principles that allow for flexibility and responsiveness while maintaining some degree of focus and clear prioritization. With the rapid TWC growth, we also discussed how he maintained and adapted the existing culture in response to that growth:

    It’s really critical for me as a leader to stay very in tune with how these teams work and function, calling out any bad behavior and correcting it quickly. I’m just as willing to roll up my sleeves and get dirty as the next guy, and I expect everyone else to do the same. Kingdom building is not something that will get you very far in how we work. If you do that [roll up your sleeves], then as part of your culture, everyone puts down their guard and realizes that nobody is any more or less important than anyone else. We are all here to accomplish the same thing.

    Common goals and people that love what they do! It’s a great mix and Koehler has some fascinating perspectives on driving cultural transformation in support of DevOps practices, not only for the efforts he successfully led at TWC but for the efforts he will continue to lead within IBM. His commitment to culture can best be wrapped up by one of his final comments:

    None of our great technical ideas has any chance of seeing the light of day if we don’t get the culture right.

    Thank you again to @brysonkoehler for the interview and insight into TWC’s amazing culture and your vision for IBM!

    About the Author / Bryson Koehler and Shelbee Smith-Eigenbrode

    Bryson Koehler is CTO, IBM Watson & IBM Cloud – GM & Distinguished Engineer at IBM Connect with Bryson on LinkedIn and Twitter

     

    Shelbee Smith-Eigenbrode is a Senior Software Engineer/IT Architect at IBM and has been in IT for 19 years.  She has worked across varying roles spanning the software development lifecycle.  Her experience across functional roles as well as performing that work deeply inside traditional silos has contributed to her passion for embracing the DevOps culture and transforming the way teams are delivering technologies by adopting DevOps practices.  She is a member of the IBM Academy of Technology as well as an IBM Master Inventor committed to driving innovation and technical advancement. Connect with me on Twitter @Shelbee_SE.

     

     

  • How Healthy is Your Office 365 Environment?

    How Healthy is Your Office 365 Environment?

    The IBM Domino environment was mostly replaced by Microsoft on-premises technology, then later replaced by the Microsoft Cloud Services. One thing that hasn’t been replaced is the need for administrators to have control and visibility of service delivery to end users.

    Whether you’re using IBM applications, Microsoft on-premises servers or Microsoft Office 365, as the messaging administrator you need to provide availability and performance metrics of the service delivered to your management. You also need to understand issues when they arise.

    In short, you are still responsible for the service that you deliver, even if the servers reside in Microsoft’s data center. So, how do you measure the health of your Office 365 service delivery and how do you anticipate issues before they impact your end users?

    Measuring Office 365 Health

    Microsoft works to maintain the 99.99 percent availability it promises from its data center as well as improve both the features it provides and the performance it delivers. The issues that arise generally impact only some features and usually only affect a subset of users with a specific configuration or in one location. It’s difficult to know exactly how these issues impact your global network, and it’s impossible to predict, but it helps to know that Microsoft offers a health dashboard. The dashboard provides an easy-to-understand view of the availability and performance of each Office 365 service out of the data center, and lets users configure alerts. The disadvantage is that the dashboard only shows service out of the data center, not any visibility into the service your users are experiencing. It also doesn’t provide information on how issues are impacting the availability and performance in your company. Nor does it take into account user complaints that can stem from many issues within the Office 365 environment.

    Microsoft provides two tools that can be useful for measuring the health of the Office 365 services delivered to your users. First, the Remote Connectivity Analyzer allows you to run a series of checks from the cloud to the cloud that will tests different part of the Office 365 services. This allows you to check if your tenant has problems with the basic functions of Office 365 by running Outlook connectivity tests, opening a mailbox, performing a mail routing or connecting to Skype to Business Online. The disadvantage is that it is performed from Microsoft Azure to Microsoft Office 365—from one cloud to another. It doesn’t provide you with any information that could help you troubleshoot what is happening at the user location.

    Microsoft’s other tool that can be useful when it comes to checking the connection is the Support and Recovery Assistant. This tool lets you perform tests to help you understand what’s happening when a location is experiencing poor Office 365 performance. However, these tests aren’t automatic or continuous, and they don’t alert you when an issue begins to arise. You still have to rely on user complaints for that.

    That leaves manual testing as the last option for administrators. Manual tests are available in PowerShell. When conducting tests, start by testing pings to Office 365, DNS resolution time, latency and end-user egress points to understand the impact of the network on your Office 365 service delivery.

    Even if these tools are not enough to measure, alert, report or troubleshoot the service to your end users, they still provide something to check the health of the Office 365 service. Microsoft understands that the service out of the data center is not what matters most; it is the user experience. That is why when people think about their Office 365 health, they should always consider the entire service route, from the Microsoft data center to the user sitting at his desk.

    When we measure health, we understand availability and performance of the service.The border between poor performance and unavailability is small, especially for end users. To understand the health of your services you need metrics, facts and alerts.

    What really matters is not poor performance but a degradation of the performance, because that is what users perceive. To measure performance degradation, you need to know the baseline, or what’s normal. This requires a tool that continuously measures the performance delivered to your end users and that can alert you when the performance is declining.

    Finally, because most Office 365 services are used in conjunction with internal infrastructure applications and appliances, they also can impact the service delivered. If you don’t monitor these applications at the same time and on the same dashboard, you won’t be able to troubleshoot the issue effectively. Some companies have hybrid Exchange, some still have Domino application servers, some are using gateways, some have proxy servers and so on.

    It all comes down to service delivered to your end users. Whether issues arise from the data center or your network, it’s important to monitor and understand what users are experiencing so you can maintain the highest level of service. Understanding the relationship between availability and performance from the user’s perspective helps you achieve better insights and oversight of your cloud or hybrid cloud health.

    About the Author / Jean-François Piot

    Jean-François Piot is vice president of Product Management at GSX, responsible for articulating the vision of the company’s product range and translating these into detailed specifications. His experience in IT processes and service management projects comes from previous business development positions at IBM and Beijaflore (IT Service Provider). He holds a master degree from the Grenoble Graduate School of Business.