devopsy.kinsta.cloud

Tag: ops

  • The 7 skills Ops pros need to succeed with DevOps

    The 7 skills Ops pros need to succeed with DevOps

    There’s a lot of confusion about what DevOps exactly is, and what it takes for enterprises and individuals alike to succeed. Some argue it’s about successfully using collaboration toolsets. It’s not. DevOps is about collaboration between software developers and IT operations teams. DevOps grew from the need for a more agile organization, and to knock down the barriers between development and operations so that applications and IT resources could be delivered more quickly, and improved through steady iteration.

    What does all of this mean, specifically, to operations professionals – and the skills they need sharp to thrive? The walls coming down are a bit of a culture clash for most large enterprise operations and development teams. And both developers and operations professionals need to ensure that they have the right skills in place to succeed in this new environment. In this post, we’re focusing on those skills ops members need in the years ahead.

    The skills that are important may surprise some. Consider the findings from CA Technologies’ recently commissioned a study, What Smart Businesses Know About DevOps. “The most in-demand skill requirements for success with DevOps were actually soft skills,” says Andi Mann vice president of strategic solutions at CA Technologies. “I expected to see skills like programming or scripting; experience with specific tools for provisioning, configuration management, release automation, or shared monitoring; or even an understanding of Agile or Lean practices. In fact, those skills were much lower,” Mann says.

    More on the results from the CA study below, but first we want to share the results of our informal anecdotal survey of a handful of DevOps experts and practitioners, well known here at devopsy.kinsta.cloud. Here they are:

    First Up: The Soft Skills
    Interestingly, it is these soft skills that are both needed and often derided. “Everyone doing DevOps, whether Dev, Ops, security, QA, project management all need [soft skills],” says David Mortman, chief security architect and distinguished engineer at Dell, Enstratius.

    1. The ability to communicate clearly and concisely
    “Sharing is a key tenet of DevOps and if you can’t share what you know and what you’ve learned, then having that knowledge is effectively useless. You’ve just made yourself a bottleneck,” Mortman says. The other skill Mortman cited along these lines is striving to actually understand the shoes in which your fellow enterprise collaborators stand. “I can’t emphasize enough the value of having empathy for everyone else and what they need to get done. Every role has its challenges and empathy for those challenges goes a long way to a more congenial workplace,” he says.

    2. Collaboration techniques
    New collaboration techniques are essential, explains Donnie Berkholz
    IT industry analyst at RedMonk, and council member and Sr. developer at Gentoo Linux. “Ops will need to learn effective, distributed collaboration techniques such as those used in open-source software development. Leading-edge DevOps organizations make heavy use of online chat and shared tooling to log both communication as well as action. In some pioneers, this even involves so-called “ChatOps” where you can make changes to infrastructure through calls to an automated bot in an online chat. Typically this includes asynchronous communication in preference to phone calls, both long-form (e.g. email) and short-form (e.g. IRC), in an archived and searchable form,” he says.

    This enables, Berkholz says, on-call ops during any crises or fires that arise, as well as eases the ability to identify and brainstorm infrastructure changes, where everyone can easily contribute their ideas.

    3. A flexible mindset
    We’re not necessarily talking about advanced forms of mental Yoga, but certainly the ability to change as knowledge and conditions change, whether those changes are technical or business driven. “The willingness to make changes and to evaluate alternate options is key to DevOps. Things are going to fail, you or a decision you’ve made are going to be the cause of that failure. Or things just won’t work as well as you’d hoped or thought. Be willing to change your mind and look for better options,” says Mortman.

    4. Good enough is never enough
    It’s about consistently seeking continuous improvement. “This mindset is laser focused on continuous improvement daily,” says BEHR. “It is the ability to unlearn and acquire new more relevant knowledge in light of complexity and emergent information – then disconfirm it with the scientific method and move on to application only where relevant,” says Kevin Behr chief science officer at CXO advisory and board advisory practice Assemblage Pointe, Inc.

    Next up: The Technical Skills

    Understanding of modern software engineering
    5. “To move to a world of “infrastructure as code,” where all systems are maintained using configuration-management tools such as Puppet or Chef, including version control, unit and integration testing, agile software development, as well as continuous delivery and deployment models,” says Berkholz. “Traditional sysadmins may still live in a world of login prompts and manual edits of configuration files, so this will be perhaps the largest hurdle in terms of knowledge,” Berkholz estimates.

    “The technical mindset I look for are skills which allow a team to treat servers more like cattle and less like cats,” says Andrew Storms Sr. director DevOps at CloudPassage. “To work effectively at that kind of systems mentality, DevOps engineers need programming and systems architectural skills. Whether that means Ruby or Python or whatever is needed to effectively work with orchestration tools, APIs, and automation. It’s less about needing to know a specific programing language more about understanding programing algorithms and robust systems design,” Storms says.

    6. Modern software architecture
    This includes cloud deployment and administration as well as distributed systems. “As we move to a cloudy world of software, ops will need to deploy, maintain, monitor, and troubleshoot production problems with this infrastructure. And yet in many cases it’s much more complex than the previous equivalent, although it brings with it significant benefits in agility and uptime if done right. Ops will need to familiarize themselves with cloud instances and services, as well as lower-level software infrastructure such as Linux distributions and databases when deployed in a cloud,” says Berkholz.

    That also includes taking a step back from the trees to consistently stay aware of the forest. “Seeing the system and its boundaries rather than just the tasks and their due dates. This understanding of the interactions between the various actors in a system and [understanding] just how those interactions are making value occur is fundamental,” he says. “Without this understanding we will not know when to push harder versus be silent,” he says.

    7. Understanding that security incident response are continuous efforts
    Security and safety in the new DevOps environment needs to embrace new ways of addressing security and QA. “This requires continually emergent solutions rather than using ‘tried and true best practices.’ The effective team will evolve through relentless self evaluation, study, and application of more relevant and effective methods to work in complex infrastructure environments,” says Behr.

    That’s all great advice, and certainly not all inclusive of the ideal skillsets. We also found it interesting that CA Technologies’ survey of 1,300 senior IT decision makers echoed the need for broader business and so-called soft communication skills when it came to DevOps success.

    These included: knowledge of business priorities/strategies/metrics (47%), knowledge of current business processes (42%), communication skills (36%), and project management and people skills both scoring 30%. In fact, experience with specific tools was dead last in the survey at 12%.

  • Is Ops the quarterback of the DevOps team?

    Is Ops the quarterback of the DevOps team?

    Without stepping too deeply into the charged debate on whether or not developers are at the pinnacle of the IT food chain, I’ve recently observed that many of the DevOps leaders I’ve been speaking with are from operations. For me it begs the question – are the ops folks quarterbacking the DevOps process at most companies? In other words let me put it bluntly, are the Ops folks leading the way forward in DevOps?

    Many DevOps thought leaders have taken great pains to describe and articulate that development, ops, qa, security and many other groups need to work closely together to make DevOps a success inside of a company.

    Some have focused in on the leadership role that developers have taken (and the associated burden that goes along with that role). For larger companies developers are often not the leads when it comes to DevOps. Developers are no doubt a key part of the equation and integral to practicing DevOps within the organization, but the Ops people seem to be calling the plays and leading the teams. This has led me to the question of who really leads the DevOps process within a company?

    Why Ops?

    As with any key initiative within a company, there needs to be a focal point, a champion or leader to advance it. DevOps is no different. So, then, how do companies organize themselves to become and execute a DevOps methodology?  Recently, we’ve spoken with three senior IT leaders – all in ops – from companies that are beyond the initial startup stage, but aren’t yet behemoth enterprises either. All three ops leaders either had the title and role of VP of Operations or VP of DevOps (without trying to get into the “is DevOps a job title controversy). Their job is to deliver code to production as fast and safely as possible. This meant that they are directing, helping even conniving their counterparts in engineering to deliver smaller batches of code faster and on a regular basis. They then push this code out to production. In essence they were playing the role of the quarterback of the DevOps process.

    Just as the quarterback directs the players, calling the play in the huddle and directing the offense, these Ops folks are quarterbacking the process of DevOps.

    As such, these senior leaders were building the overall product delivery pipeline – from ideation all the way through deployment and feedback. While the ops team may not be involved in each part of the pipeline, these leaders were integral in building out the product delivery pipeline. For most organizations that product delivery pipeline is starting with sales and marketing – what needs are the business trying to solve in this particular iteration of the product or service. From there, the various teams are understanding the implications for their particular piece of the pie. Engineering is understanding how to build the features and functions. QA is understanding how to test the functionality as well as automate the QA process. Security is making sure that the functionality doesn’t create vulnerabilities or security holes. And, ops is thinking through how to push all of these features out to the customer. As in any DevOps organization, there are a lot of moving parts with many pieces needing cross functional coordination. And, therein lies the question – we all may practice DevOps and should all practice it; but, who leads the effort?

    Clearly other senior leaders could take the lead on this activity. There is frankly no reason why the VP of Product or VP of Engineering could take the lead, but it has been an interesting observation that I have been seeing more and more ops leaders taking the reigns of DevOps at mid-sized organizations. We would really appreciate hearing who you have seen taking the leadership role in DevOps – in your organization or elsewhere. Let us know by commenting below!

  • The Push to Make Ops More Social

    The Push to Make Ops More Social

    Given that social coding is a thing, what about the other side? Operations has the same image of lone wolves solving problems in the dark. The culture of on-call is built around the concept that when you’re holding the pager, you’re the one on the front lines, so that everyone else can sleep.

    Social operations, the ops in DevOps, is going to be about the same things that made social coding important: knocking down walls between individuals; making the knowledge of a system or infrastructure explicit and available, in context. But social operations is more difficult in two major ways: operations problems are often multidisciplinary, and often temporal – meaning that you have to be social when the situation arises,not on your time!

    When I say that operations problems are multidisciplinary, I mean that solving a problem in an operational context might require a number of people with different domain expertise. The solution is going to be much more complex and may not be portable to other people. Fixing something that’s wrong with a puppet script inside of our own infrastructure is fundamentally different than contributing a bug fix back to an open source project, or otherwise contributing to an open development environment.

    And when I say that operations problems are temporal, I mean that many of the forms of social engagement we’re getting used to in a social coding environment – forums, pull requests, etc. – are eminently useless when you’re in a firefight. When something’s wrong enough that you’re being paged at 2am, waiting for someone to respond to a comment thread on a pull request isn’t an option. You have to be able to get the right eyes on the problem, in a timely manner, if you want to solve infrastructure problems faster.

    So, how can we make the operations side of DevOps social? What parts of “social” are useful in the context of an operations issue?

    Opening up the knowledge of your team is a good place to start. Having the ability to get people involved is another good thing. Basically, what can we do to make it so that you know what you need to know, whether that’s how a problem was solved in the past, who’s available to get help, or what else is going on. That’s what we’re trying to figure out at VictorOps.

  • Organizational Dysfunction: The original vulnerability

    Organizational Dysfunction: The original vulnerability

    During your teenage years, you may have figured out a way to get a ‘yes’ from your parents by playing off of the responsibility and knowledge gap between them…walking up to your mom and saying something like, “Hey Mom, Dad said he would drive me, can I go out tonight?” and then proceed to your dad and say something like, “Hey Dad, Mom said I can go out tonight and you need to drive me.” There is nothing cleverer than a determined teenager wanting to get his or her way. Such is the case of targeted attackers and the way they play one IT department’s responsibilities off another. This is the grey area where the organizational chart and separation of duty can introduce weaknesses exploited by the more advanced threats we face today.

    You don’t have to look far into your organization to see this at work. The security team does its thing, networking engineers do their thing, and IT operations teams in general all have administrative realms that define functional boundaries as to what they can see and do. To borrow from Colonel John Boyd’s OODA loop, each team has its observation and orientation (OO) process for gaining operational visibility and making decisions, as well as specific decisions and actions (DA) they can execute within their administrative realm or set of capabilities to bring change to their organization. The weakness I’m highlighting in this post is when the decision and action appropriate for a set of observations, and the observations themselves, exist in two separate departments and can be exploited by the attacker.

    To exemplify the problem, let’s begin with the question: Is a distributed denial-of-service (DDoS) attack a security or a network engineering issue? The problem I see most often is that one department will have a precise view into network telemetry for early threat detection, but will not yet have the ability to mitigate or remediate the threat.

    Or how about the situation where credentials for one of your executives are harvested, and now the attacker need not trigger any security events at all in order to carry out an attack. In this situation, the attacker merely needs to log in and plan the next stage of the attack, roaming freely and without suspicion across your network because they are only generating networking traffic. The network team does not act because everything is still available, and the security team does not act because there is nothing being denied in access logs, no firewall violations, and nothing triggered on the IDS/IPS.

    Let me be clear in that I am not saying that everyone has to become an expert in everything; specialization is healthy, but only when it does not compromise communication! In the example of Mom and Dad being played by the teenager, if Mom and Dad had a better communication protocol, the teenager’s tactics would be thwarted. If you look at some of your operational procedures, and start to think like an attacker (or teenager again), I’m sure you will find at least half a dozen communication gaps that a clever attacker could exploit.

    This organizational dysfunction is going to require change that may appear to be radical because the systems that support it are so ingrained. Big analyst firms have defined their technology categories as law and they police them as if you were breaking the law if you dare to create something different – something more cross functional. Some best practices out there are, in my opinion, worst practices because they are so focused on the operational effectiveness of the department, and they leave out other departments or the active threat entirely. Lastly, even some prescriptive compliance requirements force organizations into strict separation of duties, but when implemented, create a broken OODA loop where the ‘OO’ does not connect with the ‘DA.’

    There is an opportunity for DevOps to completely recalibrate how we deliver a cross functional operational unit which is why I took the time to write this article for devopsy.kinsta.cloud.  Through the lens of Boyd’s OODA loop, you can already see the security ramifications because the tempo of DevOps allows the defense to loop at a faster rate than the adversary – the dominant strategy of the OODA Loop.

    I hope this post is a call to action for all you big thinkers. Get creative, get innovative, and do not be afraid to erase some lines and boundaries within your IT organization if it fosters a higher level of communication amongst your teams. Think of your organization like a great band playing live music. All of the performers play different instruments, yet they are listening to one another just as hard if not harder than they are producing sound. They improvise and adapt to one another; monitoring systems are put in place so they can have just enough of each sound in their own ear. This is what effective, cross-functional teams look like.  I’m just going to say it: DevOps is Rock and Roll!

    Oh, and hey, follow me on twitter: @tkeanini

  • VMware IT Private Cloud — DevOps Provisioning

    VMware IT has deployed a private cloud based on the vCloud Suite. Dev and Ops teams are one tenant using the cloud to automate dev test provisioning and streamline application development. Automated provisioning has improved how these teams work together. And here is how.