devopsy.kinsta.cloud

Category: DevOps Culture

  • 5 Considerations to Get DevOps Right in Telco IT Systems

    5 Considerations to Get DevOps Right in Telco IT Systems

    Implementing DevOps in telco IT systems, such as business support systems/operations support systems (BSS/OSS), is not easy. This is especially true with telcos that overly rely on third-party commercial off-the-shelf (COTS) vendors or systems integrators (SIs) to implement new capabilities, and to run operations. To effectively control the vendors, telcos extensively use traditional levers found in SLAs, KPIs and RFx (requests for whatever). They also reinforce the organizational and mental boundaries between the telco and vendor staff. In a DevOps setup, such a traditional approach is not likely to produce the desired results.

    How, then, can telcos ensure DevOps success in their organization? Here are five considerations.

    Know and Understand DevOps Across the Organization

    Time and again, the understanding of DevOps varies from the top-down and across the telco organization. Some top managers believe DevOps is just a “faster version” of the traditional delivery and operations, wherein the vendor executes almost everything. It takes time for the management to fully recognize that the telco will have to take up much more execution responsibility than what it is used to. Having a correct and similar understanding about DevOps is criticat to accept the fact that the nature of the work and the extent of the telco’s involvement will change significantly. The sooner this is done, the better.

    Identify Which Applications Fit into the DevOps Model

    As part of the DevOps rollout plan, list the applications that can offer the organization significant business benefits through the DevOps model. While some applications, such as eCare or mobile-apps, will fit the bill, some others may not. Highly configurable COTS applications that do not undergo frequent code changes, such as IN or core billing, may not be brought under the DevOps model. For such applications, the traditional approach with traditional SLA parameters could fetch a better cost benefit.

    Reiterate that the Mindset Must Change

    The telco will find itself in a position where it has to work more closely with vendors than it used to do. Most likely vendor staff will be part of telco’s Agile and Ops team. The telco will have to foster a one-team thinking. For this, the first few steps toward building mutual trust should come from the telco. Do not expect the first few iterative releases to go smoothly. When that happens, don’t get frustrated—see them as opportunities to learn, adapt and fine-tune the DevOps approach within the organization. These are also avenues to reiterate that success/failure are shared responsibility, and that there will be no finger-pointing, regardless of whether a person is an employee of a vendor. In this process, an environment of acceptance naturally develops—i.e. the product owner agrees to product deviations, while the agile team that consists of vendors agrees to fuzziness in the requirements. It is highly recommended that key roles in DevOps be taken up by the telco staff, and they should involve themselves heavily in the scrum workflows.

    Obviously, classic-delivery SLA/KPI parameters and values will not fit in such a setup. Those would require a major revamp. An example: Give reward points to the team for on-time delivery and release. The number of points awarded will be fewer if not all the release objectives are met, but the team will get some points. However, if the team doesn’t deliver on time, it will not be awarded any points. Payment and commercials could be tied to the reward points. Similarly, if the telco decides to manage operations through a managed services contract, then it has to look again at the target operating model and SLAs.

    Be Prepared to Invest More

    Adopting DevOps doesn’t necesarily mean immediate cost reduction. The benefit analysis should cover aspects such as duration of a release and frequency of releases, not just cost reduction. To realize DevOps, telcos will have to incur cost for talent acquisition. Expect an increase in full-time equivalent (FTE) costs, especially for onsite FTEs. Telcos will have to spend on training and reskilling their staff on new tools and technology. Job shadowing from experts could be required until the team gets a good grasp on DevOps. There will be costs in creating new agile teams and creating new reporting structure.

    Enabling New Technologies and Tools

    It is a given that microservice-based platforms enables DevOps. This is because they truly enable agile teams to work separately with minimum dependencies and undertake continuous integration and deployment, which requires tools. Telcos will have to decide and finalize the platform, tools and DevOps framework. Some aspects to consider while finalizing the platform and tools include their market maturity, ease of maintenance and skills availability.

    Conclusion

    Habits run very deep. Changing those are not easy. This why most DevOps advocates emphasize that the mindset or cultural shift is the most crucial aspect of implementing DevOps in organizations with a history of traditional delivery and operations. This is especially true with telcos, which run on a  large set of legacy systems, both COTS and bespoke.

    — Vishal Augustine

  • Dev and Ops: Not from Different Planets

    Dev and Ops: Not from Different Planets

    At a conference, I heard a speaker remark that “Dev people are from Mars while Ops people are from Venus.” That’s catchy and captures what a fair number of managers believe instinctively. But is it really true? I’m not convinced.

    We all know that DevOps enables us to deliver software faster, with fewer bugs and much less downtime than traditional approaches. But DevOps is new, complex and often challenging for teams to embrace. Developers often guard their tools and operations shake their head each time they hear about some new asset bundling package they have to support. I think the first few meetings of Dev and Ops probably do a lot to build the notion that they are from different planets.

    In my experience, however, the opposite is true. To borrow a different popular catchphrase, they are more “brothers (or sisters) from a different mother.” When you examine the makeup of Dev and Ops people, you find more similarities than differences. Here are some examples of their common ground:

    • Consistency and Standards. Both Dev and Ops are technical endeavors that place a huge premium on things being “precisely” correct. Whether coding a service or setting up an AWS instance, “in the ballpark” doesn’t cut it. For this reason, both camps try to establish standards for how they work. Whether keeping Dev teams all using the same environment or Ops trying to keep Dev, testing and products all running the same, both need to benefit from building consistent environments.
    • Coding Skills. Whether coding or scripting, Devs and modern Ops teams need to be comfortable in a variety of highly technical tools. Build scripts, deployment, automated testing and reporting all require some coding skill.
    • Common Tools. If you look at the DevOps ecosystem, you’ll increasingly see tools that are intended for both Dev and Ops tasks. Dev people, for example, use Git for reusable code; Ops uses Git as well for configuration of things such as Docker. Also, Dev uses Jira to track bugs while Ops uses Jira to track tasks.
    • Shared Metrics. While their focus may be different (code versus pipeline), both Dev and Ops share many metrics of success—consistent delivery, fewer errors, responsiveness to customers and uptime, to name a few.
    • Test-Driven. Dev and Ops both embrace the “fail fast” mantra. They want to know if a change will break anything as soon as possible, long before it ever impacts a customer. Both often start with automated tests, which are run on Dev or local instances, to verify the changes are non-breaking.
    • Love of Automation. Neither Dev nor Ops enjoy manual, error-prone work. Both gravitate to tools that automate routine tasks, reducing variances and errors.
    • Star Wars. Finally, both groups are avid “Star Wars” fans. Just watch how they open automatic doors using the force or name servers or other things with sly “Star Wars” references (such as “Secret Rebel Base”). In fact, next time you’re in a DevOps meeting and somebody is wondering if Dev and Ops can get along, bring up Jar Jar Binks and marvel at the immediate—there will be a completely concordant reaction from both sides.

    So, are Dev and Ops from different planets? I don’t think so. I see Dev and Ops as being like those twins separated at birth, who, when reunited decades later, laugh the same, have the same twitches and both married spouses with the same first name. And in the best DevOps shops I have seen, that’s a really good thing.

    — Joshua Platt

  • Got DevOps Skills? A Look at the Upskilling: Enterprise DevOps Skills Report

    Got DevOps Skills? A Look at the Upskilling: Enterprise DevOps Skills Report

    IT recruiting is at an all-time high and organizations want T-shaped professionals with cross-functional skill sets to join their teams. While “T” represents an individual’s core competency, and the top of the T represents a broad range of related knowledge, there has been little clarity on which skills are the most important across a wide spectrum of possibilities—until now.

    DevOps Institute launched the Upskilling: Enterprise DevOps Skills survey in 2018 to focus on the human element of skilling. The feedback from more than 1,600 individual responses echoed the notion that regardless of vertical, organizational size, geography or role, DevOps practitioners need data-backed skilling statistics. As the focus within the enterprise, DevOps community shifts from “What is DevOps?” to “How do we do DevOps,” this demand is more crucial than ever. After all, “how to” requires “know-how” and “know how” requires a new set of skills. But which skills?

    According to DevOps Institute’s Upskilling: Enterprise DevOps Skills Report:

    • Soft skills and technical skills are equally important
    • IT operations knowledge received the most “must-have” votes as a functional skill in DevOps
    • Cloud and analytical knowledge are the top technical skills
    • DevOps Engineer emerged as the most popular job title (see below for figure N=812)
    N=812

    The report was developed in partnership with veteran research analyst Eveline Oehrlich, formerly of Forrester Research. This comprehensive data provides indications of which functional, process, technical and automation skills are “must-have,” “nice to have” and “not important.”

    This is particularly significant for IT professionals who are confused about where to focus their personal skills transformations in their journey toward the “T.”

    Should developers work on upskilling their testing, automation, process and/or security skills? Should operations professionals learn to code, test or explore more about machine learning and AI? What about project management and systems administration? The Upskilling: Enterprise DevOps Skills Report can be used as a snapshot of critical enterprise skills that are important to successful digital transformation.

    The Upskilling: Enterprise DevOps Skills Report was released March 4 and is now available for download. On behalf of Platinum Sponsor Electric Cloud, Gold Sponsor CloudBees and Silver Sponsor Lenovo, industry influencers and international DevOps community, we hope this data will help shrink the skills gap and support the transformational goals of both individuals and organizations. We’re already looking forward to the launch of DevOps Institute’s Upskilling: Enterprise DevOps Skills Survey 2020 to continue meeting the demand for ever-evolving skilling trends in the DevOps community—stay tuned!

    — Jayne Groll

  • Report Identifies DevOps Skills in Highest Demand

    Report Identifies DevOps Skills in Highest Demand

    When it comes to DevOps there’s general agreement that a skills shortage is holding back broader adoption. The real issue, of course, is identifying what specific DevOps skills are required most.

    With that goal in mind, the DevOps Institute this week published a report that finds the DevOps skills most in demand fall into three broad categories: automation skills (57 percent), process skills (55 percent) and soft skills (53 percent).

    DevOps Institute CEO Jayne Groll said that, given the fact that DevOps is all about automating repeatable processes, the fact that automation skills are the most sought-after is not all that surprising. But when it comes to process skills, the top five areas where organizations need the most help are software development life cycle (47 percent), understanding of process flow and analysis (46 percent), agile development expertise (42 percent), experience with test-driven deployment concepts and methods (40 percent) and system thinking (39 percent).

    The survey also identifies areas where survey respondents need additional functional skills; most notably, IT operations knowledge (52 percent), security (47 percent), IT infrastructure (44 percent), application design and development (42 percent), quality assurance (34 percent) and testing (31 percent).

    While the survey makes it clear what skills are needed most, it’s less clear whether organizations want to train people with the right attitude toward DevOps skills or find individuals with proven skills. About 30 percent of the survey respondents said they look for soft skills first, with an eye toward training or educating new hires on the process and technical skills later. But 32 percent of respondents said they first look for technical skills.

    In that context, however, it’s also worth noting that 43 percent of the C-level executives and IT management executives that participated in the survey identified business skills—defined as communication, influencing, negotiation, strategic thinking and leadership skills—as being the most important attribute of a new hire.

    Other noteworthy attributes of the organizations participating in the DevOps Institute include:

    • Two-thirds (66 percent) have already adopted DevOps to some degree.
    • More than three-quarters (76 percent) employ agile development processes.
    • Nearly half (47 percent) have adopted the ITIL framework as a best practice for IT service management.
    • Only 10 percent rely on site reliability engineering (SRE) practices defined by Google.

    Groll said the survey makes it clear there is still much work to be done in terms of advancing best DevOps practices. One the biggest challenges are all the silos within IT organizations that have their own nomenclature and culture. Each of those silos creates technical and cultural inertia that must be overcome to integrate processes end to end, she said.

    On the plus side, Groll noted the next-generation of IT professionals are being taught best DevOps practices in academic settings, which means the pool of job candidates with familiarity with DevOps practices should increase steadily. The downside is that it might take a while before those individuals learn enough about the business of the company that hires them to effectively employ those DevOps practices.

    — Mike Vizard

  • Digital Transformation Technology and Process: More Similar Than You Think

    Digital Transformation Technology and Process: More Similar Than You Think

    Right around this time of the year, the gym starts to get a little less crowded. Throughout January, it’s packed to the gills with folks freshly motivated by their New Year’s resolutions to get in shape. But by mid-February, the commitment seems to have all but dissipated.

    Why is it so hard to just commit?

    During my last workout, it occurred to me that the set of vacant treadmills next to mine was eerily reminiscent of the state of enterprise DevOps. According to Forrester, 50 percent of digital transformation efforts stalled in 2018. It seems commitment and follow-through are difficult in both personal and professional arenas. Admittedly, adopting and scaling DevOps is no easy feat. While reflecting on what does work, I found some parallels between the transformation process and DevOps itself.

    What Success Looks Like

    According to DORA’s “2018 State of DevOps report,” organizations with the highest level of DevOps execution, as measured by throughput and stability, also achieved the highest level of business performance in terms of profitability, market share and customer satisfaction. In a true case of going fast without breaking things, elite performers realized impressive performance gains:

    • 46 times more frequent deployments (throughput)
    • 2,555 times shorter lead times (throughput)
    • 7 times lower change failure rate (stability)
    • 2,604 times shorter time to recovery (stability)

    These metrics underlie a key tenet of DevOps transformation, which is breaking up large chunks of work into many small pieces.

    Shrinking the Change

    Smaller changes mean getting work out faster, with less risk. They take less time to plan, less effort to execute and are easier to troubleshoot. But, smaller code changes alone aren’t meaningful unless they are coupled with shorter release cycles. In essence, agile without CD doesn’t work.

    Stockpiling a few months worth of agile code iterations into a single monolithic release reintroduces an enormous amount of risk and negates much of the work done to adopt agile software development in the first place. Big queued-up releases often mean someone is stuck working the weekend to iron out all of the things that inevitably go wrong and no one should be stuck with weekend release duty. As Jez Humble put it, “The most important metric for continuous delivery is the number of person hours spent outside of business hours doing releases… and the goal for that is zero.”

    In practice, CD is what brings agile development to fruition; it’s why Gartner finds that 40 percent of IT departments have already invested in CI/CD with a full third planning to invest in 2019. CD is all about small pieces. In the same way agile enables smaller code changes, CD enables smaller releases.

    Not Just Smaller, but Iterative

    At GitLab we use the phrase, “just commit” as a shorthand for continuous delivery and deployment. The idea is that when you have a completely automated CI/CD pipeline, all a developer needs to do is commit code to their branch, and everything else needed to get that code running live in a production-like environment—from containerization to security testing—is automated. Similarly, getting code to production is as easy as merging the feature branch back into trunk with the entire production deployment automated. The message to the developer is, “Just commit your code and let your DevOps tooling do the rest.” This is CD in a nutshell.

    With this type of automation in place, the cost of making a change significantly decreases. This means the cost of getting started is very low, but so is the cost to keep going. Instead of spending a massive effort to map an entire path to your solution, you simply need to figure out your next step and keep iterating. According to GitLab’s “2018 Global Developer Report,” 67 percent of developers report the top challenge to getting work done is unclear direction. By shrinking the change, business leaders will find it easier to set direction for the next step without having to map the entire path. This brings us back to digital transformation.

    Commit to Digital Transformation

    Just like a monolith being decomposed into microservices, the practice of breaking large chunks into small pieces applies to driving digital transformation as well. Like Creighton Abrams said, “When eating an elephant, take one bite at a time.”

    How can enterprises shrink the change?

    For most organizations that achieve successful digital transformation, the initiative usually starts with a single team or innovation unit. This group proves the success for the rest of the organization. It develops that culture, tools and processes that other parts of the organization can then adopt. One example of this principle in practice is the transformation that Chris Hill, Head of Systems Engineering, Infotainment, led at Jaguar Land Rover. When Chris came on board, only three build engineers had the ability to release to production and it took four to six weeks to complete each release. Chris gradually scaled a CI/CD platform, step-by-step over two years, until every team across the organization was able to complete a release in less than 30 minutes. Today, the Jaguar Land Rover completes 50 to 70 deployments per day. This iterative pattern is seen over and over again with enterprise digital transformation. It’s encouraging because it means successful digital transformation can be driven by either executives top-down or engineers bottom-up.

    Keep the Fuel Burning

    According to Argyle, “91 percent of companies have at least started on digital transformation, but only half have created an organization-wide strategy.” This seems to align with Forrester’s metrics on half of the organizations that stall.

    So, how can organizations keep going and avoid the stall out?

    If I can make a confession, I am also one of those New Year’s resolution makers who just started working out in January. How it is that I’m still hitting the gym while others have dropped off? I have found that taking a “DevOps” approach is a key to my success. Instead of broad sweeping changes, I make small incremental improvements one day at a time.

    It turns out the strategy for getting started—“just commit”—is the same strategy to keep going.

    By starting small with a single project and gradually expanding scope in iterative steps, an organization can continue to improve over time at a rapid pace. Small, single-commit code changes coupled with CD increase throughput and stability. In the same way, small, incremental process improvements bring about the greatest success with digital transformation.

    — William Chia

  • 5 Questions to Ask Remote Web Developers

    5 Questions to Ask Remote Web Developers

    Great web developers are waiting to be hired. Your company could seize upon the opportunity—but only if you’re willing to be patient, dedicated and a little creative.

    Don’t scare away qualified remote web developers with a few poorly worded questions or a mistimed probe. Ask direct and concise questions that can help you understand the candidate.

    Be sure to include the questions listed below to ensure you’re interviewing a remote web developer to the best of your ability:

    1. Have You Worked Remotely Before?

    Remote work is different from many traditional forms of work. For a company to properly manage a remote web developer, the web developer must have a solid understanding and knowledge of remote working environments.

    If an exceptional web developer candidate hasn’t telecommuted before, it doesn’t bar them from the pool of qualified candidates. However, it is easier to onboard and retain remote web developers who have had the experience of working remotely beforehand.

    2. Are You Familiar with Popular Web Development Frameworks?

    Remote web developers, like any other web developer, must stay abreast of the latest trends and advancements in the web development field. To master their craft, they must keep a running knowledge of the latest tools, frameworks and web programming languages.

    An experienced web developer should, at the very least, be aware of a large and robust framework such as Bootstrap. They should be able to explain how the framework works, why web developers use it (it’s much better to utilize a framework to save time, in most cases) and the drawbacks of utilizing a premade framework.

    3. How Much Experience Do You Have with Web Programming Languages? Are You Familiar with Server-Side Languages?

    Every web developer should be proficient in HTML and CSS. Scripting languages such as Angular.js, Express.js and JSON are important, too. JavaScript adds interactivity to websites. HTML acts as the skeleton and CSS acts as the design.

    Preferably, an experienced remote web developer candidate will have three to five years of experience with these languages.

    It’s also important to ascertain whether the web developer you’re interviewing has the ability to code the back end of websites. Do they know how to set up a database? Do they understand how to store values in a website’s database?

    4. Which Content Management Systems Do You Use?

    A content management system (CMS) helps streamline organizing, adding and managing content for a website.

    One of the most popular CMS is WordPress. Web developers leverage CMS to utilize built-in structures and architectures to create dynamic blogs and alluring corporate websites.

    Setting a website up with a CMS also makes it easier for writers, marketers and designers to tweak already-developed websites. They can add content without needing the web developer to intervene.

    5. Are You Comfortable with Regular Phone and Video Meetings? Do You Have Experience with Telework Software?

    Remote work has unique challenges. One of the most challenging aspects of working remotely is maintaining communication. Miscommunication can occur more easily for remote workers, so it’s important that you keep communication lines open and stress the importance of speaking clearly and confidently.

    It’s much better if a candidate already has experience with telecommunication software such as Zoom, Skype or Slack. Knowing how to interact via these technologies will lessen the chances of miscommunication in telework environments.

    Conclusion

    How you conduct a remote web developer interview is entirely up to your organization. There are several methods for conducting technical interviews, telework interviews and startup interviews. Each one of them is effective in a unique way; very few are ineffective or harmful. Adding a few questions to your interview script could help make whichever strategy you employ more effective.

    Ask remote web developers about their experience working remotely. Remember to ask about experience with certain telework software programs and telecommunication applications. On the technical side, be sure to ask about programming skill, familiarity with web frameworks and their experience with CMS.

    Companies that seek to hire remote web developers should take note of the interview questions listed above. By asking these five questions, interviewers can gain a great deal of insight into a web developer candidate’s habits, passions and challenges.

    Using these interview questions should make the vetting and hiring process easier for everyone, helping interviewers assess a candidate’s adaptability, experience and dedication.

    — Anthony Coggine

  • Survey Finds Mixed Progress on DevSecOps

    Survey Finds Mixed Progress on DevSecOps

    A survey of 5,558 IT professionals published today by Sonatype in collaboration with CloudBees, Carnegie Mellon’s Software Engineering Institute, Signal Sciences, 9th Bit and Twistlock finds 27 percent of organizations have mature DevOps practices in place, while another 48 percent are still working on improving them.

    Despite the DevOps works that remains to be accomplished, however, the survey also noted that 47 percent of respondents still manage to deploy code multiple times a week.

    The survey, announced at the RSA Conference, also finds that 25 percent of respondents now consider security to be a fundamental element of quality assurance process. But responses among the 23 percent of respondents who identified themselves as developers suggest implementing best DevSecOps practices remains a significant challenge. For the third year in a row, about half of developers reported they don’t have enough time to address cybersecurity issue.

    The survey, however, also noted that organizations that do implement best DevSecOps practices see substantial benefits:

    • Elite DevSecOps practices are 700 percent more likely to have fully integrated and automated security practices across the DevOps pipeline.
    • Almost two-thirds (62 percent) of respondents with elite programs have an open source governance policy in place that is improved via automation.
    • Organizations with elite DevSecOps practices are three times more likely to provide application security training to developers.
    • A full 81 percent of those with elite practices have a cybersecurity response plan in place.
    • More than half (51 percent) of respondents with elite practices leverage automated security products to identify vulnerabilities in containers.

    Brian Dawson, DevOps evangelist at CloudBees, said the survey results show that despite the gains being made by some organizations, there is much work to done in terms of overcoming technical hurdles and cultural tribalism. For example, 35 percent of all respondents said their No. 1 challenge in application security is that they find out about the problem too late in the process. Nearly half (46 percent) of organizations without a DevOps practice do not have application-level credentials encrypted. Half of respondents (50 percent) with a cloud infrastructure said they rely on their cloud service provider to secure their IT environment, and 28 percent of respondents admitted they do not protect secrets such as passwords, API keys and certificates.

    The core issue most organizations need to focus on first is to firmly establish a DevSecOps mindset, said Dawson. Much progress has been made in driving adoption of DevOps practices. Now those principles need to be extended to include automating the implementation of security controls as applications are being built and deployed. The survey noted that 63 percent of organizations that have elite DevOps practices are now informing developers of cybersecurity issues directly within the tools provided to the DevOps team. Armed with that information, it becomes much easier to incorporate security fixes within the overall application lifecycle management process.

    It may be a while before best DevSecOps process are widely implemented. But as cybersecurity issues continue to be incorporated within the quality assurance process, it’s now only a matter of time before applications become much more secure.

    — Mike Vizard

  • How SRE Creates a Blameless Culture

    How SRE Creates a Blameless Culture

    People in tech love to praise failure as essential to innovation. But at far too many organizations, when failure happens, it is punished. When an IT system goes down, an individual or a team is immediately ID’d, blamed and shamed. This is counterproductive. Blame damages companies by creating a play-it-safe atmosphere and stifling innovation; even worse, when mistakes are made, they are often hidden.

    Fortunately, more and more organizations are recognizing the damage that blame can do and embracing a blameless culture. This is a big shift, yes, but companies can make the move to blameless—and make it stick—if they bring site reliability engineering (SRE) best practices to their technology teams.

    Why Blame is Bad

    I’ve worked at several fast-growth companies that suffered under a culture of blame. At one, we were scaling quite rapidly. Change was constantly being introduced to the infrastructure, which meant that it broke all the time.

    Because we had customers running their core business on our systems, our reliability requirements were very high—so high that for a while we stopped feature development altogether, in fear that new features would bring our systems down.

    This was difficult enough, but the culture of blame made it worse. There was no psychological safety net and fingers were pointed constantly. There was an us-versus-them mentality—engineers versus non-engineers, developers versus operations, developers versus developers. Developers dedicated way too much mindshare to protecting themselves and their domain.

    In a blame-rich environment, developers who should be delivering concrete results spend too much of their time in meetings trying to justify their work. There is a culture of retribution for mistakes. This leaves little room to learn and grow from failure because developers get harassed by management when they do make mistakes, or they get fired.

    Your best people—those empowered to make decisions—see their responsibility undermined and go elsewhere. Creativity and innovation slow to a crawl. Meanwhile, competitors run ahead.

    Why Blameless is Better

    In a blameless culture, everyone feels safe and no one is afraid to make mistakes. It’s a psychologically secure environment, where true DevOps can happen. Developers feel confident enough to express their ideas and take chances. Development folks and operations folks collaborate well, and everyone is aligned on the problem.

    The work environment is positive and workers are assumed to be doing their best. When mistakes happen, people aren’t blamed. Instead, any error is viewed as a manifestation of an underlying vulnerability in the systems. Attention is focused on fixing those vulnerabilities and, as a result, systems are constantly improving.

    How to Get to Blameless

    Let’s say you just had an outage and your database crashed. Don’t start by placing blame on a team or individual. Rather, start by conducting a postmortem. Ask questions and find the answers: Why did the database crash? Because Alex pressed the wrong button. Well, why? Because we didn’t have a system in place to enable automatic checking or a review process. So why did we not have that?

    Now you’re getting to blameless and you’re reaping the benefits. You’re having a productive conversation and you’re focused on fixing your systems, not affixing blame. It’s not about Alex pressing the wrong button; it’s about how your organization is not set up to ensure success. It’s about this vital opportunity to learn from a mistake and get better. Don’t fire Alex. Put a solution in place so the problem doesn’t happen again.

    How SRE Can Help

    To get to a truly blameless environment, you need to implement SRE best practices. Why? Because SRE creates ultra-scalable and highly reliable software systems, by building in automation and tooling—moving the human element out of the picture and focusing on the systems.

    One of the founding principles of the SRE movement is blamelessness. By implementing an SRE team and establishing SRE practices, your organization can move from the traditional operational model of break-fix to an environment that applies a development approach to IT operations, with the goal of improving reliability via automation, continuous integration and delivery.

    For this to happen, you will need a SRE champion to quarterback your SRE effort. You will also need organizational buy-in to blamelessness. It has to be practiced top-down and have the full support of all leaders in the organization.

    Tooling is critical, too. Look for tools and dashboards that can quickly do postmortems and find action items that need attention, that can identify the most commonly impacted services, products and customers, and the contributing factors that most often lead to incidents. The right tools will help you resolve incidents faster, by providing a clear overview of metrics related to incident identification and the actions that need to be taken.

    You want to move fast and be innovative. But can you do it reliably? With a blameless culture, you can. You’ll have a safe environment that attracts and retains the best talent. You’ll move quicker, because you’ll be focused on fixing systems that break and finding out why—not the blame game. By delivering a higher degree of reliability, you’ll build greater trust with your customers.

    You’ve heard the saying: Failure is not an option. In a blameless environment, failure is always an option—because it means that systems are always improving and innovation is always happening.

    — Ashar Rizqi

  • ‘DevOoops’ Moves: Poorly Defined KPIs

    ‘DevOoops’ Moves: Poorly Defined KPIs

    DevOps’ ultimate aim is to create efficiencies in the software delivery process. But, it doesn’t always work out that way. Sometimes DevOps initiatives lead to mistakes—something we lovingly call a “DevOoops.” CloudBees recently polled some colleagues and came up with five examples of a DevOoops in action. The following is a discussion submitted by product marketer Juni Mukherjee about poorly defined key performance indicators (KPIs).

    Scenario:

    How we define KPIs can change the game. For example, one quality engineering team defined the number of tests executed per sprint as a success metric. Humans are driven by incentives and the natural tendency of the team was to add more and more tests without even considering archival of outdated ones. Think of this: We could have a larger impact with fewer tests and, more importantly, fewer tests lower the test cycle time. So, instead of sheer quantity, we should focus on the coverage and effectiveness of tests. Similarly, a release engineering team defined the number of releases per sprint as a success metric. The number of releases reflects on velocity for sure. However, releases move bits from point A to point B without assessing the value added to business. It is critical to tie a business KPI (such as the number of new customers acquired, percentage of revenue increase, etc.) to velocity, such that we know we are investing in valuable speed and not suicidal speed.

    Management consultant Peter Drucker boiled his whole philosophy down to the following few words: “If you can’t measure it, you can’t improve it.”

    Successful DevOps leaders take this saying to heart. To understand where they’re going, how to get there and ultimately how far they’ve come, these leaders know how critical it is to establish KPIs that fit the overall objectives of their DevOps transformations. KPIs provide appropriate and continuous insight—and they plot a route to help you get to your destination.

    The big question is: Are you measuring the right things? If not, you could be spinning their wheels.

    To avoid this “DevOoops” scenario, you need to start by establishing KPIs that not only align to your objectives but can also be established so that they create a continuous feedback loop. As the project progresses, as you mature and learn more, you will likely need to maintain, monitor and possibly adjust and update these KPIs.

    What would be an appropriate test coverage goal? Teams all want to improve quality. But by how much? Seventy percent? Eighty percent? One hundred percent? This is an example of a KPI that needs to be considered in the context of your application as well as measured against your real objective.

    You can have 70 percent test or code coverage, but if that’s just treated as a measure to test as much code as possible, you may miss the mark. If you are covering 70 percent of the wrong code and the 30 percent that’s missing is critical code, you’ve put in significant effort with the risk of little or no return on quality level.

    In terms of your goals, increasing and monitoring test coverage may be an indicator of quality, but it may not measure your actual quality of work. In this case you want a KPI that’s measuring the outcome rather than the activity. Look at what is severity 1 and severity 2 defect rate in production. You can see code coverage get to 90 percent, but if you’re not seeing a decrease in severity 2 defects, you won’t really be seeing an improvement in our code coverage metric.

    Takeaways: 

    • You’re not implementing DevOps just to say you’re doing it; you need to develop goals and objectives for your initiative and align to them.
    • Once goals are defined, you need to align KPIs and success metrics to those objectives. Properly defining these success metrics and KPIs will dictate what gets prioritized when building pipelines.

    — Brian Dawson

  • DevOps Disillusionment? 4 Behaviors to Get Your Team to the Other Side

    DevOps Disillusionment? 4 Behaviors to Get Your Team to the Other Side

    For most teams, the adoption of DevOps and an agile mindset involves many twists and turns on the path to maturity. It’s not uncommon for teams to experience a “trough of disillusionment” period as:

    • Initial euphoria and “us-against-the-world” team mentality ushered in by adoption of new work patterns wanes with time.
    • Client demands and pressures to produce intensify.
    • Sustained leadership engagement and support fails to materialize.
    • Key evangelists move on to new roles.
    • Apologists for organizational status quo never tire.

    And before you know it, what once felt like a cultural movement seems mired in the realities of metrics, tools and keeping the operational wheels on.

    Pushing through these challenges and self-doubt is a critical step in DevOps maturity that can refine and strengthen a team’s character. Getting to the other side of “the trough” requires a team and its leaders to reconnect with why they embarked on the DevOps journey in the first place, and remind themselves that a lasting shift in behaviors is the ultimate goal. Where behaviors change, results follow.

    What are these behaviors? With so much written by thought leaders in the DevOps community, teams have the luxury of expressing the behaviors they intend to model in a way that resonates with their situation. Ultimately four actions, practiced faithfully, will build the team muscle memory necessary to get to the other side.

    Choose Wisely

    Overcoming the urge to please others and having the courage to say “no” to work that is unlikely to add customer value is perhaps the most important behavior of all. Doing it in a way that preserves credibility and trust is a skill that can only be learned through deeply understanding who your customers are and what problems they are trying to solve. Choose wisely and prioritize ruthlessly. Flawless IT delivery of unnecessary features is a shared failure for both customer and DevOps team alike.

    Commit Boldly

    Go “all in” on the work you choose to take on. Ensure that the desired outcomes and acceptance criteria of your work are clear before committing, and rely on those closest to delivery to determine how to get it done. Once established, game on! Commit boldly to small deliveries that add customer value and invite accountability for what you take on—knowing that customer satisfaction is the ultimate measure of your success.

    Adapt Quickly

    Sometimes, we fail. In big and small ways teams employing DevOps don’t always meet their own expectations or those of their customers. Our failures don’t define us; how we respond to them does. Learn. Adapt. Move on. Welcome the changes to requirements you know will come and adapt quickly by leaning in to the power of small and frequent deliveries to place you back on track quickly.

    Share Freely

    Teams employing DevOps are part of a larger community of like-minded individuals committed to delivering customer value quickly and making their organizations better. Take your obligations to continuous improvement seriously, share best practices freely and solicit expertise to help improve the pace and quality of your delivery and those around you. Helping others succeed has the remarkable effect of making yourself better in the process!

    — Chad Sands