Category: Blogs

  • How to Fix ‘Dark Debt’ in Your IT Organization’s Culture

    How to Fix ‘Dark Debt’ in Your IT Organization’s Culture

    Poor decisions, bad metrics, lack of communication can accrue dark debt in any company

    Buying a low-quality microwave meal from a gas station, kicking a sock under the bed before guests arrive, jaywalking … We all take shortcuts. But unlike enterprise software delivery, these decisions rarely cause any collateral damage. The ramifications of taking shortcuts in IT, however, are much more serious.

    These decisions—i.e. “technical debt,” the “rework” debt owed after opting for a convenient solution instead of a better approach that takes longer—has implications on your company’s ability to grow and adapt to change. Fortunately, since the 1990s, when the term was coined by Ward Cunningham, we’ve learned how to identify and manage most forms of technical debt, understanding how to limit its impact and manage the interest.

    Your organizational culture can accrue debt, too. For example, a company that finds it easier to hire a homogeneous team up front might discover the costly cultural trade-off of hiring diversity later. When it comes to this “cultural debt,” we are much less disciplined. We often make decisions in the dark, without understanding how they are going to play out over the long term. “Dark debt” builds up over time and creates a toxic work environment in which people are not set up for success.

    Common issues indicating dark debt include:

    • Failures lead to finger-pointing.
    • Metrics incentivize undesirable behavior.
    • Lack of candor.

    Ways to Combat Dark Debt

    Call Out Psychological Safety Indicators

    Failures are inevitable, whether we are discussing power outages, data breaches or false missile alerts. A “safe to fail” mentality promotes learning and support adaptability. If employees don’t feel safe telling the boss bad news, the latter might be the last person to know what’s going on—not a good situation for a company working to disrupt an industry, or one expecting to stay listed on the Dow Jones.

    A Net Promoter Score-like survey can help identify areas where employees experience dissatisfaction. Here are some sample statements for your NPS survey that are based on Westrum’s organizational culture model as introduced in the “2014 State of DevOps Report.”

    • On my team, failure causes inquiry and not blame.
    • Our leadership is open to hearing bad news.
    • In my organization, failures are learning opportunities and messengers are not punished.
    • People on our team trust one another.

    The survey results will shine a light on the forces shaping your culture. Instead of finger-pointing, you can learn to address issues earlier and ask questions such as, “Why did it make sense for someone to do that at that time?”

    Measure Thoughtfully

    Tell me how you’ll measure me and I’ll tell you how I’ll behave. Gaming metrics are fairly easy to do, especially when an employee is incentivized to hit an unrealistic target or avoid embarrassment. Take activity metrics, for example (lines of code, number of bugs): These metrics focus on keeping people busy. But busyness does not equal business value delivered. People can be 100 percent utilized sprinting from meeting to meeting while making scant progress on business goals.

    Instead, consider metrics that reveal progress toward important business objectives or ones that help you make better business decisions. Flow metrics such as flow load, flow time, flow efficiency, and flow velocity reveal trends that measure four key areas: workload, speed, waste and throughput (respectively). These metrics are helpful in numerous ways, such as:

    • When others imply that things take too long, teams work inefficiently or people aren’t fully utilized, it’s useful to test their opinion against the data.
    • Having a balanced set of metrics allows you to see how changes in one area impact other areas. If the amount of work-in-progress (flow load) goes up, things will take longer to do (because of context switching and conflicting priorities), which results in slower speeds (longer flow times).

    Think about the behaviors that occur in your organization by how metrics incentivize people. What we measure impacts people because people value what is measured. Think twice before automatically defaulting to free and easy metrics available in your existing toolset.

    Build Social Capital

    Ideas start out flawed and incomplete. High levels of social capital, debate and discussion are the means by which ideas flourish. Social capital is what enables people to collaborate more effectively. The absence of social capital makes it impossible for people to speak openly (due to the lack of trust), resulting in withheld feedback and invisible work.

    Not much happens without discussion. When we engage in debate, our capacity to see each other’s perspectives is realized. TED talk speaker Margaret Heffernan’s insight on social capital is compelling: “Time compounds social capital.” The longer teams work together, the more trust, helpfulness and candor they accrue. Longstanding teams, therefore, benefit from increased productivity and reduced risk.

    Unlike the costly and time-consuming transitions observed in project-managed work, where people disband after dumping their work on the operations team before starting another project, organizing teams around a product allows the people who developed, tested and delivered the functionality to not only stay in their area of expertise, but also grow and gain from social capital.

    — Dominica DeGrandis

  • There is No Magic Recipe to Adopting DevOps

    There is No Magic Recipe to Adopting DevOps

    Over the past five years, we have seen more and more organizations of every size trying to adopt DevOps practices. However, we’re still expecting adoption rates to continue to rise—we have not yet reached “peak DevOps.” The motivations behind this comes from the need to reduce time to market, deliver better quality software, reduce operational costs and, ultimately, improve productivity and stay competitive.

    More enterprises are changing their business models by digitizing their business processes. We’re also seeing increased adoption of cloud technologies, agile frameworks and disruptive technologies. But these factors will not lead to an accelerated software development life cycle if organizations don’t prioritize better collaboration between people (no matter what area they belong to).

    Successfully implementing DevOps practices is not easy. Depending on the company’s size, it can take several months or even years. Most of these enterprises are learning what it takes to transform their organization. From our experience it takes several elements, but most importantly, engaged leadership is one of the most essential aspects for successful DevOps transformations. This leadership support will help organizations create dynamic and flexible business processes that eliminate traditional silos. This means that multiple teams can work together in a collaborative way, making seamless workflows that accelerate the software development life cycle and improve ongoing product maintenance. Such workflows bring tangible benefits, such as being able to rapidly respond to changing customer demands and developing better quality software, faster.

    Implementing DevOps is Unique to Every Company

    However, when talking about what it takes to adopt DevOps, we have to understand that there is no magic recipe that works for all. Every company has its own unique history, culture, and way of working, and thus will implement it differently. This is particularly the case because DevOps is really an amalgamation of different factors, from technology and automation to creating lean work processes and creating a culture of sharing between teams and individuals that traditionally were siloed.

    In 2019, Organizations Will Look to Scaling Initial Initiatives

    Very similar to what we have seen with the adoption of Agile development, most organizations have experimented with DevOps or have launched a small-scale pilot. They are aware of DevOps and the benefits it can bring. The challenge going into 2019, for the vast majority of businesses, is how to scale these pilots enterprise-wide. For example, in its 2017 survey, Forrester Research found that while just 13 percent of organizations had fully implemented DevOps, 50 percent had started their implementation and were expanding their use.

    We’re very much still on the journey to implementing DevOps practices. For example, many of the organizations that we work with are now using automation tools to get closer to continuous integration and continuous delivery capabilities, but there is still potential to use automation to a greater extent.

    There also exists a split in the types of applications that organizations are currently using DevOps for. Again, an interesting study by Forrester Research showed that in 2017, while 68 percent of organizations stated they were already using such practices for building and deploying mobile applications, this percentage dropped to 56 percent for ecommerce websites. I also believe that organizations need to improve how they measure DevOps success: Can they create metrics, such as level of level of innovation, or tie success directly to product revenue or process efficiency?

    As we look forward to 2019, more than ever business leaders will find value in looking at the best practices that companies have already developed on their DevOps journey. But we can’t forget that while two organizations may use the same automation tool, they will have very different cultures and way of working, with different goals and objectives. That’s why DevOps takes so long to implement, and represents a fundamental shift in how most organizations create, deliver and maintain software.

    — Gabriel Vasquez

  • Why the DevOps Trinity is Key for Digital Transformation

    Why the DevOps Trinity is Key for Digital Transformation

    As digital technology continues to disrupt and transform businesses across industries and around the world, the ability to deliver high-quality software rapidly will make the difference between survival and extinction for many companies. DevOps seeks to change the dynamics in which operations and development teams interact. Ultimately, successful adoption of DevOps and the process of continuous delivery are likely to determine whether an organization thrives or fails in the digital age.

    To build a sustainable DevOps culture and achieve true DevOps success, organizations need to consider the DevOps Trinity—people, processes and tools—and understand how to connect all three:

    People and Culture

    The core tenet of DevOps is a culture of collaboration and shared goals, which eliminates finger-pointing and promotes teamwork, with everyone working together in support of the same outcome—reliable software quickly delivered to production. In essence, the goal is to enable the different stakeholders in the software development and delivery process to reach the mutual understanding and empathy needed to drive alignment on shared goals. To achieve this change, teams need to have trust, honesty and responsibility.

    Process and Practices

    DevOps aims to achieve predictable, efficient, secure and sustainable operational processes. Using a DevOps approach, simple processes become increasingly programmable and dynamic. Automation supports this objective by using technology to streamline the steps within the process.

    Tools and Technologies

    Some categories are more essential in a DevOps toolchain than others, especially continuous integration (build) and infrastructure as code (configure) tools. The common elements of the toolchain include applications for coding, building, testing, packaging, releasing, configuring and monitoring. Often, rather than a single tool, there are sets of tools (or toolchains) of multiple tools, that make up the process.

    A sustainable DevOps organization cannot be achieved without all three components of the DevOps Trinity.

    For those looking to learn more about people, processes and technology needed for successful DevOps implementation, join us in San Francisco Sept. 16-19 for DevOps World | Jenkins World. With this year’s conference theme “TRANSFORM,” attendees will learn how organizations are transforming software delivery, adopting DevOps and returning business value to their organization from DevOps initiatives.

    Don’t miss Viktor at DevOps World | Jenkins World! Be sure to register today for 20 percent off your registration using the special DevOps.com code just for our readers: JWDEVOPSCUST

    — Viktor Farcic

  • WhiteHat Security Looks to AI to Advance DevSecOps

    WhiteHat Security Looks to AI to Advance DevSecOps

    WhiteHat Security is now embedding artificial intelligence (AI) into the application security review services it provides, as part of an effort to advance adoption of DevSecOps processes.

    Joseph Feiman, chief strategy officer for WhiteHat Security, said the biggest resistance factor to DevSecOps has been the amount of time it takes to test applications for vulnerabilities. By injecting machine learning into that process via a suite of WhiteHat Sentinel Dynamic tools, which are being made accessible as a software-as-a-service (SaaS) application, developers will be more inclined to conduct security reviews as part of an integrated DevSecOps process, he said.

    The AI engine identifies vulnerability patterns in a code-based data lake of 95 million identified vulnerabilities that WhiteHat Security has identified via its dynamic application testing service (DAST). WhiteHat Sentinel Dynamic identifies known vulnerabilities in seconds, Feiman said, and any anomalies indicative of potential unknown vulnerabilities are then reviewed by human WhiteHat Security cybersecurity experts.

    Feiman said DevSecOps is going through a period of disillusionment because security reviews of code are taking too long. AI technologies represent an opportunity to reignite interest in DevSecOps by giving developers more control over an application security review process that no longer slows down development.

    In addition, WhiteHat Sentinel Dynamic enables developers to learn about their potential mistakes without necessarily requiring them to disclose them to the entire DevOps team. Any humiliation a developer might experience is avoided when the developer conducts the security review prior to committing code. IT security teams are still in charge of creating the policies that need to be implemented, but processes associated with implementing those policies make a decidedly shift left to the developer, he said.

    Feiman noted that the rise of AI within DevSecOps processes may help alleviate some of the current chronic shortage of cybersecurity expertise. There will always be a need for human cybersecurity researchers to train AI models what to look for as new vulnerabilities are discovered. But algorithms never take a day off or decide to abruptly quit because they suddenly got a better offer.

    It may take a while for IT organizations to completely trust AI engines to scan code for vulnerabilities. But currently many applications are not being reviewed at all because any time set aside to conduct that reviews is being eaten up by an application development process that has fallen behind schedule. Too many developers would still prefer to take a chance on a breach than be reprimanded for delivering code late, especially when there’s a chance to address vulnerabilities that might get discovered in the next scheduled release of a patch. WhiteHat Security is making the case for eliminating much of that risk by making it feasible for developers to quietly address vulnerabilities long before an application is deployed in a production environment.

    There’s no doubt that AI soon will be employed to automate large swaths of cybersecurity. But it may turn out that the most effective place to apply AI from a cybersecurity perspective early in the DevOps process as possible.

    — Mike Vizard

  • Education in the Age of Automation

    Education in the Age of Automation

    Automation is changing the nature of education and the ways people learn. More schools are using online learning to supplement in-class activities. In fact, some schools are completely online. Distributed computing on the internet and the automation that drives it makes this possible.

    Everything changes when the formal education process becomes embedded in the internet. Courseware delivery becomes asynchronous; people learn what they want, when they want to. Test administration becomes more about the efficient acquisition of data and evaluation thereof. Human-to-human interaction as a basis for a meaningful educational experience becomes optional. Today, it’s entirely possible to learn all you need to know to make a good living in the modern world without ever needing to sit in a classroom or interact with a human directly. Let’s face it: You can do a lot of learning from YouTube.

    As a result, more people are beginning to question the conventional wisdom of spending four years and thousands of dollars to get a bachelor’s degree—particularly now that companies including IBM, Google and Apple no longer require one to get hired.

    Coincidentally, as companies eliminate the college degree requirement for new hires, we’re seeing significant growth in technical bootcamps. One study reports that this year, 20,000 students will complete a course of study at a coding boot camp and be “job ready.” Considering that the number of students who graduated with a computer science degree from a typical college in 2017 numbered about 93,000, there’s a good case to be made that boot camps are siphoning off a number of students that otherwise would be headed to academia.

    Going to boot camp instead of college is an upward trend: The study projects that the number of boot camps are expected to grow 20 percent this year. The career path for a technical professional does not necessarily involve going college. Going to a boot camp for a year—if not less—will suffice.

    Is this a problem? I’m not sure. On the plus side, I am reminded of the scene in the film, “Good Will Hunting,” when Matt Damon’s character Will Hunting, a genius with a photographic memory who walked the halls of MIT working as a janitor, makes the profound remark:

    “… you dropped a hundred and fifty grand on a f****** education you coulda got for a dollar fifty in late charges at the public library.”

    There’s a good argument to be made that when it comes to getting a good paying job, the $15,000 median price tag for attending a coding boot camp has a better much better ROI than spending  ~$200,000, the cost of a four-year degree from MIT.

    But then, I am reminded of Aldous Huxley.

    In his book, “Brave New World,” people are grown in test tubes according to a specific formulas, the result of which is a biological caste system. Those in the Epsilon caste are grown to be 3 feet tall and abhor the outdoors. Their job is to work their way underneath the machines on the factory floor to do simple maintenance. Deltas, the next caste up in the hierarchy, do the menial chores—janitors, for example. Gammas are the bookkeepers and data entry clerks. Then you get to the higher levels, Alphas and Beta. Betas are technicians who do very sophisticated jobs, such as mixing dangerous chemical together. One mishap can result in an explosion, killing all. Finally, at the top of hierarchy are the Alphas. Alphas possess advanced analytical skills and a broad knowledge base. They are able to plan and strategize. They have a high degree of creativity. Their role is to run the society.

    Thus, I have a concern about the increasing role of technology and  automation education, and the emergence of tech-focused boot camps as the prime post-secondary educational experience for many.

    As more of those entering the workforce forgo a formal college education, we run the risk of creating a society in which most of the technical workforce is Betas. They’re able to do highly complex tasks, yet have no idea why they’re doing them.

    Let me be specific. Let’s talk about Kubernetes: It takes a lot of knowledge and experience to be good at getting Kubernetes to work. It’s a complex, evolving technology that you need to keep your eye on. Companies are spending a good deal of money making sure their IT staff are competent with Kubernetes—as well they should be. One little misstep can result in the digital equivalent of a chemical explosion. Yet, I wonder how many of those “working Kubernetes” are thinking about the bigger picture—for example, the purpose and value of the enterprise using Kubernetes. Are these engineers questioning if the technology they’re working is doing good or harm?

    Or, let’s turn the dials down a bit and ask a question that’s less loaded philosophically. How many of those working Kubernetes today wonder if an alternative orchestration technology such as Docker Swarm might be a better choice to meet the need at hand. Or are they so concerned with getting Kubernetes to run that they don’t have the bandwidth to think about anything else? Do they just defer such critical thinking to “the system architects”?

    Critical thinking, although an acquired skill, is not one that’s new on the cognitive landscape. It’s been taught for thousands of years, since the time Socrates sat under an olive tree asking simple questions that forced his students to transform their assumptions and allusions into thoughts based on reason and reflection.

    Developing the ability to think critically traditionally has been one the aims of a college education. The romantic ideal is that college is the place where students are exposed to a variety of ideas from a variety of viewpoints. The hope is that such exposure, along with instruction aimed at developing one’s ability to engage in higher-order thinking, will create graduates who will contribute to making a better-functioning, more prosperous and more humane society.

    But, that’s the traditional perception of college. The reality is changing. Maybe college is just the place you go to get the skills required to get a good job. If that’s the case, given that a college education is becoming less affordable to more people, maybe the boot camp approach is best. Yet, there is a risk.

    A good while back when I was considering enrolling in a doctoral program, I attended a required orientation at Harvard University. I sat in an auditorium composed of about 50 aspiring doctoral students. One of the department heads on stage said the following, “The objective of our doctoral programs is simple. We want to train you to run the world.”

    Yep, that’s right. And, I can say with a good deal of confidence that this objective is the same today—to create thinkers who have the skills and knowledge necessary to run the world. The Alphas, if you will.

    So, we’re at a crossroads. Automation is a liberating force in education in that it makes infinite knowledge available to anyone who wants it at a fraction of a cost of a traditional college education. All that’s required is the time and desire to take advantage of the opportunity at hand. However, if the result of such education is a workforce in which the concern of most is to ensure that the technology works and does not blow up, then the risk we run is that bigger decisions will be made by those who had the time and money for an increasingly elite education. In other words, as more people spend time focused on turning the dials and setting the levels to make the technology go, a very small minority actually will be determining the purpose and direction of the machine the technology supports.

    Still, the fact is, most of us need to make a buck to survive. This means being good at getting the tech to work. Yet, we will do well to understand that there is a Big Picture and if don’t pay attention to it, someone else who has been educated to run the world will.

    — Bob Reselman

  • When DevOps and ITSM Collide: Mistakes to Avoid

    When DevOps and ITSM Collide: Mistakes to Avoid

    In many ways the DevOps movement evolved as a reaction against the worst elements of IT service management (ITSM). IT leaders wanted to get from under the yoke of overly complex processes, bureaucracy and red tape to speed up software delivery, and to just get stuff done. Now that DevOps has kicked into full gear at so many organizations, the question remains: What do we do about ITSM?

    While the instinct in some may be to completely ditch ITSM or to just let the ITSM team operate in a separate orbit from DevOps practitioners, experts say that would be a big mistake.

    “DevOps aims to deliver high-quality software to the business faster and in shorter cycles,” explained Krishna Mohan Reddy, vice president and deputy global head of Cognitive Business Operations at Tata Consultancy Services. “However, it still does not eliminate the need for controls.”

    In many ways ITSM is the yin to DevOps’ yang—the two operating philosophies do offer contrasting viewpoints but when the operate in a single system they can be extremely complementary and synergistic.

    “Maintaining the proper balance between the ITSM and DevOps helps the organizations to scale and grow and become more organized,” said Sachin Agarwal, DevOps engineer and solution architect for Mantra Labs. Operating at DevOps speeds with ITSM frameworks in place keeps finger-pointing at a minimum and reduces the risks of failed deployments. “Both in harmony will increase the efficiency and reduce the blame-game within the organization.”

    As organizations try to figure out how to achieve that harmony, here are some pitfalls they should seek to avoid:

    Letting Each Tribe’s Preconceived Notions Hold Back Overall IT Progress

    It can be easy to get wrapped up into dogmatic fervor that your particular philosophy is the One True Path for IT success, but that really does nobody any favor. Organizations that want to pick up the best of both ITSM and DevOps need to encourage stakeholders to approach the table with an open mind.

    “You need to compromise. A lot of the new, young practitioners in DevOps and Agile will pretty much automatically discount opinions coming from people who have got ITIL certifications,” said Andi Mann, chief technology advocate for Splunk. “And vice versa—a lot of traditional operations and development teams will look at the people in DevOps and Agile and think they’re all cowboys. But they all have something to learn from each other.”

    For DevOps practitioners, it’s important to keep in mind that ITSM controls such as service requests, change management or structured management of incidents and problems don’t have to stop a team from delivering quality software at speed, said Troy Vetter, COO of Coda Global.

    “In fact, speed and quality are actually helped by being closely monitored. In IT, just like in science, if it isn’t seen, measured, and “reproducible, it doesn’t really count,” he noted. “So, ITSM is actually the perfect complement to DevOps because it aids in the visibility, measurement and reproducibility of key DevOps processes.”

    Failing to Address the Elephant in the Room: Change Management

    Analyst Roy Atkinson agrees that ITSM has a lot to bring to the table for DevOps workflows. ITSM’s rich history in thinking about and tracking operational best practices offers tons of value in improving the customer experience with the software being delivered. But on the flip side, ITSM teams can’t continue to operate as they did in the past. They need to compromise and learn new ways of accommodating DevOps approaches—particularly in the area of change management.

    “Change management has often been where DevOps and ITSM collide,” said Atkinson, who analyzes the service management market for HDI. “The stereotypical Change Advisory Board must radically revise its approach to be in sync with the enormous amount of change that DevOps can produce in a day.”

    The good news is that thought leaders in the ITSM world have been thinking hard about this issue and have been evolving practices so they can embed change management controls into a DevOps framework without holding back continuous delivery.

    “There is some excellent recent thinking in the ITSM world about how change management can serve its intended purpose without being a roadblock by acting as an enabler of change and being involved all along the way, rather than a bolt-on quality check at the end of the line before the production environment,” Atkinson said.

    Maintaining Separate Toolchains

    Often times one of the big reasons ITSM will remain a roadblock for DevOps isn’t because the ITSM folks aren’t willing to collaborate, but simply because each team relies on completely separate toolchains.

    “You might have a tool that developers use for tracking bugs or product backlog and then there’s a completely separate IT service management tool,” said Donna Knapp, curriculum development manager for ITSM Academy and author of the DevOps Institute’s course on ITSM for DevOps. “So what you end up doing is forcing developers to use some process that’s outside of their norm in order to address issues.”

    As Vetter explained, a lot of the knee-jerk DevOps reaction against ITSM has to do with an aversion to the historical view of ITSM requiring manual inputs in monolithic ticketing systems. But the technology of ITSM controls has advanced—it just needs to be appropriately implemented.

    “Self-service requests, automation flows and IT orchestration are now our front door to actions, and IT service management can and should be happening in the background, automagically created, maintained and measured by our DevOps tools,” he said.

    — Ericka Chickowski

  • Black Box vs. White Box Monitoring: What You Need To Know

    Black Box vs. White Box Monitoring: What You Need To Know

    The word monitoring possesses a meaning most people are familiar with. But what does monitoring actually break down into if you’re monitoring servers, applications and hardware?

    One way of answering this question is in terms of black box and white box monitoring. The latter term refers to application monitoring, and the former to server or hardware monitoring.

    In this article, I’ll explain white box vs. black box monitoring in more detail.

    What is White Box Monitoring?

    White box monitoring is the monitoring of applications running on a server. This could be anything from the number of HTTP requests your web server is getting to the response codes generated by your application. Other types of white box monitoring include:

    • Monitoring MySQL queries running on a database server.
    • Looking at the number of users utilizing a web application throughout the day, and alerting if this goes above a predefined threshold.
    • Considering the above example of HTTP requests—splitting these out into monitoring the different kinds to ascertain how the application is performing, or whether users are getting served the correct content. For example, a 403 would demonstrate a user has tried to get to a part of the website they’re not allowed to visit. Likewise, a 200 would indicate their request was successful and they were served the content.
    • Performing advanced detection of behavior we don’t expect to see, such as a user not going through the normal steps you’d expect when signing into your application or resetting a password.

    What is Black Box Monitoring?

    Black box monitoring refers to the monitoring of servers with a focus on areas such as disk space, CPU usage, memory usage, load averages, etc. These are what most in the industry would deem as the standard system metrics to monitor. Other types of black box monitoring include:

    • Monitoring of network switches and other networking devices such as load balancers from the system metrics perspective, as defined above.
    • Looking at hypervisor-level resource usage for all virtual machines running on the hypervisor (such as VMware, KVM, Xen, etc.).
    • Alerting on hard disk errors that may present a problem if a disk isn’t replaced soon (using SMART, for instance).

    Differences Between Black Box and White Box Monitoring

    There are differences between these two types of monitoring. Traditionally, systems administrators would take care of both white and black box monitoring; however, with the advent of DevOps and modern changes in the IT industry, we’re increasingly finding that application developers are taking responsibility for the monitoring of the applications (white box) they’re writing and as a result, are building monitoring solutions or writing checks for monitoring systems deployed by DevOps engineers.

    Systems administrators and DevOps engineers tend to take responsibility for the monitoring of black box items such as servers. There is some crossover where DevOps engineers can also take responsibility for white box monitoring, but this depends on the business or environment you’re working in.

    White Box vs. Black Box Monitoring: Which is More Important?

    It’s important to realize the significance of both types of monitoring. Historically, there was a gap in monitoring applications, and this presented lots of problems because black box monitoring would pick up issues with systems, such as high load or high network traffic, but there would be no information on the application side to show why this was happening—and more often than not these issues are caused by application issues, not server issues.

    A good example of white box monitoring and its importance can be demonstrated with a black box monitoring check alert, and showing how the two work together. Take, for example, a black box monitoring alert that tells us that our server’s CPU usage is at 100 percent. We go on to investigate this issue and see that MySQL processes are the cause of this alert. If we have white box monitoring in place to also monitor the queries running in MySQL, the amount of connections into MySQL and the amount of time it is taking for queries to run, then we have a lot more information to help diagnose the issue. This could allow us to demonstrate that an application is running a query that is too resource-intensive or is badly designed, and provide feedback to the application teams with hard evidence.

    Conclusion

    There have been many changes to the way in which we implement monitoring systems over the last few years. We’ve moved away from implementing purely black box solutions to implementing white box solutions alongside them. And there are new solutions available that allow both types of monitoring to be performed from within one single application.

    This is incredibly important, as it allows us to have more synergy between our development and systems teams, plan properly for capacity enhancements and deal with issues that may arise. As a DevOps or systems engineer in the modern IT industry, I think it’s vital to take these different monitoring methodologies into account when building out any support/monitoring solution for your team.

    — Keith Rogers

  • Revisiting My Bet on DevOps

    Revisiting My Bet on DevOps

    On Dec. 29, 2015, I bought some stock. It was a bet on DevOps that my wife kindly tolerated. Since then, the S&P 500 has risen 31 percent and my DevOps portfolio is up 113 percent. This blog is a retrospective on a bet that worked out.

    I was inspired by Nicole Forsgren’s talks and the “State of DevOps Report,” whose data showed better profitability, market share and stock price performance among companies claiming to do DevOps well. DevOps, after all, exists to help the business win, and as digital transformation sweeps industry after industry, technologists increasingly are in a position to make that difference.

    My process has been very, very simple: Companies whose talks at DevOps conferences convincingly convey a passion for DevOps are ones that I would consider buying. Would I do deep research on each one? Nah—whatever I could research aside from DevOps would already be priced in by people who understood that content better than me.

    At the one-year mark, I summarized the results. The crazy project was up 12 percent against an S&P that was up 8 percent—a nice little win. The first six months of 2018 have been tremendous, though: The S&P is up almost 2 percent and the portfolio is up 42 percent.

    A key disclaimer is that I am trying to read corporate culture from the outside. For insiders, I am sure I will have some of this comically wrong.

    Remixing the Strategy (and Portfolio)

    Over the past year and half, I’ve made a handful of trades based mostly on two triggers.

    The first trigger is a signal that a company is losing its commitment to DevOps. I did this with Nordstrom: Its DevOps leaders all moved on to different companies. For example, Courtney Kissler went to Starbucks and Suzanne Conniff was off to T-Mobile. About the same time at a meetup, a Nordstrom employee told me that DevOps had fizzled. I was out. In the last year, Nordstrom is up 7.5 percent while retailers I kept are up: Target 46 percent and Amazon 74 percent. That said, Target is a bit of a difficult case.

    The second trigger is silos. I worry about an either IT-centric view of DevOps or DevOps in just a small slice of the business. Before I invest in a new stock, I want to know the transformation is taking root enterprise-wide.

    Barclays for Bank of America is an example of chasing the all-in company that didn’t work out: negative 8 percent growth versus 25 percent at BoA. Overall though, shifting toward all-in and unicorns has been solid.

    Who’s in Today

    • Amazon: 121 percent
    • Barclays: -15 percent
    • Capital One: 26 percent
    • Cisco Systems: 52 percent
    • Etsy: 340 percent
    • Live Nation: 93 percent
    • Netflix: 235 percent
    • Nike: 20 percent (since December 2017)
    • Target: 0 percent

    Amazon: Amazon’s net gain at 121 percent looks a lot like the portfolio as a whole (113 percent). The shares I purchased in 2015 are up 150 percent. The company is a natural fit, given that its CEO wrote the “two pizza rule” and demands microservices. It is also a hedge, as the company directly competes with much of this list including Target, Netflix and Etsy. With cloud and DevOps often joined at the hip, Amazon’s cloud revenue is also a rough proxy for DevOps growing overall.

    Barclays: Did I invest in a British bank shortly after the Brexit vote? Yeah. It was a bit of a test of the “Everything except DevOps is already priced in” rule. This one hasn’t panned out, but from the outside Barclays is among the banks doing best at DevOps.

    Capital One: Has performed about average among banks that I looked at this morning. I adore its start-up attitude and how the company goes about getting talent and contributing to the community.

    Cisco Systems: Cisco has had some great talks and is undergoing a digital transformation of sorts of its own. The company is doing fine.

    Etsy: Holy cow, Etsy. Etsy is an interesting case in that it has been a DevOps darling for many years. With the stock flagging a bit, the CEO and the CTO (our DevOps hero, John Allspaw) were shown the door and the perception was the grown-up business types were taking over. I increased my bet based on the thought that technical foundations pointed ruthlessly at business problems might work out. When the stock price doubled, I cashed out my original investment.

    I’d be better off had I not sold any, but with as volatile as Etsy is, I was happy to route some of the winnings over to a bet on Nike.

    Live Nation: I saw a great talk from a Live Nation guy at DevOpsDays Rockies, and the company’s stock has paid off nicely since, nearly doubling. Live Nation gets it, and shows a passion for satisfying customers. Being Ticketmaster, it has had a risk of digital disruption knocking it down for years and has built up a strong innovation capacity—exactly what incumbents need to do.

    Netflix: DevOps skills? Check. Good at using data to delight customers? Check. Worth twice what I bought them for? Check. I wrote most of this analysis in early July. Since then, Netflix has had a bad earnings call. It’s still a massive winner so far.

    Nike: I loved Nike’s talk at DOES 2017. But what really struck me was DOES London. In the morning the company spoke about a one-team culture and engaging digitally—good stuff. In the evening a colleague complained his kid was spending $200 on sneakers. A week later, Nike’s stock spiked 10 percent. Yup.

    Target: Oh, Target. Target has done some great DevOps work, but word on the street is that too much of what it did at first lacked a connection back to the business. However, since the security fiasco with stolen credit cards, that has turned around. While its stock is right around where I bought it 2.5 years ago, the last year has shown strong momentum: Target is up 45 percent in the last 12 months.

    2018 Has Been Great

    Over the project’s first two years, I had a nagging worry: I was beating the market, but was I just picking riskier stocks and being rewarded in a bull market? So far the market has been flat in 2018, showing modest 1.7 percent growth through June 30. The DevOps Portfolio, however, is up 42.2 percent.

    Only the banks are trailing the S&P.

    So What Have We Learned?

    • It’s about customers and the business: There appears to be a moment of DevOps backlash when alignment to business outcomes isn’t quite strong enough. When the DevOps culture is strong but redirected back to catering to the business, things can go better. That’s one way to read Target and Etsy’s recent performance. When the transformation fizzles … see you, Nordstrom.
    • Banks may be different: This system really hasn’t worked out terribly well for me in the banking space. I suspect that is because most banks are undergoing similar transformations and may not be as able to differentiate as retailers do. At DOES London we heard similar stories from RBS and Barclays while Lloyd’s is winning DevOps awards at other conferences. I also lacked representation from the digital disruptors—PayPal would have been a good balance to CapOne or Barclays.
    • DORA should be a hedge fund: I hope they have a secret one.

    I’m thankful my that my wife was on board with this little project and I’m very happy that we can take our winnings here as we plot our second honeymoon.

    Of course, professionally I’ve been betting on DevOps a long time, working on UrbanCode continuous delivery tools the last 15 years. That worked out with an acquisition by IBM as it beefed up its DevOps solutions portfolio.

    — Eric Minick

  • OpenStack and Open Source MANO: Technologies for NFV Deployment

    OpenStack and Open Source MANO: Technologies for NFV Deployment

    We have experienced how open source software technologies revolutionized the application development process, which ultimately resulted in digital transformation across various industry verticals. Open source technologies now are disrupting the telecom sector for building 5G internet, which will be powered by network functions virtualization (NFV) and software defined networking (SDN). With NFV and SDN, multiple network functions and control and management operations in telecom networks will be software-driven; enabling the cloud-native and DevOps approach.

    There are various communities emerged to innovate the software stack behind NFV and SDN technologies such as OpenStack, ONAP, ETSI Open Source MANO (OSM), OPNFV, OpenDaylight, Open Networking Foundation (ONF), Open Contrail, Open Baron, On.Lab, CORD, Kubernetes, ONOS and Open-O. In this article, we will take a look at how a combination of data center management platform management platform, OpenStack and ETSI Open Source MANO (OSM) will be useful for NFV deployment.

    OpenStack for NFV

    As we know, OpenStack is mainly known to be the largest pool of open source projects that collectively form the software platform for cloud computing infrastructure. This infrastructure is used widely in private cloud use cases by many enterprises. After an introduction of NFV by ETSI, OpenStack has emerged as a key infrastructure platform for NFV. In most of the NFV deployments, OpenStack is used at the VIM (virtual infrastructure manager) layer to give a standardized interface for managing, monitoring and assessing all resources within the NFV infrastructure.

    Various OpenStack projects (including Tacker, Neutron, Nova, Astara, Congress, Mistral and Senlin) are capable of managing virtualized infrastructure components of NFV environments. As an example, Tacker is utilized to build a generic VNF manager (VNFM) and NFV Orchestrator (NFVO), which help in the deployment and operation of VNFs within NFV infrastructure. Additionally, integrating OpenStack projects introduces various features to NFV infrastructure, including performance features such as huge pages, CPU Pinning, NUMA topology and SR-IOV; service function chaining, networking slicing, scalability, high availability, resiliency and multisite enablement.

    Telecom service providers and enterprises that have implemented their NFV environment with OpenStack include AT&T, China Mobile, SK Telecom, Ericsson, Deutsche Telekom, Comcast and Bloomberg.

    Open Source Mano (OSM) for NFV

    The management and orchestration (MANO) layer is responsible for orchestration and complete lifecycle management of hardware resources and virtual network functions (VNFs). In other words, the MANO layer coordinates NFV infrastructure (NFVI) resources and maps them efficiently to various VNFs. There are various options available as three-dimensional software stacks for MANO. But ETSI-hosted OSM is largely preferred due to large activity at the community level as well as its highly mature framework, production readiness, ease of initiation and constant feeding of use cases by members.

    Virtual network functions (VNFs) forms network services and may need updates for feature addition or patch for functionalities. OSM provides a method to invoke the VNF upgrade operation with minimal impact in the running network service.

    With the continuous community support and involvement for feature innovation, OSM has now evolved to bring a continuous integration and continuous delivery (CI/CD) framework at MANO layer.

    The latest release 4 of OSM has brought a large set of features and enhancements to the OSM framework, impacting functionality, user experience and maturity, which enables various enhancements for NFV MANO from usability and interoperability perspective.

    OSM has steadily adopted the cloud-native principles and can be easily deployed in the cloud, as installation is container-based and runs with the help of container orchestration engine. A new northbound interface aligns with ETSI NFV specification SOL005 and provides dedicated control of the OSM system. Monitoring and closed‐loop capabilities have also been enhanced.

    The next version of OSM release 5 is expected to be launched in November and will be bundled with more 5G-related features such as network slicing and container-based VNFs.

    Why OpenStack and Open Source MANO for MANO layer in NFV?

    Both OpenStack and OSM have a large community known for its rapid pace for innovating NFV and higher contribution by all scale companies to enhance current features and develop new capabilities for core projects under it.

    For NFV, OpenStack standardizes interfaces between NFV elements and infrastructure. OpenStack is used for commercial solutions offerings by companies including Canonical/Ubuntu, Cisco Systems, Ericsson, Huawei, IBM, Juniper, Mirantis, Red Hat, SUSE, VMware and Wind River. A large percentage of VIM deployments are based on OpenStack due to the simplicity in handling and operating various projects targeted toward providing full potential storage, compute and networking for NFV.

    With last two releases (3 & 4), OSM has evolved to support integration for cloud-native approach by enabling CI/CD frameworks into orchestration layers. Cloud readiness involvement of OSM is the key benefit along with OpenStack, which has proven architecture for private as well as public clouds. OSM deployment into NFV infrastructure has become very lean, where one can start with importing docker containers into production. On the other hand, OpenStack is known for enabling simplicity to manage virtualized and containerized infrastructure. Organizations can realize the full benefits out of integration as NFV MANO using OSM and OpenStack due to lean and simple management and deployment.

    Conclusion

    Since the introduction of NFV and SDN, telecom service providers and vendors were evaluating the use case offered for telecom networks, and due to this application of both, were in under maturity stage. Service providers and vendors were concerned about dynamic orchestration and automation of infrastructure deployment and constantly upgrading virtual network functions (VNFs). But due to contribution by open source communities such as OpenStack and Open Source MANO (OSM), they are now at the stage to apply cloud-native principles to 5G network implementation and operations. Furthermore, with contributions by communities to OpenStack and OSM projects, we will see more enhancement to 5G network enhancement technologies including network slicing, mobile edge computing (MEC) and cloud radio access network (cRAN), which will benefit emerging technologies including IoT, AI/machine learning, real-time analytics, augmented reality (AR) and virtual reality (VR) with low latency and high-bandwidth features.

    — Sagar Nangare

  • Threat Modeling: The Why, How, When and Which Tools

    Threat Modeling: The Why, How, When and Which Tools

    You may have heard of threat modeling as a structured activity for identifying and managing threats. Threat modeling can be applied to a wide range of things, including software, applications, systems, networks, IOTs and business processes.

    Threat modeling’s motto should be, “The earlier the better, but not too late and never ignore.” Without threat modeling, your security is a gamble—and in today’s business environment,  you’re sure to lose.

    When you design an application, you will face several security issues during different phases of the software development life cycle (SDLC), and so having threat modeling in the SDLC from the beginning can help to safeguard that applications are being developed, with security built in.

    Simply put, threat modeling is a procedure to identify threats and vulnerabilities in the earliest stage of the development life cycle to identify gaps and mitigate risk, which guarantees that a secure application is being built, saving both revenue and time.

    Why Threat Modeling?

    • It is better to find security flaws when there is time to fix them.
    • It can save time, revenue and the reputation of your company.
    • To build a secure application.
    • To bridge the gap between developers and security.
    • It provides a document of all the identified threats and rated threats.
    • It offers knowledge and awareness of the latest risks and vulnerabilities.

    How To Do Threat Modeling

    Many people think only security engineers can do threat modeling. That’s not true. Anyone, from developer to software project manager, can threat-model. In fact, I would suggest they should also know a little bit of threat modeling as part of their work.

    Let’s look at the elements of threat modeling:

    Assets: What valuable data and equipment should be secured?

    Threats: What the attacker can do to the system?

    Vulnerabilities: What are the flaws in the system that can allow an attacker to realize a threat?

    Steps to Threat Modeling

    Step 1: Identify the assets (database server, file servers, data lake stores, Active Directory, REST calls, configuration screens, Azure portal, authenticated and anonymous web user, Azure AAD client apps, database users, DB administrators)

    Step 2: Outline details of architecture on which the valuable asset is being processed. It may include the software framework, version and other architectural details (ASP.net web application connection to cloud data stores and third-party services using JWT tokens).

    Step 3: Break down the application regarding its process, including all the sub-processes that are running the application. We create a data flow diagram (DFD).

    Step 4: List identify threats in a descriptive way to review to process further.

    Step 5: Classify the threats with parallel instances so that threats can be identified in the application in a structured and repeatable manner.

    Step 6: Rate the severity of the threat.

    Threat Modelling Methodologies

    STRIDE (Uses application-centric approach)

    PASTA (risk-centric approach): Process for Attack Simulation and Threat Analysis

    TRIKE (risk-based approach with unique implementation and risk-modeling process)

    VAST (Visual, Agile and Simple Threat modelling)

    OCTAVE (focused on assessing organizational (non-technical) risks that may result from breached information assets.)

    When to use Threat Modeling

    In simple words, at the early stages of the SDLC, perform threat modeling:

    • Every time there is a change in the system’s architecture.
    • After a security incident has occurred or new vulnerabilities are introduced.
    • As soon as the architecture is ready.

    It is not a mandate to perform threat modeling at the early stages of the SDLC; you can still pick up threat modeling at any stages even if it is close to deployment.

    Threat models are continuously changing, and the models you prepared today may not be efficient tomorrow. And it is difficult to say that you are covered from all the threats. But if you have performed threat modeling and done whatever it takes to minimize your exposure to security risks, at least the impact of something very bad happening will be manageable (again, hopefully, but not a guarantee).

    Tools to Perform Threat Modeling

    I have used two tools for threat modeling, both of which are free to use.

    1. OWASP Threat Dragon
    2. Microsoft Threat Modelling Tool 2016.

    Let’s take a look at both the tools:

     

    Microsoft Threat Modelling Tool 2016         OWASP Threat Dragon

     

    Full version available for free (as of now) Alpha version available, flaws are still there. It is an OWASP incubator project, so it is at its early stage.
    Installable desktop application available. The tool comes in two variants: an online web application for GitHub and second an installable desktop application.
    Documentation easily available on Microsoft website. Default threat template is available. You have to search, default threat template is present
    I found the UX better, unique methodology available, the user can understand threats better. Guidance and feedback in drawing a model. Good UX. Fun and engaging user experience.
    Presently using this tool for threat modelling. Maybe I will use. I think will continue to improve.
    End-user documentation is available. End-user documentation is also available. Automated threats.
    Threat Grid, Template editor, Mitigating existing DFDs. Mitigating existing DFDs. Powerful threat mitigation. Rules have not yet been coded as it is the Alpha stage.
    Comes with a base set of threat definitions using STRIDE categories. Even in its current stage (alpha), you can create threats from those STRIDE categories.

     

    — Debarghya Pandit