One of the key tenants of the DevOps principle is that your environment has to be prepared for anything. If your shiny modern delivery chain does not support the adoption of new DevOps tools, or does not help you learn, and iterate, then it will ultimately fall into the same category of solutions as Waterfall, and will be on a clear path to Legacy.
Category: Blogs
-
Making the Case for Secure, Defect-Tested Software Development
Creating a software security initiative in any organization is no easy feat. Often times, organizational culture or politics can provide development managers with a strong counterargument for implementing software security concepts. Unfortunately, building software without a consideration for security has become a less viable option given the increase in compliance pressures and widely publicized data breaches leading software consumers to expect more security from developers.
There is no way around the fact that attacks have become increasingly more sophisticated and applications have become the central avenue of these attacks. As a result, organizations are increasingly confronted with the task of assuring secure, defect free, software development. Leaders within these organizations are finally coming to realize that vulnerable and defect rampant software undermines the productivity, privacy and security of their businesses, employees and consumers alike.
As organizations are beginning to accept the fact that there is a need to test for best practices and defects throughout the development lifecycle, it is important that they are provided with the resources to make this happen. To change the way large organizations build software, an enterprise-wide initiative is required. At its core, an initiative to change the way an organization tests and validates its development processes and practices are a fundamental business process improvement effort.
Typically, large organizations with unique operational requirements find themselves building custom software systems to address their specific needs. But building custom software on time, on budget, and without bugs is difficult. Building software that also complies with an organization’s software security policies presents an even more difficult challenge. Although the tools and practices to build secure software are maturing, most organizations find internal hurdles to be more daunting. Organizational impediments including culture, differing software development approaches, and short-term business drivers make it more difficult to effect meaningful change. In order for a software security initiative to be successful, organizations must take a phased approach that considers organizational culture at each step.
Sure, there is the option of treating the symptoms by deploying web application firewalls or running an automated scanner against applications, but this does not get to the root cause and solve the problem for most organizations. It highlights that there must be deeper process improvements involving the systems development life cycle, particularly for higher level business logic or authorization vulnerabilities.
Despite the hurdles that exist when approaching the issue of secure development, most organizations do realize that there is a problem that needs to be addressed. Organizations focus on technical means to write more secure code and strategies for putting controls around the software. The next step is to then educate executives on the process of leading a software security initiative as these initiatives are most likely to fail due to organizational issues, not technical issues. This means taking a disciplined approach by characterizing the landscape, securing champions, defining standards and strategy, executing, and then sustain the effort. These steps, tailored to the way an organization operates, will help ensure that corporate-wide efforts to secure applications are as productive as possible.
So in conclusion, it is not an impossible feat to address security at the root cause, software development, as part of standard operating procedure. It just takes a little forward thinking to demonstrate its undeniable value and get the corporate buy-in needed to change the development culture. That being said, once the culture is changed don’t rest on your laurels. Always observe what’s occurring within your organization to determine whether there are new technology risk areas emerging. If you simply focus only on well trodden areas like web applications, you may find yourself behind the 8-ball once again when it comes to new advances in software development, like mobile and the Internet of Things.
-
Never Mind the Bollocks, Here’s the DevOps
What is DevOps?
“DevOps is a cultural and professional movement” –Adam Jacob
[movement (n): a group of people working together to advance their ideas.]
So, who are these people and what’s the big idea?
The people are system administrators, developers, product managers, QA technicians, security analysts, network engineers, and anyone else involved in the business of developing software.
The big idea is to increase the value your product or service delivers, more quickly with less effort.
(Having a product or service that is designed and built well isn’t inherently valuable. The value comes when your customer is able to use your product or service to solve a problem. DevOps is all about delivering value quickly and effortlessly.)
DevOps practices and technologies are evolving, but here are some of the core concepts paraphrased–be sure to read the links in the footnotes for the full explanations:
The Three Ways
- Systems Thinking
- Learning how everything is interconnected, from concept to customer
- Amplifying Feedback Loops
- Cutting the time between action and reaction, failure and success
- Culture of Continuous Experimentation and Learning
- Recognizing failure is important data about how and where to improve
CAMS
- Culture
- It’s about how people relate and work, not something you can buy
- Automation
- Faster, less error-prone processes help amplify feedback loops
- Measurement
- You can’t get where you want to be until you know where you are
- Sharing
- Effective teamwork ignores organizational boundaries
ICE
- Inclusivity
- Everyone is responsible for (and benefits from) these practices
- Complex Systems
- Because simple systems are simple
- Empathy
- Real collaboration requires real understanding of another’s point of view
That’s the theory from a very high level; the following Q&A provides some broad explanations of related concepts. These are actual questions I was asked in a recent discussion of DevOps.
Q: What specific skills or experience do I need before I can put “DevOps” on my resume?
A: It depends. At this point in time when recruiters and/or hiring managers list DevOps in a job posting they are almost always looking for a “DevOps Engineer” (i.e. a sysadmin who knows just enough coding to be dangerous), or a “full-stack developer” (i.e. a developer who knows just enough about system management to be dangerous). The reality is that (as of today) there is no DevOps certification, organization or even manifesto, so anyone who pretends to authority on what is and isn’t DevOps is overstating their case at best, a charlatan at worst.
Q: What is source control?
A: Source control is a repository for files that make up a modern software application during development. It maintains a historical record of changes so it’s easy to review (or rollback to) changes made to an application’s source files. The most popular source control technology today is Git, and GitHub is one of many online services used to manage Git repositories.
Q: What is Jenkins?
A: Jenkins is to task engines or “build servers” what Kleenex™ is to facial tissue. It’s currently the most widely used software product for automating various functions of software build and deployment. For example, a system might be configured such that any code change pushed to source control triggers a job in Jenkins to build, test and deploy that code.
Q: What is configuration management (e.g. Salt, Ansible, Puppet, Chef)?
A: Configuration management is a means of scripting server configurations so it’s easier to track, deploy or make configuration changes to large groups of systems.
Q: What are containers?
A: In the same way that server virtualization allows you to have many isolated “virtual machines” on a single physical machine, container technology allows you to have many isolated application environments on a single operating system. Containers have been around in Unix/Linux for a long time, they just boomed in popularity when Docker came along and made them easy to use. The next version of Microsoft Windows Server (2016) will introduce Windows containers.
Q: What are microservices?
A: Microservices are small, independent components of a software application that are built, tested and operated in isolation from the core product. Using microservices enables development to be faster and more nimble than possible when working with a large, monolithic application.
Q: What is Continuous Integration/Delivery/Deployment?
A: Continuous Integration is the practice of continuously integrating (and testing) new code with your existing source code. Continuous Delivery is the practice of automating as much of the build/test/release process as possible while still requiring a manual push to production. Continuous Deployment is the practice of fully automating build/test/release all the way into production.
Q: Are there any conferences or meetups I can attend to learn more?
A: Yes! The original DevOps conference is DevOpsDays and there are many of these events around the world annually. A popular conference with large enterprises is the DevOps Enterpise Summit. Various DevOps-related meetups (#CoffeeOps on Twitter) are springing up around the country. Check the CoffeeOps.org website for details, or search for DevOps at Meetup.com.
Q: [Do I need to understand everything above | Is this all I need] to be or do DevOps?
A: Absolutely not, on both counts. You can successfully implement a DevOps way of thinking/working by incorporating all or just some of the above, and there are many other tools and practices around delivering value more efficiently. Naturally there are extensive resources available (a few linked in the footnotes) to do a much deeper dive on all these subjects.
- Systems Thinking
-
Companies Failing to Innovate or Evolve Legacy IT Infrastructure
Biggest hurdles to innovation include lack of skills, no long term planning, poor funding and collaboration, survey finds.
We’ve been writing a lot about what it takes for IT organizations to transform their business-technology systems to drive agility and innovation. Interestingly, according to a recent study conducted by the Business Performance Innovation (BPI) Network, most IT professionals think their businesses are failing, or pretty close to failing, when it comes to driving forward with IT transformational efforts.
The report, Bringing Dexterity to IT Complexity: What’s Helping or Hindering IT Tech Professionals, found that those on the front-lines making their organizations networks, cloud deployments, data centers, and applications work believe that a serious lack of planning, skills gap, lack of budget, and poor communications and collaboration with business managers remain serious obstacles in their organizations to innovation and the IT transformation they need. This is despite most organizations, CEOs, CIOs, repeatedly citing the importance of technology in their efforts to innovate and compete.
The survey is based on responses from 200 IT workers in North and Latin America (38%), Europe (23%), the Middle East/Africa (20%) and Asia/Australia (18%). The survey was sponsored by IT infrastructure and services provider Dimension Data. However, IT workers say their organizations’ biggest weaknesses are long-term planning, application development, data analytics and software engineering.
The survey found that roughly 80 percent of IT respondents spend more than 50 percent of their time maintaining and troubleshooting older IT systems. Important findings from the survey include more than 80 percent of frontline IT workers say they spend over 50 percent of their time troubleshooting and maintaining legacy systems instead of driving innovation. And 17 percent say they spend 90 percent of their time on routine maintenance tasks.
“Instead of ushering their companies into a new age of highly agile innovation, IT workers are hindered by a growing list of maintenance tasks, staff cutbacks and aging infrastructure. All the while, growing tension between IT and their business colleagues has proven a roadblock to the type of communication that fosters collaboration, innovation and true IT transformation. Some 43 percent identified serious communication issues,” the report stated.
The survey’s major findings reveal:
- Only 35 percent of respondents rate their company’s ability to adapt to new transformative technologies as good or very good.
- Over 70 percent of IT workers report they have not even begun or are just “getting started” on the road to IT transformation.
- Just 15 percent have a clear and detailed plan for transformation. Over 80 percent say their plans provide only general direction, need updating or don’t exist at all.
- Almost 45 percent said improved collaboration between IT groups and business leaders is critically needed. Only 18 percent said there are active cross-functional teams in their companies today. Another 14 percent said they rarely speak with business managers or speak only out of necessity.
The survey also found that respondents believed (52 percent) the business managers who own a new initiative or application want too long too bring IT or developers into the process, and 48 percent say that business managers fail to properly fund their initiatives. The survey indicated that roughly half of respondents believe business managers think poorly of the IT team.
Also interestingly, about one-third of respondents reported that their organizations have yet to move to cloud computing. Only 13 percent think they will move all of their systems to the cloud, while 37 percent foresee most of their systems moving to cloud. While 34 percent use the cloud for data storage and 45 percent for software-as-a-service applications.
Finally, 11 percent of respondents said the cloud “doesn’t make sense” for their business.
-
Clarity of Purpose
Here are some other ROELBOB’s you might like
W Edwards Deming Finds His Roots
The Pink Slip
Another Pink Slip
Hot off the Press – Other great DevOps.com content
DevOpsSec: 1 in 16 Chances
CIOs Must Fight for Relevance in Face of Tech Storms
— ROELBOB
-
Multi-Tiered Escalation Policies are Paramount
As any team charged with keeping systems up and running will tell you, having one person on-call isn’t enough. What happens if your on-call engineer sleeps through his alert? What happens if his phone’s battery dies without him knowing, or if he gets an alert at an inconvenient time, like when stuck on a bus or in traffic? It will happen.
In today’s always-on world, companies can’t afford for that to happen without a backup plan. One or more people should always be waiting in the wings, ready to spring into action if a primary on-call is unable to perform his or her duties to the best of his or her abilities at any given time.
Engineers who perform backup duty don’t need to be as “on-call” as a primary engineer — no pagers needed here — but what backup engineers lack in readiness they can make up for in numbers. It often makes sense to have multiple backups or an entire team of backups.
Operationally mature companies often group the currently-on-call engineer and all of his or her backups into an escalation policy. Escalation policies set the order in which each person is alerted and the delays between alerts. While there are many ways to organize these escalation policies, here are some of the patterns most often used:
Primaries Aren’t Enough; Determine a Secondary, Too
Most companies have a primary on-call engineer, usually determined by a calendar that designates when members of an ops team are on call.Operationally mature companies take their on-call practice one step further and supplement this rotation with a rotation of secondary on-call engineers. Why a secondary rotation instead of just paging the whole team when the primary falls through? If a large team is paged, you get a ‘tragedy of the commons’ effect— it’s urgent that somebody respond to the issue, but not clear how urgent the issue is that each individual person respond. Having a designated secondary removes that confusion.
The secondary rotation is set up to shadow the primary rotation and generally has the same members as the primary rotation. The difference is that the order of the engineers in the secondary rotation is usually offset from that of primary rotation— if your primary and secondary are the same person, you’re going to have a bad time. For example, if a primary rotation contains Alex, Bob, then Charlie, then a secondary rotation may contain Bob, Charlie, then Alex in that order. Companies often find that if last week’s primary is this week’s secondary, the it helps make sure there’s somebody with recent context who can help if the primary runs into trouble. It is becoming more commonplace for organizations to have both a primary and secondary on-call rotation; but, in an environment where customers expect zero downtime, secondary rotations are no longer an optional tool.
Enable a Strong Support System
Larger companies and companies with significant numbers of operations tasks often deploy a separate support team that handles all basic operations tasks before the organization elevates the tasks to an on-call engineer. This team — the first tier support team — usually knows how to handle all of the smaller, repetitive problems that crop up often and have clear resolution procedures. These first-tier teams often offer their support to multiple engineering teams and are placed first in an escalation policy.If first-tier teams cannot resolve the issue, escalation policies typically dictate that primary then secondary on-call engineers must complete the task. However, there is one more support network operationally mature organizations must implement: management. Management is ultimately responsible for a company’s systems, and needs to be in the know when severe problems happen— both to hold their teams accountable and to muster additional support. At PagerDuty many of our EPs have a rotating manager schedule at the top of multiple escalation policies, and many of our customers use this pattern as well.
Put Your Tiered Teams to Work
In practice, an organization policy for a hypothetical “Database Ops” team might look like this:- Assign the incident to the user who is on-call in the First-Tier Ops Team schedule
- Assign the incident to the user who is on-call in the Primary DBA schedule
- Assign the incident to the user who is on-call in the Secondary DBA schedule
- Assign the incident to the team lead
- Assign the incident to the dev manager
This escalation policy would unfold over time, with timeouts of between 10 and 30 minutes before each subsequent escalation, depending on a team’s needs.
Implementing an escalation policy that involves multiple tiers of resolution ensures that devops teams address all alerts — whether business-critical or minor — in a timely manner. Limited downtime is paramount in today’s fast-pace business environment. Customers expect their applications to run smoothly, or they expect to switch providers if devops teams cannot resolve issues fast enough. Organizations will be much less likely to see their processes fail, and much more likely to see happy customers, if they implement a multi-tiered escalation policy.
-
An obituary to a Managed Test Service
1st October 2020, Enterpirseborough , F500 Inc street
Managed Test Service, 66, died Friday, 3rd October, 2020 at the Delivery governance war room in the IT head quarters of F500 inc. following a terminal illness caused by a lifetime burden of 34 SLA’s, debt burden of 44 KPIs , 29 painful claw backs by procurement due to failed deliveries and overdose of turnaround specialists and consultants
MTS leaves 4 programs , 45 projects, 12 application centres and a number of small changes behind.Born and raised in the HQ , MTS was a lifelong resident of the borough.
In the early years, MTS would help do functional testing for the core system portfolio and eventually taking ownership of non-functional testing and user testing. After graduating from the delivery transformation high school , MTS attended the SLA University, graduating with a degree in cost optimisation and skill super-speciaisation. MTS eventually joined the IT Board, where he was the voice of quality(or so he believed)
His attention to detail and obsession with centralisation of anything that had Test in it’s name was legendary. He was proud and preached the virtues of , albeit bloated and inflexible, test service
However,he struggled with newer delivery approaches of Agile and DevOps which spread like cancer throughout the delivery organisation. Devops challenged the fundamental idea of skill super-specialisation and independent testing that MTS championed and built his empire on. DevOps challenged the way of living of MTS as unsustainable
DevOps valued people and interactions over process unlike MTS which obsessed about process
DevOps valued working software while MTS was worried about comprehensive documentation
MTS got caught up in time-wasting contract negotiations instead of allowing his team to focus on customer collaboration
MTS preferred a predictable plan while his delivery friends had already moved on with adaptable plans
A funeral service was held on Monday 6th October from the funeral Home of Procurement , with a private burial.
-
A DevOps Coders Time Allocation
Here are some other ROELBOB’s you might like
A Unit Testing Dillema
The Pink Slip
Another Pink Slip
Hot off the Press – Other great DevOps.com content
DevOpsSec: 1 in 16 Chances
CIOs Must Fight for Relevance in Face of Tech Storms
Upcoming DevOps.com Events
Webinar: SecDevOps: The Marriage of SecOps and DevOps
— ROELBOB
-
Getting Rugged DevOps Right
Two Perspectives
Jack, an accomplished application security pro, tells me, “The developers won’t talk to us. It’s like we speak a different language. They are releasing new builds so fast, how could they check each one for security vulnerabilities? We can’t move as fast as they do.”
Then in the next moment, Diane, a DevOps pro, let’s me know, “Our current security team’s tools and practices can’t keep up with the pace we have now established. If it doesn’t fit, it doesn’t get done here.”
I have heard this little ditty ‘bout Jack & Diane so often in the past two years, it could be a hit record. I imagine you have heard it too.
The Slow Lane Won’t Lead to Rugged DevOps
Let’s visit Jack’s world, first.
When visiting a well-known bank last month, they shared insight on their current application security practice. They have 10 full-time people employed (offshore where personnel costs are lower) to analyze a set of 100 applications. The bank has over 2000 applications, but the security practice can’t scale to access their entire portfolio. In 2.5 years, they had only been able to cover 5% of the application portfolio.
The application security team employs an application security tool originally built for waterfall-centric development timelines. As much as they want it and need it to move faster, it is not really for “continuous” velocity.
Each scan of an application can require anywhere from four hours to two days to get an assessment report. The reports average about 30,000 potential defects which then require further analysis. That analysis may take up to two weeks in order to sift through thousands of false-positives and false-negatives. While this security practice is critical to the bank, it cannot keep pace with development practices that are aiming to go much faster in order for the bank to remain competitive in their industry.
The Continuous Gap
Sound familiar? It should. This scenario is playing out all across the DevOps and Continuous Delivery landscape. It is reflective of what I call “The Continuous Gap”.
Leaders in Rugged DevOps practices aim to minimize the “Continuous Gap”. In Diane’s world, DevOps teams pursue “shift left” strategies in an effort to build quality in from the beginning. Shifting left enables them to bake in the right code, components, configurations, and practices from the beginning. When done right, velocity increases dramatically, while operating costs are minimized, and unplanned/unscheduled work is reduced. Investments can then shift more in favor of innovation over upkeep and maintenance.
The Old-School Rub
Jack’s world of traditional application security practices often wait to check for security vulnerabilities at testing or release stages of the SDLC. There are two reasons for this: (1) some security tests like static or dynamic analysis can take hours or days to perform and (2) security teams cannot embed themselves earlier in the SDLC (due to political, organizational, or operational obstacles) without dramatically slowing development in a high velocity world.
If you are using old-school application security tools, you will continue to be blocked from getting more deeply entrenched in development. New velocity requirements call for new approaches.
The New-School Shifts Left
A number of newer application security solutions and practices been introduced recently that are development and developer-centric. For example, imagine if build managers could analyze every open source component used in every build for known security vulnerabilities.
What if, developers could access information inside their IDE that would inform them if a component they were planning to use in an application was known to be vulnerable? What if developers had access to spell-check like alerts of security issues as they were creating their code?
Furthermore, imagine if information was available that not only told the developer of a known security defect, but directed them to an alternative safer version of that component to use. Components would be automatically vetted against corporate security policies at the instant they were chosen by a developer. When 90% of an application is composed of open source and third party components today, automatic analysis early in the lifecycle save millions of dollars. This how Rugged DevOps rolls. This is Diane’s world.
The evolution of Rugged DevOps is forcing application security practices to shift left. But Diane’s world (and your’s for that matter) are not just about increasing velocity. All organizations need to consider operational costs to both detect and remediate security vulnerabilities in their applications. We all realize that the further along we are in the application development lifecycle the more costly fixes become. Feel free to apply your own cost figures to the chart below, but you should also note that these calculations do not include breach cost, help desk calls, maintenance, time-to-market, reputation or stock price.
Cost considerations for Rugged DevOps practices compared to waterfall approaches. Rugged DevOps: Shifting Gears…Quickly
Shifting from old-school to new school approaches doesn’t take long. On a recent visit to one of the largest insurance companies in North America, they shared their tale of transition. Of five major applications that run their business, one in particular had 40,000 different files that needed to be analyzed. Their old-school approach involved a lot of manual code investigation and required about two-and-a-half weeks to complete the analysis of that large application. The new approach that used automated analysis of security vulnerabilities took two minutes — and produced identical results.
A government tax office wanting to shift security analysis of applications from hours (outside of development – Jack’s world) to seconds (inside of development – Diane’s world) was able to complete the first pass analysis of their existing portfolio of 600 applications within two working days. The security team was not only faster at identifying known security vulnerabilities in applications, but could also quickly identify alternative open source and third-party software components that were safer to use. The security team was not simply discovering issues, but the new class of Rugged DevOps solutions were aiding in guided remediation.
Questions to Ask About That Rugged DevOps Solution
There are several companies and open source projects that now offer application security solutions that work at extremely high velocity. When exploring solutions to match the velocity of your Rugged DevOps practices, consider asking the following questions:
- How long does it take to complete the analysis of a small, medium, and large application?
- Do developers use these tools, or are they only meant for security personnel? Do you have members of development teams that act as positive references for your tool?
- Does the analysis require further investigation of the findings, or does it pinpoint the root cause of the issue?
- Does the solution only identify vulnerabilities or does it help guide remediation of issues?
- What is the balance of false positives to positive alerts generated by the solution?
- How many people are employed to support the solution in environments similar to ours?
