devopsy.kinsta.cloud

Tag: Government

Microsoft’s GitHub Hit by Major Outage as AI-Driven Demand Strains Infrastructure

GitHub, the Microsoft Corp.-owned code hosting platform serving more than 180 million developers, is still reeling from a widespread outage on Monday that severely disrupted software development pipelines globally.

The hours-long incident was the latest in a relentless series of reliability failures for a service struggling to keep pace with an unprecedented surge in artificial intelligence (AI)-assisted coding traffic.

The disruptions began around 9:40 a.m. EDT, initially manifesting as performance degradation across core services. The issue rapidly escalated, causing error rates to spike near 20% for web interface and API traffic, while archive and raw repository content downloads suffered a staggering 50% failure rate.

Key capabilities — including GitHub Actions automated testing, webhooks, GitHub Pages, and the AI pair-programmer Copilot — were heavily compromised. Downdetector logged thousands of user reports at the peak of disruption, with minor outage spikes simultaneously impacting Microsoft Teams.

By midday, GitHub engineers reported they had identified the problematic component and deployed corrective action. While incident reports began to subside, the root cause remained undisclosed. (In a blog post Thursday, GitHub Chief Technology Officer Vlad Fedorov said the outage occurred when traffic reached a new peak, and a critical infrastructure component in a Central U.S. data center failed to scale with it. He outlined steps to avoid future disruptions.)

“Continuity plans should start with what the business must keep running, not a guess about what will fail,” said Sterling Wilson, IGEL’s field chief technology officer for business continuity and disaster recovery. “An attack and an availability outage call for different responses: one may demand containment and forensic preservation, the other failover to alternate workflows. But both expose the same gap when an organization hasn’t identified its critical dependencies, decided which services and users recover first, or tested how work continues when a primary platform goes dark.”

The service failure arrives amid growing developer fatigue over GitHub’s operational stability.

The platform recorded eight separate degradation incidents in July alone, followed by an Aug. 6 disruption that GitHub itself labeled “unacceptable.”

“We’ve made writing code nearly free, but we haven’t made understanding it any easier,” said Traversal co-founder Raaz Dwivedi. “Writing code used to be the bottleneck. Now understanding how it behaves across a complex production system is. If AI ends up writing most of the world’s code, there simply won’t be enough humans to debug it all. We’re about to generate more code in the next twelve months than in most of software history, on systems that were never designed to be understood at that speed — that gap is exactly where outages like GitHub’s come from. Keeping up with AI-written code will require AI that can understand production systems as deeply as it can generate code.”

Engineers have pointed to structural bottlenecks as the ecosystem buckles under massive traffic increases. Earlier this year, GitHub leadership acknowledged that AI-assisted workflows were placing significant stress on backend infrastructure. To cope with the shift in how software is built, the company embarked on an ambitious strategy to expand platform capacity thirtyfold.

The frequent downtime is prompting technical leaders to re-evaluate their reliance on single-provider ecosystems.

As GitHub increasingly centralizes source control, continuous integration, and AI generation, an infrastructure outage effectively halts the modern software assembly line, industry experts point out.

“Ten years ago, a GitHub outage might be viewed as a minor inconvenience that only affected a small portion of the business. Today, GitHub hosts source code, CI/CD pipelines, and infrastructure-as-code that organizations rely on every day,” said Alka Tandon, senior director of product management at Rubrik Inc. “When the platform goes down, it can immediately halt entire business units and delay product releases, security fixes, and other critical work. The scale of this week’s GitHub outage shows just how quickly that dependency can become a business problem.”

For engineering organizations worldwide, the recurring disruptions are shifting redundancy from a secondary consideration to an operational imperative, say security experts.

“The safest models in the world spent the last few weeks explaining how and why they got into networks nobody authorized, completely unintentionally,” Kfir Fleischer, vice president of cyber research and product at Dream. “Those are the models with sophisticated supervision. The threat facing government surfaces is even larger from open-weight models without professional supervision. Zero trust for governments isn’t a judgment on model owners. It’s a decision about what governments refuse to depend on. Countries must assume every credential is stolen and assume every model is jailbroken. Then they must build a sovereign national AI stack that holds.”

August 18, 2026
Oracle Bill is 5x Client’s Budget ¦ Toyota Out of Space

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Birmingham looks like the Detroit of the UK—is it Oracle’s fault?
Plus: Was Toyota’s factory failure caused by running out of disk space? (more…)

September 7, 2023
Enabling Cloud Adoption in the Government
In 2022, federal agencies are estimated to spend $7.8 billion on cloud computing. While this spending on cloud computing is necessary, it’s imperative that the government uses its allotted funding appropriately to leverage the cloud’s full potential and avoid common pitfalls.

Avoiding Common Cloud Adoption Pitfalls

I’ve frequently witnessed government agencies get locked into a cloud reseller that doesn’t give them the full capabilities offered by AWS, Google Cloud or Microsoft Azure. Most people outside of the federal market don’t realize that the government does not buy cloud directly from the cloud providers—apart from some very large contracts like the upcoming JEDI replacement contract vehicle, JWCC. They buy through a network of reseller partners that offer volume discounts and try to bundle managed services to help with cloud management.

This approach gives the reseller, not the customer, the power to hold the keys to their kingdom—and can make it difficult for the customer to easily switch between cloud providers in the event of poor performance, lack of access or billing transparency. The federal government would benefit from keeping a small, but purposeful, cloud project management office (PMO) in-house. Thus ensuring that root access is available to all the accounts, subscriptions and projects to provide freedom to the agency to more easily change partners as their cloud maturity evolves.

I’ve also seen government customers unfocused on the holistic end-user experience of actually using the cloud. The result is fragmented collections of different systems, processes and steps that cause engineers to be frustrated and inefficient. Customers must transform the end-to-end cloud provisioning process so that self–service is paramount and existing technology investments are seamlessly and transparently integrated to the end-user. When done correctly, innovation will be the byproduct, because focus can go towards developing the next generation of mission systems and tools—rather than trying to determine how to get a cloud account to begin a project.

Three Steps Government Agencies Can Take to Get Value from Cloud Infrastructure

Challenges exist, but there are proven steps the government can take to get the most value from their cloud infrastructure.

Define purpose: First and foremost, understand the purpose the cloud infrastructure intends to serve. Are you looking to reengineer cloud architecture, become more cost-efficient, or transform the innovation potential within your organization by introducing new services that can take you farther, faster? Understanding this purpose and your goals will go a long way towards success.

Quantify ROI: To get the best return on your investment, understand what resources are already available and which are necessary for your operations. To fully take advantage of the cloud, it is important to make sure basic costs are under control, such as ensuring AWS EC2 or Azure VM instances are not overprovisioned or underutilized. It’s equally important to make sure you take advantage of reserved capacity reservations for your predictable workloads; these can save you 30% or more. Last but not least, keep a focus on fully leveraging technology solutions to help bring all the pieces together in an integrated user-friendly experience across cloud providers.

Consider the outsourcing approach: Bringing in cloud resellers and contractors can provide guidance to your organization and help you make better choices. Just like building or renovating a house, agencies need general contractor-type companies that can help to manage and orchestrate the process of migrating to the cloud and all of its moving pieces. These contractors can also help to locate the various specialists required to get the job done for each unique situation. Long term, having a small, efficient cloud PMO is something every agency should strive to have in-house; however, given the demand for cloud engineering talent, the near-term focus should be to ensure flexibility into engagements with partners, so that they don’t hold back the potential of the organization.
As digital transformation and cloud migration continue, it’s important for the government to stay on pace with its adoption. As of now, most agencies are still relying on manual steps to validate controls and get Authorizations to Operate (ATOs) approved. Making the simple swaps to more automated processes, learning how to leverage cloud resources to their full ability, and outsourcing operations to contractors can make acceleration easy. There has never been a greater need for cloud enablement. The time to scale up is now.
January 14, 2022
Government Has a DevOps Lesson for Private Sector
The missions of government organizations and DevOps are well-aligned, according to F5 system engineer Scott Van Kalken, who works with many organizations at various levels of government around Australia.

That’s because government organizations provide services to the community, and DevOps is about providing services to a community of users.

He sees more use of DevOps in the government sector than in corporations.

Increasingly, DevOps is embedding security from the start of a project. “That’s really exciting for me,” he said, as it shows DevOps is working and allows faster iteration with security issues being picked up earlier.

For example, the COVID-19 pandemic has resulted in more people working at home. That makes security considerations even more important than ever.

All levels of government have seen massive increases in remote working capacity and functionality. Applying DevOps principles to security practices has allowed them to adapt rapidly to those increased levels of working from home.

One large department took just three or four days to go from 1,000 to 20,000 people working from home. Van Kalken said this was possible partly because they “automate absolutely everything they are doing,” allowing rapid implementation of security policies. They quickly stood up additional cloud infrastructure with the right security settings by using DevOps practices. Similarly, DevOps teams can use APIs to easily and quickly apply security policies to SaaS applications.

Cross-disciplinary Cooperation and Coordination

One government department that Van Kalken is involved with was new to DevOps but very good at getting representatives of the relevant groups into the same room. They all focused on reaching the desired outcome with “no turf wars” and agreed that establishing appropriate security policies was part of that process.

In his experience, the private sector tends to be more siloed, which can make it more difficult to make the right security-related choices from the outset.

For government organizations with more mature DevOps practices, “it’s just an incremental ramp-up,” he said. The collaboration between the various functions makes a big difference to successful outcomes.

So the role of the security organization changes from being akin to a nightclub bouncer (“No, you can’t”) to that of a partner in the delivery of services.

The private sector can learn from this, Van Kalken suggested. Government organizations have a “laser-like alignment” on service delivery. In contrast, the differing and competing objectives of various departments or functions within corporations may be a handicap.

Most organizations have adopted one of the published security frameworks, but project teams have the responsibility to decide how to comply with them. Government entities are different, he said, in that they must at a minimum meet the requirements of the Australian Government Information Security Manual (ISM). That leads to more alignment between departments and functions within the organization.

DevOps and Security Hygiene

From a more general security perspective, “governments are targeted in various ways” as they increasingly make services available online, he said.

“Government is no different to every retailer, every bank, and so on,” in the sense that all the usual threats apply, said Van Kalken. But some players specifically attack government organizations because of the types of data held about individuals. Fortunately, DevOps practices assist in responding rapidly to changing situations.

As a simple example, organizations can quickly deploy new policies to distributed firewalls by using DevOps principles including automation. That gives skilled individuals more time to deal with any exceptions that occur.

Organizations can apply similar practices to improve security hygiene by more quickly rolling out new versions of commercial software.

“That element of speed is something very new to the security space,” he said.

Recommendations

Van Kalken’s recommendations are:
- Get all the right people—including security—together in the project team from the outset.
- Ensure they all understand the objective.
- “All swim in the same direction.”
- Make sure everyone continues to work collaboratively.
The rest “will just happen if you do these things,” he said.
August 11, 2020
Continuous Delivery: No Excuses
Jez Humble’s (@jezhumble) career has spanned roles through coding, infrastructure, and product development across three continents and organizations of varying sizes. To say he knows a lot about continuous delivery is a total understatement. In 2010, he and Dave Farley literally wrote the book on continuous delivery—and if you have not yet read it, I would suggest you pick up a copy. It’s a fascinating read.

I’ve heard Jez speak over the years at a number of DevOps conferences, and I was excited when he accepted our invitation to speak at All Day DevOps. At the conference, Jez presented “Continuous Delivery Sounds Great But It Won’t Work Here,” during which he addressed the four reasons he consistently hears from organizations on why continuous delivery won’t work in their organization:
- We’re regulated
- We’re not building websites
- Too much legacy
- Our people are too stupid
Jez also outlined the real reasons behind the stated reasons: Their culture sucks and/or their architecture sucks.

Jez spent his talk addressing each objection.

We’re Regulated

Jez noted it took Amazon four years to go from monolithic to service-oriented architecture (2001-2005) but the company did it. Do you think Amazon is the pillar of free enterprise, free of regulation? Not so. Being a publicly traded company, Amazon is heavily regulated; it is subject to Sarbanes-Oxley regulations; it handles personal and corporate financial and proprietary data; and it provides private cloud services to the U.S. government, including highly sensitive data for the Department of Defense.

Amazon Web Services’ GovCloud, in fact, provides the cloud platform for cloud.gov, the 18F project Jez worked on to address the authority to operate process within the government.

The old process had screened every software application before it was allowed on a federal IT system. It stifled innovation because it duplicated efforts multiple times over. There was no cross-agency assumption that if one agency approved a software package, it was okay at another agency. Cloud.gov moved controls from operations into infrastructure as a service. Now, agencies can receive software through cloud.gov to integrate into their applications.

The bottom line is continuous delivery is excellent for heavily regulated environments because it gives assurance to regulators by removing opportunities for human error.

We Aren’t Building Websites

Jez’s favorite example is HP LaserJet firmware. HP had a 400-person team spanning three continents. Obviously, the firmware was critical for new printers, but the team was moving slowly. Only 5 percent of their time was spent on innovation. HP tried hiring, firing, insourcing and outsourcing. With business school tried-and-true solutions exhausted, and knowing it had to eliminate non-value adding work to increase innovation, the company looked to continuous delivery with two business goals in mind: 1) Get firmware off the critical path; 2) Achieve 10X increase in development productivity, measured by percent spent on innovation.

HP had other goals:
- Reduce hardware variation
- Create a single package
- Implement continuous integration
- Implement comprehensive test automation
- Create a simulator
Note: the chart above doesn’t show the 23 percent of time spent managing the automation tests. Time well-spent.

In the end, between 2008 and 2011:
- Overall development costs were reduced by ~40 percent
- Programs under development increased by ~140 percent
- Development costs per program decreased 78 percent
- Resources now driving innovation increased by 8X
Too Much Legacy

Jez’s example here is Suncorp, one of Australia’s largest insurers (note: Also falls under the “We’re heavily regulated” excuse). Suncorp was burdened with loads of mainframes and legacy systems, but it built a set of test suites with open source tools to move into continuous delivery.

Our People Are Too Stupid

Jez looked outside of the software development industry into an old-line manufacturer: automotives. GM’s worst performing plant had been closed down. GM fired every one of the workers from the plant. The workers were notorious for problems ranging from sabotaging cars to drinking to gambling on the job.

Remarkably, GM’s union leaders convinced Toyota to hire the old GM employees to learn the Toyota processes, and Toyota hired the GM employees to run their new NUMMI plant. Toyota and the old GM employees showed it wasn’t the people that was the problem; it was the leadership, the management system and the use of obstacles. At the GM plant, an employee had a job and a certain amount of time to do it. If it didn’t get done, the assembly line kept running. At the Toyota NUMMI plant, employees were empowered to ask managers for help—without retribution. The car had to be in a good state before proceeded to the next stage of the production line.

In software development, this is continuous integration. Toyota’s NUMMI is the inspiration for Lean in the United States.

Jez’s take-home point for any organization is that results aren’t achieved when you adopt practices but don’t implement the culture. You need to:
- Agree and communicate measurable business goals
- Give teams support and resources to experiment
- Talk to other teams
- Achieve quick wins and share learning
- Keep going
Jez dives into more details in his talk, which you can watch here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free of charge here.

— Derek E. Weeks
February 23, 2018
DevOps Heals at NIH
Cancer sucks. But with folks like Sarah, DevOps is helping make a difference in the race to a cure.

Sarah Elkins is not curing cancer herself, but she is employing DevOps practices to help those who are. Sarah supports the technology infrastructure for those who are trying to cure cancer at the National Institutes of Health (NIH).

Sarah Elkins (@configures) configures technology solutions at the National Cancer Institute (NCI), where she has worked for nine years. NCI is a federal agency, part of the National Institutes of Health. It supports more than 700 websites, from basic HTML to content management systems to complex bioinformatics systems that have multiple tiers and thousands of servers. It uses multiple operating systems, containers and physical and virtual machines (both on-premises and some AWS). Developers range from lone scientists to large teams.

Sarah and I have been following one another on Twitter for several years, but I recently caught the session she delivered in the “DevOps in Government” track at All Day DevOps. Sarah led a discussion about how the National Cancer Institute is automating its builds and deployments—showing it is possible even in a large bureaucracy.

Her talk covered the processes and technologies that enable software to move from source code repositories all the way to production servers at nci.nih.gov, including the use of GitHub, Jenkins, Nexus and other technologies, with a variety of teams involved.

We Need to Talk

Sarah’s cure begins with engagement: development, infrastructure and security all working together, just as teams of specialists work with patients to help them beat cancer.

To help speed development and approvals—a necessary evil that can grind development to a halt if not managed well—they agree that applications drawing on an existing technology catalog are approved as operational and security teams provide guidance to development on what is required for an Authority to Operate (ATO). In other words, they are trying to preapprove as much as possible and communicate earlier in the process.

DevOps Practices are Maturing

Source code is primarily kept on GitHub, including a public repository for non-proprietary source code, and NCI primarily uses Maven or Apache Ant for build scripts. Infrastructure teams provide XML templates to provide consistency.

Most NCI software relies on build dependencies on open source software components. The organization uses artifacts during builds and utilizes Sonatype’s Nexus as its repository manager.

All of this supports the automation of builds and deployments at NCI. For builds, most active projects are either on or migrating to Jenkins. Build artifacts may be .zip, .war/.ear or Docker images.

For application deployments, NCI uses development, quality assurance, stage and production stages. Most teams use some form of automation, from simple (copy content, stop/start container) to more complex scripts. Developers are allowed to perform deployments for robust applications, and some manual orchestration is required—for instance, for database timing or related applications.

As an organization, NCI is moving toward continuous integration (CI), with varying progress among teams.

Building In Security

NCI also now includes security early, often and automatically. The organization has deployed self-serve/on-demand Nessus scans, which allow developers to see, on their own, how they are doing as soon as the application is stable. Security teams run AppScan and Twistlock for Docker image scanning during Jenkins builds, just before deploying and frequently scanning the repository in between. For issues found, development and infrastructure teams work together to remediate security concerns.

At the end of her talk, Sarah offered up three takeaways:
1. DevOps at NCI is a work in progress (isn’t it everywhere?)
2. The wide-ranging needs at NCI require flexibility, communication and teamwork
3. There is no shortage of work
Since it is a work in progress, what opportunities are ahead at NCI?
- More automation for individual applications with Jenkins
  - Developers can perform container restarts on lower tiers
  - Database updates (some projects are using Liquibase already)
  - Some applications are still manual/batch scripted
- More Docker and improving Jenkins/Docker instances
- More orchestration automation and Puppet/Ansible integration
- Security improvements through
  - Scan dependency artifacts before building
  - More integration and automation
Sarah dives into more details in her talk, which you can watch here. If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. DevOps in Government is one of the five tracks.

— Derek E. Weeks
February 5, 2018
Cybersecurity Improvement Act of 2017: The Ghost of Congress Past
It seems like yesterday when when Representative Ed Royce proposed legislation titled the “Cyber Supply Chain Management and Transparency Act.” Almost three years have passed since Congressman Royce introduced his bill in November 2014 and helped educate the world about serious security vulnerabilities (such as Apache Struts and Heartbleed) lurking inside open-source components that are commonly used by anyone and everyone building modern software applications.

Broadly speaking, Royce’s bill would have required technology vendors selling software to the U.S. government to do the following:
1. Provide customers with a bill of materials documenting all open-source components utilized in the software application;
2. Demonstrate that the component versions utilized in the application have no known vulnerabilities (CVEs from NVD) for which less-vulnerable alternatives are available;
3. Provide a mechanism to promptly remediate new vulnerabilities when they are discovered.
Despite dying a quiet death, the Royce bill helped to pioneer an important conversation between government and industry leaders with respect to the age-old question of software liability.

This conversation continues to evolve—and just recently Sen. Mark Warner, a Democrat in Virginia, and Sen. Cory Gardner, a Republican from Colorado, introduced the “Internet of Things Cybersecurity Improvement Act of 2017“—an attempt to force companies selling IoT devices to federal agencies to adhere to new security standards.

Just like the Royce bill before it, the newly proposed legislation from Sen. Warner would require vendors selling IoT-connected devices to government customers to do three simple things:
1. Provide written certification that IoT devices do not contain hardware, software or firmware components with any known security vulnerabilities or defects listed in the NVD or similar databases.
2. Notify government customers of any new security vulnerabilities or defects subsequently discovered.
3. Provide a mechanism that allows for any future security vulnerability or defect in any part of the software or firmware to be patched or fixed in a timely and secure manner.
As Bruce Schneier observed more than a decade ago, there are no real consequences for having bad security or having low-quality software of any kind. Even worse, the market often rewards low quality. More precisely, it rewards additional features and timely release dates, even if they come at the expense of quality.

Although the Royce bill failed and the future of the Warner bill is yet to be determined, there is an increasingly steady breeze blowing from Washington, D.C., that is gently nudging the entire software industry toward a future in which vendors will no longer be immune to liability for damages due to known security vulnerabilities or defects.

About the Author / Matt Howard

A proven executive and entrepreneur with over 20 years experience developing high-growth software companies, Mr. Howard leads Sonatype‘s corporate marketing, strategic partnering, and demand-generation initiatives. Prior to Sonatype, he co-founded, developed and successfully sold two software companies. Earlier in his career, Howard led sales and marketing at USinternetworking (acquired by AT&T) and Groove Networks (acquired by Microsoft), where his teams distributed workgroup collaboration products to enterprise customers. Howard holds a Bachelor of Arts degree from The George Washington University and a Master of Arts from George Mason University.
August 17, 2017
Dismantling Inefficiency: The DevOps Point of View

I recently caught up with Paula Thrasher, application delivery lead at CSC and DevOps pioneer in government circles. In our conversation, we discussed a number of important topics including cross-functional teams, software supply chains, paying down technical debt and her search for passionate people to hire. To succeed in managing large projects, she’s taking a different approach—one you might want to apply in your own organization.

Thrasher: I’m Paula Thrasher with CSRA and I work for a division company we call Delivery Services Organization.

Derek Weeks: What was your presentation at the Rugged DevOps conference about today?

Thrasher: We had a panel about the people, process and cultural aspect of DevOps, and the security context, as well. When you’re moving to the Ops transformations, what kinds of behaviors are both challenges and opportunities.

There was a comment at the beginning session about how any sort of transformation that relies on human nature has failed. So you can’t change how we work, but you can leverage how people are motivated to try and encourage the right behavior. I think that’s the thing that’ll resonate with the security professionals.

Link to Panel: https://www.youtube.com/watch?v=IbrY8mm9aI4&index=7&list=PLotLY1RC8HovW-XnZbItSBupjbTzFkfJw

Weeks: You’re setting up a new organization and have a lot of projects that you’re going to take on. Is it possible to achieve a lot with smaller teams? Do you need a big team to do rugged DevOps?

Thrasher: I think the main thing that we’ve been focused on is creating the team that is sort of the center of the universe. I think the most important thing is to have a team that is cross-functional. A lot of times when you think of teams … this is the development team, this is the Ops team, this is the security team … the question is, do you have all the players on the same team, or are they on different teams? I think if you find yourself in a situation where any one team has to deliver to another team, instead of actually being responsible for delivery of the actual business submission function, you’ve got some horrible web of dependency you’re going to have to untangle.

The big focus of what we’re trying to do in the new organization is build that completely cross-functional team and to enable them as a team to be placed on different projects, different missions. So instead of doing single-band rotations, where a single employee moves onto a new project, we’re actually trying to move full teams. As a team, they learn how they work best and how they can deliver. It makes a really powerful solution for actually solving problems. I’m really excited about that.

Weeks: A lot of people at the Rugged DevOps Conference are interested in the rugged DevOps phase because they’re concerned about being left behind by DevOps. Is that a real concern for security teams: getting left behind?

Thrasher: Yes, we definitely see a dichotomy in the world of how security interacts with development teams—between the traditional showing up at the end and inspecting versus really embedded security. It’s so clearly going to this idea that you can’t just inspect security. Security can’t be this checkbox at the end. You really have to build it in from a computer. You have to actually make design decisions about security. You have to make sure the developers know secure coding practices. You have to have tools that scan, check all the time, and get feedback. You can’t just add that at the end.

I think it’s a real challenge to security folks, just as it was for everyone—developers and Ops people—to change how you think to that systemwide view. You can’t just be a phase at the end. We actually have to be part of a whole equation, that whole systems thinking. I think that is so instrumental to what DevOps really is as a group.

Weeks: One of the things that you recently talked to Mark Miller about was an idea of using pictures to help with collaboration and to get everyone on the same page. Can you tell me about that practice?

Thrasher: Back to this human nature piece, I think humans are inherently really visual thinkers. I also think you’re asking people to bring cross-costuming across social teams. Well, if you’re a security person and you’re talking to an Ops person, you probably don’t speak the same language. Or if you’re a developer and you talk to a security person, again, you don’t necessarily speak the same language. But when you speak visually, you can understand each other better. You don’t bring your acronyms to the table. We’re talking about a diagram.

I also think it takes the personal element out of it a little bit. We’re talking about the system. We’re not talking about each other. The security team is not accusing the developers of being a bunch of renegades. They’re talking to them about a process and how they process it. It keeps the conversation where it needs to be, which is on making the process better and delivering better software.

Weeks: How is the growing interest in software supply chains changing the way you’re thinking about DevOps?

Thrasher: I think simplicity is an underused tool in security. Recently, one of our DevOps teams that was going in their rapid fashion had a deployment that had a vulnerability in it. They had to respond quickly, and they actually did respond quickly. But when we unpacked it, in retrospect, we realized: “How did we get here? How did we let this vulnerability get in?” We had all sorts of scanning. We had, supposedly, all the right things to have prevented that security flaw. Yet, how we got there was this Venn diagram of all these things overlapping. It was one of those security bugs that slips through the cracks.

We realized what happened is we had gone to over-engineering; we had over-engineered for performance. In doing that, we had overlooked the interaction of three components in a very particular circumstance. It was like a “every third Tuesday if you spun around three times” kind of bug. But I think that’s exactly what happens a lot. That’s super common in terms of those last defects, and there’s quite a bit that comes out of that overlap of complexity.

Fewer, more secured suppliers reduces the lift of having to upgrade updated components. It also means if you find an issue, your unpacking of that issue is a lot easier because you have a lot less things going on. All engineers need the reminder of “keep it simple.” But I think that’s a real powerful thing. Fewer, better suppliers, from a supply chain standpoint, is one of the mantras that I really appreciate because I just see it match. It really makes a difference.

Weeks: You’re working with a lot of government agencies across the board. What’s their interest in rugged DevOps practices, or just DevOps practices that you’re bringing into play?

Thrasher: I think in the government space, and it’s true for any enterprise, everything has to have security. There is not an agency that we can work with that doesn’t have privacy data, or heavily secured data, in the defense and intel agencies, or just straight-up vulnerabilities from being a target. The scale of the enterprise means there’s a lot of potential systems and a potential surface area for attack. You can think to all the high-profile attacks that have occurred in the federal government recently, and it’s made people really alarmed!

One of the things that’s very interesting is that the whole federal government did a “cyber sprint” last summer, where they did nothing but pay down technical debt, and get rid of systems and their vulnerabilities. It was wonderful! And I think every organization should do just that: spend one month a year doing nothing but eliminating old frameworks and applications. That really simple, easy thing often gets overlooked because businesses are always saying, “More, more, more, more, more.” However, we actually measured the metrics of the teams after the “cyber sprint,” and all of our teams’ velocity increased. After we let teams upgrade legacy components, all of them had an improvement, which is sort of like a “Well, duh!” But I think it’s a win-win really, that if we can allow ourselves to invest in paying down our technical debt we can get better results.

I think that in a government space, it just has to be part of the equation. So much of what we do is so instrumental to everything.

Link to interview: https://www.youtube.com/watch?v=5veGeDAaJSI

Weeks: As we started off the conversation, you have a new group. You have a lot of projects that are going to get underway. Are you hiring?

Thrasher: Absolutely.

Weeks: Who are the type of people that you’re looking for, if someone wants to get in touch with your company?

Thrasher: We are at CSRA.com, and if you go to our jobs and our careers website, you can see some of the openings we have. Actually, I’m really excited. One of the projects we did internally was building a whole new hiring model. We used big data. We built in some DevOps, and we’re taking advantage of our own technology to do that.

Like I said, what we are really trying to do is build good strong teams, and so that’s something we’re very focused on. So one of the pilots we’ll probably be doing shortly is a team-based hiring model where a team actually hires for themselves.

We’re looking for folks who are just really passionate about solving digital government. And I think that’s the main focus of us is really just being passionate about the mission we’re trying to do. We’re trying to help the government be that digital agency. Looking at the interactions we have with some of these Silicon Valley apps, how come we don’t have that interaction with the government? That’s what we want to create. So we’re looking for people who are passionate about our mission and willing and interested in solving problems. Hopefully we can build a great team.

Weeks: Excellent. Thank you.

If you loved this interview and are looking for more great stuff on Rugged DevOps, I invite you to download this awesome research paper from Amy DeMartine at Forrester, “The Seven Habits of Rugged DevOps.”

As Amy notes, “DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros’ expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps.”

August 30, 2016
DevOps Leadership Series: Gov Does DevOps (Part 2)

During my second day at DevOpsDays DC, I had the opportunity to catch up with a couple more industry thought leaders. First up, John Willis, (@botchagalupe) DevOps Days core organizer and co-author the upcoming “DevOps Cookbook”.

In this episode of DevOps Leadership Series, John highlights his first introduction to DevOps and the journey it has taken him on, and what about the current state of DevOps makes him excited. He says DevOps is really happening everywhere. Today large organizations and CIO’s of government agencies are now experimenting with DevOps at all stages. He references legacy companies that do not necessarily have “aggressive IT”, and how they now want to share their DevOps experiences.

https://www.youtube.com/watch?v=jU1O2bCU_hc

After that, I had the chance to chat with Leon Fayer, (@papa_fire) Vice President at OmniTI, following his DevOps Days DC presentation. Leon illustrates the things he found most interesting from the DevOpsDays Conference. He noted that a lot of people have started to pay attention to the monetary strategies, and how to monitor from top-to-bottom in a DevOps fashion. Leon was happy to see that people are now more generally interested in how to generate and collect data more accurately to solve their problems.

https://www.youtube.com/watch?v=kx4Y3P3Njro

June 17, 2015