devopsy.kinsta.cloud

Category: DevOps Practice

  • The Great Market Convergence is Coming

    The Great Market Convergence is Coming

    When you look around tech markets—almost any tech market—you can see an obvious overlap. Tons of markets that are complementary and that take different approaches to solving the same problem, or markets that have grown into each other over time.

    This happens in highly active marketplaces. People have great ideas and implement them but someone else thinks a different approach is warranted, and the two don’t directly compete until they have to. There is competition within a market, and then there is competition of markets. I think we are about to see a massive increase in competition of IT and, specifically, DevOps markets. I admit to trying to write this without veering into topics I’m currently covering, so bear with me.

    Let’s take dynamic app protection as a topic. You can do that through your DevSecOps vendor, you can do that through your WAF vendor, you can ask around about AAP vendors if your WAF vendor isn’t one, you can do that through a variety of plug-ins and tools offered for web hosting software like Apache or NGINX … Indeed, IDP is actually just attempting to protect your apps dynamically. I’m sure I’ve missed several other markets that have rushed to fill the gap where live applications were getting nailed by hackers. I chose one market, but pick one in DevOps and start digging. You’ll find the same crazy amount of overlap.

    The thing is, we don’t need fifteen solutions to one problem; we need one or two—a handful at most. And combining the fact that most of these markets are way overdue for consolidation (considering the stresses the global economy is placing on trade) and I suspect we’re going to see a fair amount of that consolidation happen in a relatively short period of time.

    This matters to you for a couple reasons. First, if you think one approach to solving a problem is significantly better than the others currently available, your organization should be investing money there by paying for products and services. “More of us should have bought Betamax” is not a great look. Second, someone needs to keep an eye on how these changes impact the IT environment. If one of your DevOps tool’s vendors is in a merger, inevitably one of the merged organizations takes over and everyone does things their way. That’s not necessarily bad, but can be a scary time if, suddenly, you get an email about a change to a system that’s core to your department. Be aware of what’s going on out there and at least have thought about how you would respond if apps—or an entire market—disappear or are significantly changed.

    And, honestly, because it is the apps we are always talking about—DevOps, mostly—that are most likely to converge with a nearby market, it can seem huge. But remember that the core systems went through this long ago, and while they may go through another iteration of “merge this with that” in the future, it won’t be a huge volume like what the currently very active and young markets appear to be about to suffer.

    So don’t worry too much about which make/gradle/whatever variants you are using or what languages, which overlap incredibly, but tend toward niches. Focus on market spaces still growing and expanding. Know what might be at risk, and have awareness of whether you need to care today or you still have time before the market in question implodes or merges with nearby ones.

    And own it. We both know that when the time comes, whether you were prepared or not, you’ll rock a solution out—because that’s what you do, and there is no one behind IT to hand off to. So keep making your organization look great and just stay aware of the wider IT market trends so you’re not caught completely off-guard.

  • How to Avoid Crappy Culture and Keep Engineers Happy

    How to Avoid Crappy Culture and Keep Engineers Happy

    Software engineers know how much they are worth—everyone wants them and there aren’t enough of them to fill that demand. While the need for tech talent is greater than ever, the pandemic has shifted priorities for many engineers. And they’re not alone. It’s a reality shared with millions of others in a variety of fields around the world.

    If the so-called Great Resignation has taught us anything, it’s that everyone is tired of crappy workplace culture. People are beginning to question why they’d stick around at a place where they don’t feel valued and supported. And for those companies not investing time and energy into their culture, good luck holding on to software engineers. They know their value!

    In this uber-competitive environment, the best way to keep software engineers happy and make the company a desirable place to join may seem counterintuitive, but hear me out: Just back off. Trust engineers to do their jobs well.

    Put in place the building blocks for a supportive environment where people feel comfortable taking risks and voicing their ideas—even contradictory ones—in positive ways. Then, get out of the way. Leaders have a responsibility to make these values stick and to hire people who fit them, even if that means not always picking the most qualified person available.

    Don’t Be Afraid of “Dumb Ideas”

    I’ll be the first to say it—I have a lot of dumb ideas. I’m thankful to have worked in places that encouraged me to share those ideas. It taught me to focus on building a company culture that values openness to new thought processes. Because you never know when something that might seem like a dumb idea could actually become the next big thing. 

    No one wants to feel shamed for speaking their mind. Encouraging people to share their ideas, even those that aren’t fully developed, fosters a sense of trust in the team and can lead to greater creativity and innovation. It also makes the company stronger by reducing stress on employees and avoiding negative outcomes like burnout and other health concerns that may ultimately lead to turnover.

    Getting to this point of psychological safety in the workplace is not an easy process. It has to be highly intentional given the standards that are often at play in the professional world. For too long, people have been thought of as replaceable cogs in an organizational structure, not as humans with unique and valuable skills, talent and shortcomings.

    It’s about recognizing that people are human and often want to feel authentic in how they work. That’s why it’s valuable to talk about new ideas to improve the company and its products, as well as talking about how you talk; set a defined standard for how important topics are discussed as a team. 

    Fostering a supportive culture doesn’t mean there aren’t disagreements. But there are ways to offer contrary opinions without being a jerk about it. This mutual respect between engineers allows people to look at things with a critical eye without hurting anyone’s feelings.

    At CodeSee, we do a weekly product review. And this often leads to statements like “the product does this, but I wish it did this.” This isn’t a criticism of anyone, and everyone knows that. It’s everyone helping each other in the pursuit of making things better.

    Don’t Hire Assholes

    Instilling these values in a company is a rewarding experience that allows engineers to flourish. But none of that matters if they don’t also show up in the hiring process. The same things we place importance on in our day-to-day operations are top priorities in hiring: Humility, trust and self-reflection.

    We’d rather have a decent engineer that fits our culture and embodies these values than a great engineer that isn’t supportive of others. We don’t subscribe to the idea that it’s OK to treat people like dirt just because you’re a genius.

    These ethos show up in the questions asked to prospective employees. Two of our go-tos are fairly simple. We often ask interviewees to define three strengths and weaknesses. It’s a number that requires some real introspection and helps us see if a person knows where they could stand to improve and have identified personal growth strategies. We’ll also ask, “When we call previous managers, what will they have to say about you?”

    These questions are meant to tease out some self-reflection, naturally. If they can put themselves in the shoes of their past managers, that shows a high level of self-awareness and empathy.

    Surprisingly, these questions have gotten under the skin of more than a few applicants. I’ve actually had people yell at me, the person doing the interview, because they’ve been made to feel uncomfortable. Obviously, that kind of response told us everything we need to know.

    Tools for Success

    Institutional knowledge is an asset to any team. However, it puts additional strain on senior engineers, who are already stretched thin, to hold that information in their head and teach/explain it to less-experienced colleagues. 

    This is something that needs to change to free up senior engineers to focus more on the big picture. We can solve that with tools to visualize code and by investing in models that provide these models to engineers to improve training. 

    Today, many engineers spend 60% of their time reading code. What if we could reduce that to 40%? Or 20% or even 10%?  If all our engineering talent spent 40% of their time doing other stuff, think about where we could be?

    However, productivity tools can be a double-edged sword. Instead of giving more freedom and trust to software engineers, some companies are using technology to wring every last ounce of productivity out of the workforce, efforts that are surely doomed to fail by encouraging talented engineers to start planning their exit.

    The bottom line is that everyone needs engineers. Losing at least some is inevitable. But if you are thoughtful about creating a supportive, innovative culture—and maintaining it—then it’s a lot easier to compete.  

    But it’s no simple task. Every decision you make—from who you hire to the products you build to the way you talk to each other in meetings—plays a role in shaping culture. Engineers will be compensated everywhere they go, so the best chance to get ahead is to build the best, least-crappy culture possible. 

  • Optimizing Security in Data Collection Processes

    Optimizing Security in Data Collection Processes

    How you collect and process your data has a direct impact on the security of that data. By following best practices for data collection and processing, DevOps professionals can minimize the risk of data breaches and other security threats. In this article, we’ll share some best practices and tips for optimizing your data collection processes for better security.

    Make use of rotating proxies

    One way to optimize data collection processes is to make use of rotating proxies. This will help to prevent your IP address from being banned or blocked by web servers. Rotating proxies will also help to increase the anonymity of your data collection process.

    Use a Secure Connection

    When you are configuring data collection processes, it is important to ensure that all data is transmitted using a secure connection. This can be accomplished by using TLS/SSL encryption for all communication channels. This will help to protect your data from being intercepted by third parties and will also ensure that the data remains confidential.

    Implement Access Control Measures

    Another important way to secure your data is to implement access control measures. This means that you will need to restrict access to the data to only those individuals who need it. You can do this by using role-based access control or by implementing a least-privilege model. By doing this, you can help to prevent unauthorized access to the data while still giving different business functions across the organization access to the data and tools they need to do their jobs well. 

    Use Data Encryption

    Another way to protect your data is to encrypt it. This means that the data will be converted into a coded format that can only be decrypted by authorized individuals. This can help to prevent unauthorized access to the data and can also help to ensure the confidentiality of the data. There are a variety of encryption methods available to you, including public key encryption and symmetric key encryption.

    Make use of Data Loss Prevention Measures

    Another way to protect your data is to make use of data loss prevention measures. This can help to prevent accidental or unauthorized deletion or modification of the data. Data loss prevention measures can include things like setting up auditing and logging, implementing data recovery plans and using access control measures.

    Create Backup Copies

    It is also important to create backup copies of your data. This way, if the data is lost or corrupted, you will still have a copy that you can use. You should store the backup copies in a secure location and you should encrypt them to help protect the data. The reason backups are such an important part of modern cybersecurity comes down to the threat of ransomware and the rise of cyberextortion.

    Ransomware groups around the world have been increasingly targeting businesses of all sizes in recent years in an effort to extort money from them. And while some businesses have been lucky enough to avoid being targeted, others have not been so fortunate. The end result can be a total loss of reputation and even bankruptcy.

    Monitor Activity

    Another way to help secure your data is to monitor activity. This means that you will need to track who is accessing the data and what they are doing with it. This information can be used to identify unauthorized access and can also help you to determine if the data is being used appropriately.

    Restrict Physical Access

    It is also important to restrict physical access to the data. This means that you will need to ensure that only authorized individuals have access to the physical location where the data is stored. This can help to prevent unauthorized access and can also help to protect the data from being damaged or destroyed.

    Destroy Data When no Longer Needed

    When you no longer need the data, it is important to destroy it. This means that you will need to delete the data from all storage devices and ensure that it cannot be recovered. This can help to prevent unauthorized access and can also help to protect the data from being used inappropriately.

    Implement Security Policies and Procedures

    It is also important to implement security policies and procedures. This means that you will need to develop policies and procedures that govern how the data is to be used and accessed. These policies and procedures should be reviewed regularly and updated as needed.

    Educate Employees

    Another way to help secure your data is to educate employees. This means that you will need to provide training on the importance of protecting the data and on the procedures that should be followed. This training can help to ensure that employees are aware of the importance of security and can also help to prevent unauthorized access.

    Employee education is your first line of defence against cybercrime because your people constitute your largest threat and cybersecurity vulnerability. This is not because of maliciousness or nefariousness, but simply because most people are not aware of many of the necessary cybersecurity best practices when it comes to data collection and handling.

    Conduct regular audits

    It is also important to conduct regular audits of the data. This means that you will need to review the data regularly to ensure that it is being used appropriately and that no unauthorized access has occurred. This can help to identify any security issues and can also help to ensure that the data is being used correctly.

    Use encryption

    As mentioned before, another way to help secure your data is to encrypt it. This means that the data will be converted into a coded format that can only be decrypted by authorized individuals. This can help to prevent unauthorized access to the data and can also help to ensure the confidentiality of the data.

    Implement access control measures

    Another way to help secure your data is to implement access control measures. This means that you will need to restrict access to the data to only those individuals who need it. This can help to prevent unauthorized access and can also help to ensure that the data is being used appropriately.

    Conclusion

    When it comes to data security, there are a number of steps that you can take to help protect your information. By taking the time to implement these measures, you can help to ensure that your data is safe and secure.

  • The Pros and Cons of Embedded SREs

    The Pros and Cons of Embedded SREs

    To embed or not to embed: That is the question.

    At least, that’s one of the questions that companies have to answer as they decide how to implement site reliability engineering. They can either embed SREs into existing teams, or they can build a new, separate team.

    Both approaches have their pros and cons. The right strategy for your company or team depends, of course, on your needs and priorities.

    What is an Embedded SRE?

    An embedded SRE works as part of a non-SRE team. They might join development or IT operations teams, for example—although they could also be part of quality assurance teams, security teams or any other unit within the organization that can benefit from their expertise.

    Embedded SREs are the opposite of creating a dedicated, general-purpose SRE team. In the latter approach, an organization hires talent that collaborates with non-SRE teams but who aren’t integrated directly into them.

    There are several other organizational models that fall somewhere between these two poles. But typically, the two main types of structures are embedded or dedicated teams.

    Advantages of Embedded SREs

    Embedding SREs into other teams offers several benefits.

    Arguably the most important advantage is that embedded SREs ensure the highest level of collaboration between site reliability engineers and other stakeholders. When they work directly alongside other engineers on a continuous basis, it’s a pretty safe bet that their perspective will inform every decision that the other engineers make. Separate teams will still generally collaborate with other teams, but not necessarily with the same level of dedication as embedded SREs.

    Embedded SREs are also beneficial for organizations where the site reliability engineering concept remains new, and where stakeholders still need to learn what SREs do and the value they can bring. It may be a bit challenging to get buy-in when you propose creating an entirely separate team; indeed, developers or IT engineers may even perceive such a team as a threat to their own jobs (even though in reality, of course, they complement other teams instead of competing with them). But when you add SREs to larger teams, it’s likely to be easier to get everyone on board with the concept and for other types of engineers to appreciate how SREs make their jobs easier.

    Finally, embedded SREs work well for organizations that are on the small side and don’t need an entire team. If you have just several dozen engineers on staff instead of several hundred, hiring a few SREs to embed into your existing teams probably makes more sense than investing in a brand-new team.

    The Challenges of Embedded SRE

    The embedded model has its drawbacks, too.

    One major risk is that the SRE’s influence within a larger team may not be strong enough to have a major impact on how that team operates. A single SRE working alongside a dozen developers or IT engineers, for example, may struggle to integrate site reliability engineering techniques and tools into the existing team’s workflow—especially if it’s a team that already has strong opinions about how it should operate.

    Along similar lines, a single SRE working within a team using the embedded model could end up being stretched too thin to do his or her job well. That’s especially true if other team members expect them to “own” functions that should really be a collective, team responsibility—like responding to incidents or using chaos engineering to test reliability. The latter processes may be led by SREs, but they shouldn’t be their responsibility alone, and the function can start to break down when other team members think of them as the person on whom they can simply “dump” all of their reliability-related tasks.

    Finally, the embedded model can be difficult to scale, which is one reason why it doesn’t work as well at large organizations. If you have dozens of different engineering teams, embedding an SRE into each one requires a lot of management effort. It also deprives the various SREs within your organization of the ability to work closely with each other. In that type of scenario, it probably makes more sense to create a dedicated team and have those SREs interface with other teams as needed, rather than trying to distribute site reliability engineers across the business by embedding them directly into other teams.

    Conclusion

    In general, then, embedded SREs make most sense for companies that have relatively small engineering organizations, and in which the concept of the function has not yet received complete buy-in. For large businesses that have numerous engineering teams, it will likely be easier to manage SREs by creating a separate team just for them.

    That said, your mileage may vary from the norm, and it may be worth experimenting a bit to determine which model—embedded or not—makes the most sense for your company. Keep in mind that you can also use a hybrid approach wherein you embed SREs into some teams, while simultaneously maintaining a dedicated team that leads site reliability initiatives for the organization as a whole.

    After all, site reliability engineering is all about thinking creatively and coming up with innovative solutions to challenges—even the challenge of figuring out how to structure the SREs within your organization.

  • What’s the Difference Between SLI, SLA and SLO?

    What’s the Difference Between SLI, SLA and SLO?

    If you’re just getting into site reliability engineering (SRE) or platform engineering, you’ve probably come across a bunch of new terminologies, like SLI, SLA and SLO. These benchmarks are commonly referenced in the day-to-day life of an SRE but may seem foreign to outsiders. So, what are the differences between these abbreviations?

    Well, one of the hallmarks of the SRE approach is to gauge performance and set targets for application metrics. These targets can be thought of as reliability thresholds. If a site’s latency reaches a certain point, for example, it may break the agreement the software provider has with its consumer.

    SREs create and monitor service-level performance benchmarks for all sorts of metrics, such as uptime, latency, error count, mean-time-to-recovery and throughput, among others. But not all these performance goals are part of public agreements. Companies usually take a more proactive approach to setting more internal targets to stay far away from breaking external agreements and user trust. It’s become a best practice to set these baselines to retain functional, reliable systems.

    As mentioned, the type of targets range from firm partner commitments to more internal goals. Below, we’ll attempt to define, in simple terms, the differences between service-level agreement (SLA), service-level objective (SLO) and service-level indicator (SLI). While these concepts share many similarities, it’s important to understand how they are applied in practice. Keep in mind that each company may have a nuanced understanding of these terms and may conceive and apply them differently.

    Service-Level Indicator (SLI)

    First off, service-level indicators (SLIs) refer to the actual metrics produced by software services. This is a direct measurement of a service’s behavior. These are the real numbers that indicate overall performances, such as error rate and latency over time.

    One example of SLI would be the number of successful requests out of total requests over a one-month period. Say an engineer used an application monitoring tool that ingests data from production logs. This data showed that out of one million requests made, ten requests failed. An SLI for availability would thus be 99.999% uptime.

    Service-Level Agreement (SLA)

    Second, most technologists are familiar with service-level agreements (SLAs). Even if you’re not, you’ve likely agreed to many throughout your history as a digital user. SLAs are like a pact between the software provider and the software user or client. These binding commitments often note availability expectations that must be upheld. SLAs may also include responsiveness to incidents and bugs. It depends on the contract, but if an SLA is broken, some kind of penalty may be incurred such as a refund or a service subscription credit.

    These days, the average business relies on many cloud-based SaaS, PaaS and IaaS. Just to initiate something as simple as an online payment might require hitting multiple remote servers. But if all these services just went offline whenever they felt like it, our digital operations would come to a grinding halt. SLAs are thus necessary for enterprise software contractual agreements to ensure both parties meet specific standards.

    Once an engineer tracks SLIs and has an idea of typical behavior, they can then set an SLA that makes sense. If we take the example above, an SRE may consider guaranteeing a lower threshold than what is being actively monitored. This would ensure that application performance, at its current pace, doesn’t break any legal or contractual obligations. In this scenario, perhaps the SLA mentions an availability objective: No more than 100 requests fail per one million requests made. This would essentially equate to 99.99% uptime.

    Service-Level Objective (SLO)

    Lastly, service-level objectives (SLOs) are similar to SLAs but explicitly refer to the performance or reliability targets. An SLA may refer to specific SLOs. Or SLOs may be tracked just for internal purposes. As Google described, “the availability SLO in the SLA is normally a looser objective than the internal availability SLO.”

    You don’t want the end users to be the first people clamoring about a 400%+ latency rise in your mobile web apps. Thus, SRE teams typically keep a close eye on performance to ensure they never even get close to breaking SLAs. Some do this by setting and monitoring internal baselines that are more ambitious than the SLA threshold.

    If we take our example above, if an SLA guarantees a service uptime of 99.99%, the business may set an internal target of 99.995%. In other words, for every one million requests, no more than 50 should fail. If software systems aren’t hitting these marks, it’s a sign that the company must reevaluate designs and search for bottlenecks. Or, perhaps engineering teams have goals to improve the average downtime for the next quarter—an internal objective could be set at a higher standard than the currently perceived performance.

    As the Google Cloud blog described, “every service should have an availability SLO—without it, your team and your stakeholders cannot make principled judgments.” It’s a good reminder to be modest in your reliability commitments, they added, as consumers will come to expect this level of performance. On the other hand, setting more ambitious internal performance targets has the benefit of delivering a better result than the agreement on paper, increasing a software service’s competitiveness.

    Real-World Performance Benchmarks

    Keep in mind that the above numbers are simply for demonstration purposes. One interesting resource for real-world figures is API.Expert, a service that queries popular APIs and posts weekly performance statistics. Since APIs are the heart of many UI-based platforms (and our digital economy at large), these benchmarks stand as a good indicator of average uptimes and latencies in the industry.

    For example, at the time of writing, API.Expert’s Enterprise APIs collection ranked Microsoft Office365 and Pivotal Tracker at the top, both with a 100.00% pass rate and a 220 ms and 248 ms latency, respectively. On the other end of the spectrum, Docusign is at 99.93% with an 877 ms latency and Box is at 99.99% with 414 ms latency.

    SLIs, SLAs and SLOs—Oh My!

    Although it may sound good to an unpracticed ear, an SLA of 99.99% still equates to 52 minutes and 36 seconds of downtime per year. That’s nearly an hour of downtime in which customers are left scratching their heads or, worse, searching for other options. For traumatic health care situations, a loss of connectivity could be a matter of life and death.

    Although creating SLAs and SLOs is important to gauge system health, the reality is that it can be challenging to track and enforce them. “These agreements—generally written by people who aren’t in the tech trenches themselves—often make promises that are difficult for teams to measure,” according to the Atlassian knowledge center.

    In summary, SLIs demonstrate the real behavior of software systems. These metrics inform the creation of SLAs, which must be met to ensure B2B agreements. These SLAs often reference particular service-level objectives (SLOs) that must be met, which usually give more breathing room around SLIs. Lastly, in a digital economy with accelerating digital expectations, it makes sense to monitor internal SLOs and improve baselines over time.

  • Shift Left is Only Part of Secure Software Delivery

    Shift Left is Only Part of Secure Software Delivery

    We’re living in the age of accelerated consumption and delivery. You can get a seemingly infinite selection of products delivered to your door within two days, for free, from thousands of miles away. You can access an endless variety of services online within mere seconds: Movies, music, games, education and even health care. These modern marvels are afforded to us by relatively recent advances in software development, delivery and operations. 

    Among those advances are the adoption of DevOps practices, continuous integration and continuous deployment (CI/CD) pipelines, and automating everything from infrastructure provisioning to testing and quality assurance. These advances enable developers to deliver software faster and think about security earlier in design and development to ship safer software.

    What is Shift Left” and Why Does it Matter?

    “Shift left” is a philosophy for addressing the pitfalls of traditional waterfall-style development. In waterfall, a number of IT teams work over the course of many weeks or months, culminating in a big and beautiful software release. Of course, people make mistakes, forget things, and maybe don’t communicate well with each other. When it turns out that application elements don’t interoperate well, the teams frantically attempt to fix what’s possible and jettison what isn’t, with a very serious deadline staring them in the face. Products inevitably ship late, still somewhat broken, and with security bolted on as an afterthought.

    To shift left means to introduce security process and tooling earlier into the design and development phases. Security should be just as critical as functionality and quality. It’s a great idea because discovering a critical, build-failing vulnerability in a critical dependency after you’ve built dozens of functionalities on top of it hurts a lot more than if it’s found before you build anything at all. Maybe fixing the problem is as simple as using a newer version of a dependency, but now you have to make sure everything you’ve customized still works as intended.

    There are a number of forms of application security testing (AST) that aim to detect code-level issues early in the systems development life cycle (SDLC). Two common types of AST when shifting left are software composition analysis (SCA) and static application security testing (SAST). Put simply: SAST tests the custom code your developers write; SCA tests dependencies you include in code. 

    Assessing the security of modern applications requires both testing types. If you believe shift left means not only finding problems but also fixing them, then developer-friendly tooling is necessary. Exactly what that means can be subjective depending on developer workflows in the organization, but at the very least, developers need self-service, seamless integration into their existing CI/CD toolchains and actionable results. Because most developers aren’t security experts, security findings must prioritize the riskiest problems, point to the source of the issue and provide contextualized fixes, rather than making the developer hunt for them on their own. 

    Augmenting AST With Runtime Intelligence

    One challenge with doing a lot of early testing—especially security testing—is that the more you test, the more findings you discover. Unfortunately, some of the results will be false positives and others may be true positives that pose a relatively low risk. Development teams with a high degree of security expertise must sift through and vet findings to prioritize flaws that are severe and actionable, but such practitioners are in short supply and this type of work is tedious.

    Addressing the pitfalls of security testing isn’t simple, and it involves gathering as much contextual information as possible to begin to reason about the risk associated with each finding, usually expressed by what exploitable code is actually exercised. 

    Reasoning about vulnerability risk is not easy. We try to rank the criticality of issues to decide on an action. We can split the problem into two parts: 

    • The context of the vulnerability itself includes metrics of severity, like the CVSS score, and threat intelligence, like whether exploits are publicly available or attacks are prevalent in the wild. This information is readily available from scanning tools and third-party sources like NVD, MITRE and CISA.
    • The afflicted asset’s environmental context, however, is specific to your application and your infrastructure. Only your organization can really fill this part in. For example, is the impacted application protected by mitigating security controls? Is it exposed to the internet? Does the vulnerable component even get used at all? This is difficult or impossible to know until the application is running in its intended production environment. 

    Shift left is a way forward for modern cybersecurity, but the approach can be vastly enriched by pulling in runtime intelligence. This helps to prioritize what to fix first, removes friction, saves cycles of numerous IT teams and reduces your organization’s vulnerability risk.

    What Happens After Release?

    The whole point of shift left is to release secure software faster, but what happens next? Your flawless, beautiful code is delighting your customers with its stunning feature set and impressive performance. Then, on some idle Friday afternoon, right before what should have been a long holiday weekend, you get hit with a horrendous zero-day that you could never have predicted even if you tried. 

    When that happens, you need to respond by quickly identifying all of the affected workloads running in your environment and prioritizing them for remediation. You may need to take those workloads offline to patch immediately, limit network access to them, or otherwise mitigate the problem if a patch is either unavailable or taking too long to deploy. The point is that in real life, shift left is only one part of a successful security program, and the protective, detective, and remediation activities for your production environments on the right are also critically important to your organization’s safety and success.

    Conclusion

    Shift left alone was never going to solve all security problems, but it does create foundations for reliable, fast and secure software delivery pipelines. Security and development teams alike often suffer from a deluge of noise and few useful signals. What they really need are consistent and prescriptive ways to make sound decisions about risk. Gathering context from multiple sources through multiple layers of the stack, including runtime signals, substantially improves the quality of the information that DevSecOps teams receive from their tools and must act on.

    Effi Goldstein, director of products at Snyk, co-authored this article. Check out Sysdig and Snyk’s 2022 Container Security Trends: Expert Panel Livestream for more information. 

  • The Problem With Security

    The Problem With Security

    As an IT professional with decades of experience at every level of technology and organizations—from cell phone prototyping to banking enterprise architecture, from entry-level to CTO—I can state definitively that I have spent an inordinate amount of time dwelling on and dealing with security. And I know I’m not alone. This is a common theme. You could almost make a sitcom out of it. “What are your developers doing?” Uhhh … writing code? “That’s security!” I could go on, hitting on everything from users to cloud instances and you could fill in the “That’s security!” punchline every time.

    That’s a problem. But not in the way you might think. I spent many years thinking I was somewhat unique in that I always rose to the occasion and approached security issues head-on. Most of the organizations I’ve worked for didn’t have dedicated security (I’ve probably written about that before), so some of us rose to the occasion.  Nowhere near all of IT, but enough to feel like we were protected.

    I now realize that I’m not all that unique; that more of us are doing the “learn security to be secure” thing than are not. I see it in modern organizations all the time. And I think that the real problem is this: Most of these people, like me, did not see themselves as security staff—but they are. Most organizations don’t recognize them as security staff—but they are.

    We’ve long harped on cultivating security recruits from this pool, and I’m now going to suggest that you go even further. Make these people part of your planning. If you know that person X on the ReallyCoolApp team is ferreting out security stuff, ask them to just send an email to update InfoSec once a week. Don’t make it a huge burden, and don’t try to force them into a pure security role if they don’t show the aptitude and interest (you really don’t want unwilling security staff) but make use of their talents and drive. Certainly, if this person is doing a good job of making certain ReallyCoolApp is mostly locked down and DevSecOps tools are running against it for a secondary validation, then your limited InfoSec resources can be directed elsewhere.

    Replicate this throughout the organization, and “That’s Security!” becomes a benefit rather than a hassle to try to lock down. You already do some of this—since the automation of passwords, most identity management is handled by operations staff, with security merely providing policy and being available to ops if needed. I’m simply suggesting that you expand this type of approach to include every IT department (and some non-IT departments in distributed environments). Don’t recruit everyone to be InfoSec, but recruit them to do a little bit of InfoSec in their area of specialty.

    People have always said “Security is everyone’s problem,” which is true, but it’s far too vague to be useful. This is a workable intermediate. “Security is every team’s problem” and recruiting the people on each team most interested in InfoSec to formalize their involvement is a better approach.

    And keep rocking it. We need security everywhere, because ne’er-do-wells have made everywhere a part of the InfoSec battleground. So recruit people everywhere without moving them from their current roles/responsibilities. Don’t let the sweet systems you’ve nursed through the years get whacked by attackers when a little recruiting and nurturing will fill the gap.

  • How Engineers Can Contribute to API Security

    How Engineers Can Contribute to API Security

    I’m an engineer, not a security professional. I’ve worked at several startups over the years and have discovered ways to shift my engineering mindset to include a security focus and to incorporate security into my coding practice. 

    I’ve found that when you work with others, an engineering organization can significantly contribute to API security. But how we think about software makes it difficult to get started. There is an alternative perspective—a road not taken—that must be considered when it comes to securing endpoints from threat actors. 

    The Road Not Taken

    Engineers create and optimize applications to solve problems for real users. Engineering time is often in short supply, so we optimize the speed at which we can write those applications. To speed up our processes, we simplify problems and find reusable patterns or techniques. Add a bit of glue, and you have software.

    This perspective is constructive. Thoughts come first and software comes next. And when refactoring existing code for software, we need to think this way. But this utility comes at a cost: If the software works, we only see what it should do. The real story behind the scenes can be much more complex.

    A few years ago, I worked with an engineer on new endpoints for an authorization API. We were working out an issue in the implementation where I could call the endpoints out of order and bypass a challenge. We worked through the example, and he said, “But they are supposed to be called in 1-2-3 order.”

    Attackers don’t share this perspective. Instead, they want to manipulate the software to act in their own interests. They ask, “What is possible?” instead of “What is it supposed to do?” and find ways in which the system fails. To an attacker, the glue is more interesting than the design.

    While probing, an attacker may be thinking some of the following questions: 

    “The password reset endpoint is on v2? Is v1 still running?” 

    “What happens if I skip the verify endpoint?” 

    “What happens when I pass null in the challenge? Empty string? SQL injection?”

    As creators, we’ve converged on this constructive perspective. To build secure software, we need to consider the road not taken and incorporate greater security into our applications.

    Playing Defense

    When you are defending software, your limited time is a disadvantage. There are many engineers—and some have been writing software for many years—which means that you have a lot of code to defend. Each application depends on hundreds or thousands of open source projects, from libraries to deployment tools to server operating systems. 

    Security tools can help you think from both perspectives, especially those that automatically discover endpoints and track response statistics. You can use this data to understand how code maps to your public API surface. 

    In addition to tools, here are several suggestions for on the ground techniques that you could try out.

    Read the Code

    Pick an application, find out where the authentication and authorization code is and read it. Even if you aren’t a strong coder, you can learn something interesting. Better yet, try and take the attacker’s perspective and look for ways you can exploit the code and make it work for you. For instance, you may see the following:

    “SQL injection might be possible in this area. I might be able to extract information about other accounts.”

    “Authorization is complex and confusing in this area. I might be able to use my standard user account to execute actions on other accounts.”

    “Logging is very weak in this sensitive area. I might be able to hammer this endpoint without being noticed.”

    “Code or comments indicate the developer assumes the application is secure for a particular reason. Can I break those assumptions with an external request?”

    The knowledge gained in performing an exercise like this is priceless. You can use it to guide the rest of your API/AppSec work (pentesting, scan remediation, etc.). Further, you’ll often find logical vulnerabilities that scans couldn’t detect.

    Read Code Reviews

    Engineers make decisions in pull requests (PRs). Content in past PRs and review comments have tons of context and can be used to help inform your decision-making. Look through the history, then think through some security-focused review questions:

    “If the code tries to enforce security constraints, can I think of a way around them?”

    “Are there obvious security mechanisms that should be part of this change but are missing?”

    “Would unusual attacker behavior appear in logs?”

    Examining these reviews and understanding the broader context can provide you with valuable information. You might find a sensitive action without any authorization logic, ways a user account can access data owned by other users, or a path through the password reset flow without a valid challenge token.

    Work With Engineers

    If your engineers are working on security-relevant code, ask for some time to talk. Find out what they are working on (and why). Help them think like an attacker, and explain some of the techniques an attacker might use to break their app. And remember to take the road not taken while you work to ensure the security of your applications. 

  • Nurturing a Developer-Centric Culture

    Nurturing a Developer-Centric Culture

    The tech world is constantly, rapidly changing and many tech companies are growing up. They are adding to their product portfolios, increasing the number of employees, servicing more users, expanding global reach and more. Typically, when an organization grows up, it doesn’t happen overnight. It takes a lot of hard work, onboarding new talent and a collaborative culture to bring business ideas to fruition. This growth often involves integrating new roles, methodologies and cultures to help a business succeed. One of these new cultures to emerge was DevOps.

    DevOps evolved from the need to take a new approach to how developers work. As technology advanced, developers sought new ways to approach their strategies and DevOps emerged as a way to explain this process. It stems from the concept of shortening the systems development life cycle and focusing on continuous delivery with high software quality. This approach empowers developers to perform end-to-end tasks—from development through testing to deployment and operations in production. Many now recognize that DevOps is a set of practices that combines software development (Dev) and IT operations (Ops) and what eventually evolved is a culture that supports developers in the ability to fulfill the jobs at hand. 

    Built on the foundation of engineering culture and dedicated to empowering engineers, the DevOps culture is intertwined with the concept of continuous delivery and gives developers the responsibility and ability to deploy their products end-to-end. This means developers can move fast and work independently. While DevOps usually refers to the operational aspects of a software development life cycle, it can be beneficial to take it a step further. In a truly developer-centric culture, the engineer is involved in the entire life cycle of software from inception, product, QA all the way to operating it in production. For example, an engineer who is working on their product can progress quickly because they can operate as though they are within their own startup. Engineers can release a feature in a few minutes once they develop it. However, to thrive, this culture needs an entire organization’s support. 

    Being involved does not necessarily mean being fully responsible. Product managers, for example, are responsible for the product, but it’s essential that the engineers are involved in the product definition. This goes for all aspects of the product development life cycle, where engineers need to be involved in everything.

    There are effective ways leadership can support a developer-centric culture. The more this culture is embraced, the more successful a company’s and the industry’s engineers will be. There are a number of ways organizations can effectively support DevOps and build and sustain a developer-centric culture.

    1. Understand developer-centric. Without understanding what DevOps is, it’s hard to nurture its growth. It’s important to keep in mind that it’s a concept of empowering developers to seamlessly work through the SDLC. Many view DevOps as a role or a job description with a specific set of responsibilities, but it’s much bigger than that. It’s a culture. 
    2. Set guidelines, not rules. Developer-centric cannot and will not fit into a box. Most companies create checklists of to-do’s for processes but those ultimately fail. Instead, companies should focus on creating guidelines to encourage creativity and critical thinking so engineers can effectively plan, assess and reflect on every portion of their contribution to the final product and the process. It’s key to have engineers that are proud of their work and proud of what they’re working towards.
    3. Nurture engineers to take ownership and responsibility. Involve engineers at every step of the product life cycle, starting from product inception. C-level leadership and other managers need to trust the team to think proactively and execute effectively. The team was hired with a purpose; don’t micromanage and accept that some developers might do things differently than others—there’s not necessarily only one “right way.”
    4. Foster a collaborative environment. Collaboration between engineers—as well as between engineers and other departments—can spark creativity and solutions to problems or create new products entirely. Furthermore, hearing and respecting different viewpoints can inspire positive change throughout the entire organization. 
    5. Embrace mistakes and encourage continuous improvement. There’s no right or wrong way to do things, so it’s important to empower engineers to be creative and have the flexibility to make mistakes. Their mistakes can lead to insights and knowledge that might not have been discovered otherwise. Pain equals growth. Making mistakes and taking risks make individuals, departments and an organization better.
    6. Empower engineers both inside and outside the organization. As companies grow, their responsibility to the tech community grows, too. The term “empowering engineers” should be seen as a responsibility both inside and outside an organization. The stronger and more empowered engineers are, the stronger and more successful the industry will be as a whole.

    In conclusion, as companies grow up, it’s important to be fluid with the roles and cultures that morph to accomplish goals and objectives. Developer-centric is an example of when culture evolves to accommodate and tackle a new demand within a business. If you think of a process of software development at an assembly line, we take the assembly line and make it a circle so that developers that write the software are responsible for the whole life cycle from writing it to operating it. It’s a dev-centric culture. Now that dev-centric is here, it needs to be nurtured and supported by leadership, both inside and outside the company to thrive.

  • The Rising Demand for DevSecOps Talent

    The Rising Demand for DevSecOps Talent

    Demand for DevSecOps talent and skills is growing even faster than the demand for DevOps.

    The global DevSecOps market is anticipated to register a CAGR of 32.2% over the next few years, and revenue is projected to grow from USD $2.55 billion in 2020 to USD $23.42 billion in 2028. No wonder the number of job postings for DevSecOps is rising.

    Many organizations are implementing DevSecOps in their workflows. Some famous names include Netflix, Etsy, Twitter, Facebook and Google. Those who are already well-versed in DevOps and who have a security background will have an advantage over others in this field when looking to get hired.

    However, it is not just about the tools and technologies—there is a lot more to DevSecOps than that. It is also about culture and new ways of working; about automating the security process and making security a part of the continuous delivery pipeline.

    What is DevSecOps? What’s inspiring top business leaders to invest in this domain? And how will acquiring these skills help you in your career?

    What is DevSecOps?

    DevSecOps is a set of practices that combines the speed and agility of DevOps with security controls. It aims to help organizations deliver software faster while ensuring that the applications are secure. The term ‘DevSecOps’ was first coined by John Willis and Damon Edwards in 2009. They defined it as:

    “DevSecOps is the practice of operations and development engineers participating together in the entire service life cycle, from design through the development process to production support.”

    Later, Gartner offered the following definition of DevSecOps:

    “DevSecOps is the dynamic software development approach that incorporates security as a critical element of the application delivery process from design to production.”

    In simple terms, DevSecOps can be defined as a culture and set of processes that help organizations to deliver secure software faster. As the demand for DevSecOps professionals rises, it is an excellent time for software engineers, developers and DevOps professionals to consider a career in this domain. Here are some of the top reasons why you should consider a career in DevSecOps:

    1. It is a rapidly growing field

    The DevSecOps market is anticipated to grow at a CAGR of 32.2% between 2020 and 2028. This rapid growth is creating a lot of job opportunities for DevSecOps professionals.

    2. It is an interdisciplinary field

    DevSecOps is not just about the tools and technologies. It is an interdisciplinary field that requires skills in development, operations, and security.

    3. High-paying roles

    According to a recent survey, the average salary of a DevSecOps professional is $123,000 per year or higher; much higher than the average salary in other fields.

    4. Work with the latest technologies

    DevSecOps is all about automating the software delivery process. It requires a deep understanding of the latest DevOps tools and technologies as well as automation, AI and machine learning.

    5. Stay ahead of the latest trends

    As a DevSecOps professional, you need to be on the cutting edge of the latest trends in the industry. A role such as this helps you keep your skills up-to-date and stay ahead of the curve.

    Why the Major Investment?

    The following factors are inspiring top business leaders to invest in DevSecOps:

    1. The need for speed

    In today’s fast-paced world, businesses have to deliver software faster. DevSecOps helps organizations speed up the software delivery process while ensuring that the applications are secure.

    2. The need for agility

    DevSecOps enables organizations to be more agile and adapt to ever-changing market trends. The ability to quickly deliver software helps organizations stay ahead of their competition and do so while reducing the risk of data breaches, cyberattacks and malware.

    3. The need for security

    As cyberattacks continue to rise, businesses are investing in DevSecOps to ensure that their applications are secure. DevSecOps helps organizations detect and fix security vulnerabilities earlier in the software development process when they are easier and less costly to fix.

    In 2021, there were around 414,02,809 cybersecurity incidents worldwide and that number will likely increase. With the help of DevSecOps, businesses can protect their applications from attacks.

    4. The need for compliance

    With the help of DevSecOps, businesses can automate the compliance process. It helps save time and resources otherwise spent on manual compliance processes.

    5. The need for cost savings

    DevSecOps helps organizations save money by automating the software delivery process. Automation helps reduce the need for manual labor, which can help cut costs.

    Professional Skills and Requirements

    Like any other field, DevSecOps also has its own set of requirements and skills you need to know to be successful. While many are specific to technology and tools, others are more general soft skills.

    1. The need for cross-functional skills

    Since DevSecOps is an interdisciplinary field, professionals in this role need to understand both development and operations. They also have to be familiar with the latest tools and technologies in software development, IT operations, security, CI/CD, containers and more.

    2. The need for collaboration

    DevSecOps is all about collaboration between different areas of the business and different teams. Professionals need to be able to work closely with other team members.

    3. The need for continuous learning

    As the DevOps and security fields are constantly changing, professionals need to constantly update their skills and be familiar with the latest trends in the industry.

    4. The need for patience

    The software delivery process is often slow and complex. It can be frustrating for professionals who are used to working in a fast-paced environment.

    5. The need for adaptability

    As the DevOps and security fields are constantly changing, professionals need to be adaptable. They should be able to learn new tools and technologies quickly and know how to apply that knowledge to solving technology problems.

    Getting Started

    If you want to become a DevSecOps professional, you need to understand both software development and operations. You also should be familiar with the latest tools and technologies. You can read other professionals’ blogs, take formal courses or attend a coding bootcamp, take courses at a community college or university or teach yourself. You should attend conferences and talk with other technology professionals to share knowledge. In addition, you need to work on your soft skills, including communication, collaboration, teamwork and leadership.