Category: DevOps Onramp

  • A DevSecOps Process for Node.js Projects

    A DevSecOps Process for Node.js Projects

    Node.js is an open source development platform for running JavaScript code on the server side. Node is useful for developing applications that require a persistent browser-server connection and is often used for real-time applications such as chat, social applications, or news feeds. 

    DevSecOps is a way for development, security, and operations teams to work together throughout the project life cycle. It is often implemented by adding security to existing DevOps teams and agile processes.

    The idea behind this collaboration is to incorporate security early in the development process. This is called shifting security left. As security moves to the “left” of the development process, application vulnerabilities and bugs are discovered and addressed faster. This leads to more secure product releases with a reduced need to apply patches and troubleshoot in production. Applying DevSecOps to Node.js development can improve both the efficiency and security of Node.js projects.

    Common Node.js Attack Vectors

    Here are a few cyber threats that affect many Node.js applications.

    Cross-Site Scripting (XSS)

    XSS attacks inject manipulated client-side JavaScript code. Web apps are vulnerable to XSS if they don’t validate inputted hostnames from the DNS. Attackers can use forms to enter malicious scripts and hack into the system. In some cases, entering code into the filename can trigger cross-site scripting in the target browser. 

    A hacker can use your file upload form to hack into your system. There are many ways for this vulnerability to lead to a system compromise. Though usually mitigated by modern front-end frameworks, in some cases it may happen that putting javascript into the filename will trigger XSS in a user’s browser, allowing the theft of session IDs. In other cases, an attacker might get sensitive server information by uploading a file to the Node.js server and running it by exploiting a path traversal vulnerability.

    Code Injection

    Developers are responsible for writing secure application code, but it’s impossible to fully guarantee the code’s security when using open source components. A code injection attack involves injecting malicious code into the target system and forcing the application to execute it, allowing attackers to access the codebase. Inadequate data validation often results in code injection vulnerabilities.

    Default Cookie Names

    Cookies help web apps identify specific users by storing user actions in the underlying infrastructure (e.g., an eCommerce shopping cart that remembers the items a customer selects). 

    Developers should use custom names for cookies (rather than defaults) to prevent attackers from identifying and targeting cookies and extracting data. 

    Brute Force Attacks

    Brute force attacks often affect Node.js applications. An attacker randomly generates millions of passwords to find one that works, logging in to the application. You can prevent brute force attacks by strengthening the authentication mechanisms, limiting the number of permitted login attempts from a single IP, and encrypting stored passwords (using bcrypt.js).

    Cross-Site Resource Forgery (CSRF)

    This type of session hijacking forces an authenticated user to execute malicious actions on the target web application. The attacker hijacks a legitimate user’s session to bypass security controls and change the application’s state. CSRF can force users to transfer funds or change their account details, potentially compromising the web application’s security. 

    Node.js Security Best Practices

    Implement Strong Authentication

    A compromised, weak, or incomplete authentication mechanism is a common vulnerability. Many developers think of authentication as something that “just works”. However, authentication can easily be bypassed by attackers if not implemented correctly.

    In Node.js, you have the option of using the default Node.js authentication solution. There are a few things to keep in mind: 

    • Do not use Node.js’s built-in password library when generating passwords, use Bcrypt. 
    • Limit failed login attempts and do not notify users if their username or password is incorrect. Instead, return a generic “Invalid Credentials” error. 
    • Have a robust session management strategy. 
    • Implement two factor authentication (2FA). This can be done using modules like node-2fa.

    Alternatively, use one of many third-party solutions available for application authentication, which may be superior to built-in Node.js functionality. 

    Avoid Errors That Reveal Too Much

    There are many things to consider when building error handling functionality: 

    • Don’t give the user any details they don’t really need. Never return the entire error object to the client. It may contain information that you do not want to disclose, such as paths, other libraries you are using, or secrets. 
    • Wrap the path in a catch clause so that Node.js doesn’t crash if the request triggers an error. This prevents the app from crashing continuously, allowing attackers to send malicious requests that cause the app to crash.
    • Don’t expose your Node.js app directly to the internet. Use a component like a load balancer, cloud firewall or gateway. This means DoS attacks will be one step removed from your Node.js application.

    Using Security Linters to Capture Vulnerabilities in Code

    Code linters help identify various issues in code before compiling. They let you detect the most common issues and ensure you follow best practices.

    Code linters have their own rules, which you can customize according to your requirements. Before using a linter, make sure you enable rules related to security vulnerability detection in the linter configuration.

    Server-side Logging and Monitoring

    A good logging library provides developers with powerful troubleshooting and monitoring capabilities. On the other hand, unnecessary or excessive logging affects application performance and consumes more resources. Therefore, you should only use reasonable log levels when deploying applications to production environments.

    When constructing log messages, they must be in a format that can be read and understood by humans and machines. Logging ambiguous messages can lead to misunderstandings among developers. An error message must not only describe the problem but also advise what the user can or should do to fix it.

    It is also important to ensure that your logging mechanism captures all relevant information such as IP addresses, usernames, and actions taken. 

    Never retrieve or store sensitive information in application logs. This violates compliance standards such as PCI and GDPR. However, there are times when you need to record this information for troubleshooting – in this case, it is recommended to mask or anonymize sensitive information before writing it to the log.

    Implementing Node.js Security in a DevSecOps Organization

    Node.js security best practices are well known and security awareness is growing among Node.js developers. However, many organizations don’t have a clear organizational structure for implementing development best practices. Here are a few steps your organization can take to transition a Node.js development team to a full DevOps model:

    1. Adding mandatory guard rails for developers

    For example, adding static application security testing (SAST) within the IDE and on every commit. Linters should be deployed centrally with a consistent configuration for all developers in the team.

    1. Automated security testing in staging environments

    During the CI/CD process, when a Node.js application is deployed to a staging environment, there should be automated tests that can identify security misconfigurations and integration-related vulnerabilities. Dynamic application security testing (DAST) tools are ideal for this purpose.

    1. Runtime protection

    No DevSecOps process is complete without a safety net to identify security issues in production. This requires the use of tools like web application firewalls (WAFs), which can detect and block malicious traffic, and runtime application self-protection (RASP), which can proactively respond to attacks. Both these tools provide feedback to DevOps teams to help them identify and remediate vulnerabilities in the underlying application.

    By implementing these three stages, you can be sure that Node.js security is not just an ad-hoc practice but a formalized, coordinated process involving developers, operations teams and security experts. 

    Conclusion

    In this article, I explained the basics of DevSecOps and how to secure Node.js:

    • Implement strong authentication—In Node.js, you have the option of using the default Node.js authentication solution.
    • Avoid errors that reveal too much—Don’t give users more details than they actually need when displaying errors.
    • Using security linters to capture vulnerabilities in code—Code linters help identify various issues in code before compiling.
    • Server-side logging and monitoring—A good logging library provides developers with powerful troubleshooting and monitoring capabilities.

    I hope this will be useful as you implement Node.js security in your DevSecOps organization.

  • State of DevOps Report Finds a Rise in Platform Engineering

    State of DevOps Report Finds a Rise in Platform Engineering

    The speed of software development velocity is increasing, and to enable this, platform engineering is emerging to evolve the DevOps practice. Interest in platform engineering is rising—so much so that Puppet emphasized platform engineering in its 2023 State of DevOps Report.

    A whopping 94% of respondents agreed that platform engineering is helping their organizations better realize the benefits of DevOps. The move toward platform engineering has increased over time, bringing benefits such as increased release velocity, reliability and efficiency.

    Below, we’ll highlight some of the key takeaways from the report. We’ll define what platform engineering is and highlight its benefits. We’ll also consider obstacles on the path to embracing a platform-based development culture.

    Platform Engineering Supplants DevOps

    DevOps has been in use for a decade or more at most institutions, maturing steadily with new automation and infrastructure abstraction. The latest evolution of DevOps is a practice known as platform engineering—51% of organizations have already adopted this practice within the last three years and 93% said it’s a step in the right direction.

    Platform engineering is the discipline of designing and building self-service capabilities to minimize cognitive load for developers and to enable fast-flow software delivery.

    In essence, platform engineering is the process of creating standardized, self-service infrastructure that helps developers get up and running quickly. These self-service platforms typically handle capabilities like deployment (43%), operation of infrastructure (43%), monitoring production applications (42%) and security and compliance (42%).

    These shared internal platforms are maintained by a central platform team that embraces a product mindset to iterate the technology based on user feedback.

    State of Platform Engineering

    Platform engineering is relatively new— 27% of platform teams have formed within the last two to three years. Notably, big tech companies hopped on the bandwagon years ago, and non-tech companies are now following. Regardless, 70% believed their timing was “just right.”

    The goals of a platform engineering team are numerous. According to the Puppet report, 54% said problem-solving is the top goal. This is followed by educating and empowering developer teams (47%), setting best practices (46%) and optimizing iteration (44%). A platform engineering team’s scope of work is primarily focused on automating workflows and building and managing infrastructure.

    The ‘correct’ number of internal platforms is a moving target. The majority of organizations (57%) have between two and four platforms, and 30% have five or more internal self-service platforms. This demonstrates that although platform engineering is somewhat centralized, it’s also flexible to the needs of an organization.

    Benefits and Hurdles

    So, what are some benefits as opposed to traditional DevOps? Well, the most common benefit is improved system reliability, at 60%. This tracks, as a standardized platform could reduce human error and missteps in the development workflow. Following this benefit is improving efficiency (59%), speeding up delivery time (58%) and improving workflow and process standards (57%).

    The most common driver of adoption is increasing the speed of delivery. Slight more than half of respondents (51%) said development speed has somewhat increased since the inception of the platform team, and 42% say it’s increased by “a great deal.” Other top drivers include the need to scale, free up engineering time, control infrastructure spending and avoid duplication across teams. Those with more mature adoption tend to be more satisfied with the result, indicating that results increased over time.

    Of course, no new technological introduction comes without its hurdles, and platform engineering is not immune from certain pain points. For example, 34% said that cycle time is slower than expected. This might be due to nascent industry adoption and a lack of best practices.

    Just under one-third (32%) also noticed resistance to platform team adoption, and 32% reported a lack of communication around changes. Organizations will likely have to increase transparency and reduce siloing to encourage productive platform adoption that satisfies the needs of various stakeholders.

    According to the report, product management skills can help realize these goals. Some of the most fundamental product management skills here include strong communication skills (61%), problem-solving expertise (60%) and the ability to foster collaboration (54%). Many companies will also have to convince senior management to invest in platform engineering—nearly half agreed that senior management doesn’t yet understand its value.

    Future Outlook

    Confidence around platform engineering is increasing, especially for teams with mature DevOps approaches. As a result, the practice will likely gain more traction in the coming year—71% of respondents reported their firm plans to hire people with platform engineering experience over the next 12 months.

    Over the next year, organizations that adopt the practice will want to increase awareness of platform capabilities and set expectations across the organization. This will also include setting governance policies and satisfying the needs of specific developers, departments and the organization as a whole.

    Above, we’ve scratched the surface of the current state of platform engineering, its benefits and its pain points. For plenty of other insights, you can pick up a copy of The State of DevOps Report 2023: Platform Engineering Edition from Puppet by Perforce here.

  • Best of 2022: Using Event-Driven Architecture With Microservices

    Best of 2022: Using Event-Driven Architecture With Microservices

    As we close out 2022, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    Microservices architecture is on the rise, already forming a key part of several current transformation projects, breaking down traditionally monolithic applications into self-contained, independently deployed services that are identified using domain-driven design.

    Stats from a recent study found that more than 60% of respondents have been using microservices for a year or more. In particular, finance and banking lead the way in using microservices, but many other industries are following suit including retail/e-commerce and telecoms.

    Not only do microservices hold the potential for enhanced scalability, flexibility and durability, but the time taken to deploy and maintain components is also significantly improved. This isn’t the case for all microservice architectures. Major impediments come in the form of ecosystem integration challenges and weak distribution processes.

    Busting the Myths of Distributed Computing 

    The implementation of microservices is more complex than one may first think, exacerbated by the fact that many DevOps teams fall into the trap of making false assumptions about distributed computing.

    The list of distributed computing fallacies was originally addressed in 1994 by L. Peter Deutsch and others at Sun Microsystems, and still holds true today. There are key fallacies that hold special importance to microservices implementation: reliable, homogenous, and secure networks, latency is zero and transport cost is zero.

    The smaller you make each microservice, the larger your service count, and the more the fallacies of distributed computing impact stability and user experience/system performance. This makes it mission-critical to establish an architecture and implementation that minimizes latency while handling the realities of network and service outages. 

    Uncovering the New, Updating the Old

    Microservices require connectivity and data to perform their roles and provide business value, however, data acquisition/communication has been largely ignored and tooling severely lags behind. We can see this with API management or gateway products that only support synchronous, request/reply exchange patterns. This only goes to exacerbate the challenges of distributed computing – they don’t have the ability to integrate with or acquire data from legacy systems.

    At the same time, eventing/messaging tools have also been stuck in the antiquated, non-agile world, being incompatible with many of the guiding principles of microservices such as DevOps and self-service. 

    Solving the Logistical Maze of Integration 

    As services become smaller and their purpose more singular, the potential for reusability increases, but that’s contingent on the ability for services to collaborate. Acquiring data for greenfield systems is easy, but microservices almost always come as a side effect of digital transformation, modernization — or the need to build new capabilities at a more rapid pace. 

    The Disconnect

    Most existing systems live on-premises, while microservices live in private and public clouds so the ability for data to transit the often unstable and unpredictable world of wide area networks (WANs) is tricky and time-consuming. 

    There are mismatches everywhere: updates to legacy systems are slow, but microservices need to be fast and agile. Legacy systems use old communication mediums, but microservices use modern open protocols and APIs. Legacy systems are nearly always on-premise and at best use virtualization, but microservices rely on clouds and IaaS abstraction. 

    The case becomes clear – organizations need an event-driven architecture to link all these legacy systems versus microservices mismatches.

    Creating the Perfect Microservices Harmony 

    The smaller the service, the less value it offers the end-user – value comes from orchestration. Historically, orchestration was handled by a central component like BPEL engines or ESBs, or, these days, API gateways.

    Orchestration is a good description – composers create scores containing sheets of music that will be played by musicians with differing instruments. Each score and its musician are like a microservice. In a complex symphony with a hundred musicians playing a wide range of instruments – like any enterprise with complex applications – far more orchestration is required. 

    Microservices are Well-rehearsed

    So, an example of microservices orchestration is when a microservice performs a series of steps within the code. The input or output of a microservice is a data event that has domain significance. But what is key is that since the microservice is merely producing an event, it does not have knowledge of if or when it will be processed. Other services must register their interest in the event, or set of events, and react accordingly. This all needs microservices orchestration. 

    Time to Shift the Focus

    Event-driven maximizes agility as it liberates data from being ‘at rest’ to being ‘in motion’ – consumable in real-time. Choosing the right eventing or messaging platform is one of the most critical steps to realizing the vast benefits of microservices.

    Microservices per se are not enough. Gartner has identified this need as far back as 2018 when it spotlighted ‘event-driven’ as one of its top strategic technology trends, citing “Digital business is event-driven, so organizations need to invest in event-centric design practices and technologies to exploit digital business moments.”

    The Full Potential of Event-driven Microservices

    Merging event-driven architecture with microservices can house major benefits. Developers can build highly scalable, accessible, robust and versatile systems that can digest and aggregate exceptionally large amounts of events and information in real-time. 

    Not only should modern events platforms be able to be deployed in every cloud and platform as a service, but they should also support DevOps automation. In such an ever-changing world, developers need a seamless self-service experience – only made possible with event-driven microservices.

  • Best of 2022: Making the Move From Developer to Team Lead

    Best of 2022: Making the Move From Developer to Team Lead

    As we close out 2022, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the latest in our series of the Best of 2022.

    After working in their role for some time, software developers may wonder what career opportunities await them beyond coding. Certainly, taking the next step is not obligatory. Some people feel comfortable remaining developers for their entire professional careers.

    However, for those who wish to explore other options, there are a number of different growth paths to choose from—whether it’s working as a solution architect, a Scrum master or a product owner, just to name a few examples.

    Another option to consider is a team lead—considered by many to be a position that is a culmination of long-time efforts, passion and dedication as an IT specialist. But is becoming a team lead an opportunity that an ambitious developer should consider?

    What Makes a Great Team Lead?

    Unlike developers, who usually work on a specific aspect of a project and communicate mainly with other developers, team leads—as the title suggests—are focused on coordinating the activities of and collaborating with the entire team. They also spend far less time dealing with technical issues. A team lead is a leader, a representative and a manager of the team and, as such, their responsibilities involve communicating with both the team members and stakeholders within and outside the company.

    A team lead is the first person a developer would go to when a problem occurs and is the person responsible for suggesting a solution. Bug fixes, tooling and library choices or thoughts on how to approach a specific code-related problem—these are the team leads’ specialties since their technical knowledge and experience are what propelled them into a leadership role in the first place.

    The Go-To Point of Contact

    If you want to improve an application or a test, the team lead is the person to talk to about it. They should know the ins-and-outs of technology solutions and be able to explain to less-seasoned developers why something works and something else doesn’t, or why it may be a good idea to do X instead of Y.

    This doesn’t mean, however, that they are all-knowing. Asking other experienced developers for their insight, opinions or to share their knowledge is something good team leads do without fear of losing their authority. They should always be open to other people’s suggestions and welcome innovative ideas.

    In addition, team leads should think about a project as a whole, including all the people who are working on it. They can direct developers’ efforts and tell their teams what to pay attention to while making improvements, sharing best practices and taking care of code infrastructure and project architecture.

    When to Step Up

    It seems obvious that experience, knowledge, passion and communication skills are all things that ultimately make a good team lead. And these skills aren’t developed overnight—they often start as junior engineering or DevOps professionals and gain technical, social and leadership skills before they can guide an entire team or teams.

    Sometimes, this progression is a natural consequence of an individual’s personality and work style. A team may notice that one member has more extensive knowledge and/or that one particular person is a natural leader who can motivate others and direct their activities. At other times, team leads emerge spontaneously because of their technical knowledge or they volunteer to gain valuable experience.

    For a developer, taking the next step may seem like getting thrown into the deep end. But many current developer skills will be useful in their new role.

    For instance, attention to detail—something most developers are known for—can be invaluable as a team lead. Relentlessly analyzing and improving code, for example, is likely to come in handy in a team lead’s work.

    The Importance of Soft Skills

    Technical know-how, experience, passion, hard work and conscientiousness are the keys to becoming a successful team lead, but it’s also necessary to enjoy working with other people.

    A team leader’s job requires communication, collaboration and people skills more than ever before. Collaborating with team members as well as other stakeholders within and outside the company means that a team lead is a central figure in the IT and business processes. Being tech-savvy is great, but leading a team goes far beyond that since it’s about knowledge and experience and being able to communicate about that knowledge and experience.

    Another key competency is fluency in English. Using English for coding purposes and communicating in English are two different things. As a team lead, you must talk to your clients and team members, many of whom work remotely and may be in different countries. Of course, you can develop your language skills as you go when you’re a team lead; as with most skills, learning by doing can help with confidence and fluency.

    Building language skills can happen simultaneously with building technical and interpersonal skills. Knowledge of various technologies, frameworks and functionalities; experience in working with them; finding ways to change and improve things—all of these give team leads a good reason to practice language skills by talking with colleagues, supervising projects and advising their stakeholders within and outside the company.

    Making the Move to Team Lead

    With their bottomless willingness to work harder, learn more and take on more responsibility, team leads are an incredibly important role in software development. If you feel that constant development—both technical and interpersonal—is the right course of action for you, then consider becoming a team lead.


    Paulina Gatkowska, test automation lead and Robert Musil, Python team lead at STX Next contributed to this article.

  • 5 Best Practices for Implementing Cloud-Native DevOps

    5 Best Practices for Implementing Cloud-Native DevOps

    As the world transitions to cloud-native offerings as an industry norm, DevOps is gaining traction for its critical role in supporting more efficient IT infrastructure. DevOps is designed to boost collaboration and communication by streamlining the automation process to expedite the creation and deployment of applications. Implementing cloud-native DevOps requires organizations to make a massive cultural shift. Without this change, businesses are unlikely to be as competitive or fully utilize the innovations the cloud has to offer. 

    There’s always a risk in adopting new technology. Following are five best practices to accelerate your company’s implementation process and address potential challenges head-on.

    1. Think cloud, adopt the cloud.

    Optimize technology with a variety of tools tailored for the cloud. Understanding how technology works in the open source environment that cloud-native DevOps relies on is as essential to successfully managing the components as it is to selecting the right tool for you. Don’t get locked into using one tool. Instead, select the right tool for each task while remaining as cloud-agnostic as possible.  

    2. Implement a dynamic security strategy.

    Not considering security is one of the most common pitfalls organizations encounter when using emerging technology. Security must be built into cloud-native development workflows and thought processes and diligently monitored. If funds are available, appoint a chief security officer responsible for scrutinizing DevOps security in the cloud and creating playbooks for each alert triggered. Cybersecurity is becoming more complex due to the unpredictable nature of adversaries and the sheer volume of technologies involved in each solution. One way to start is with a bug bounty program that can bring hackers to your side of the court.

    3. Commit to ongoing training.

    Perform a self-audit and/or penetration testing to reveal weaknesses in company technology, procedures and people. Invest in training to effectively address any gaps in cloud skills. Providing development opportunities will also improve employee engagement and retention, which is a challenge for many employers. Make sure your employees give ongoing feedback to the training’s content, its maintenance and evolution.

    While it may be tempting only to emphasize new-hire training, the rapidly changing technology landscape requires training efforts be geared toward ongoing learning for the entire team. This investment will pay dividends, with DevOps and the cloud netting major long-term savings. To make training more efficient, it can be conducted remotely. Integrate and target training based on employees’ job functions,  but don’t forget about non-IT employees—they must also have a basic understanding of cloud and cloud-native.

    4. Once, twice, automate!

    Automation is critical to minimize software development life cycle friction and eliminate dependence on manual labor. Provisioning networks, infrastructure, application deployment and setup can be automated and versioned with infrastructure-as-code (IaC) tools (such as Terraform). There are also endless automation opportunities for containers, cloud computing, security, testing, monitoring and more. For instance, containers allow developers to compartmentalize applications to work on components without worrying about their potential impact in other areas–making it easier to construct consistent, tangle-free deployments. However, it may be challenging to manage multiple versions, requiring infrastructure and testing investment.

    5. Performance-test your tests.

    Automated testing will save developers time and reduce human errors. It’ll also allow developers to take advantage of doubling up on testing, making sure every change is stable and benchmarked. Test during the development stage and lay the groundwork for an effective CI process, especially if you roll out patches regularly. Automated testing is easier in a cloud-based system and isn’t dependent on sufficient hardware to make it run properly, allowing cloud hosts to scale up or down easily.

    Final Takeaways

    Cloud-native DevOps is revolutionizing the way applications are being built and managed, and it starts with how we perceive our challenges. Embracing DevOps culture enables organizations to leverage cloud and cloud-native solutions to create reliable services that are easily scalable. While it may look resource-intensive, cloud-native DevOps is becoming a business necessity.

  • Shift Left Testing in Microservices Environments

    Shift Left Testing in Microservices Environments

    By now, it’s common knowledge that the later a bug is detected in the software development life cycle (SDLC), the longer it takes and the more expensive it is to fix that bug.

    In 2017, the Ponemon Institute found that it cost around $80 on average to fix a defect detected early in the SDLC and around $7,600 to fix the same defect if it was detected in the production stage.

    Relative Cost of Fixing a Defect Based on Development Stage

    Design

    Implementation

    Testing

    Maintenance

    Cost of Fixing Defects

    1x

    6.5x

    15x

    100x

    While these numbers can be debated, it’s clear that early detection of defects can save organizations significant time and money while enabling those organizations to deliver higher quality software more rapidly. For these reasons, many organizations are implementing shift left testing practices, including in microservices environments.

    Shift left is a set of practices intended to find and prevent defects early in the software delivery process instead of during the testing and maintenance stages. The idea behind shift left testing is to improve software quality by making actionable test signals and feedback available as early in the SDLC as possible.

    Here, we’ll highlight how shift left testing, in combination with test automation, can better enable both innovation and fast delivery of high-quality software. We’ll also address some of the challenges involved in implementing shift left testing in microservices environments, as well as introduce a valuable strategy for successful microservices implementation.

    Shift Left Testing Benefits

    The waterfall model of development involved the explicit passing of responsibilities between highly specialized design, development, QA, and release teams. It also involved lengthy feedback loops.

    Scrum and agile methodologies made the entire SDLC more flexible and nimble by introducing sprints and allowing more frequent iterative development and delivery. Further, DevOps and DevSecOps focus on removing the silos between development, operations, and security through tooling and automation. As a result, the time to market and quality have improved dramatically. Adding shift left testing into the mix better positions teams to handle the broad range of responsibilities from the design stage through the maintenance stage as effectively as possible.

    Shift left testing focuses on prevention rather than detection. Shift left benefits include the following:

    • Increase efficiency by eliminating bugs earlier in the SDLC.
    • Reduce human errors and associated costs.
    • Increase delivery speed and reduce the time between releases.
    • Improve the quality of software.
    • Gain a competitive advantage.

    It may be helpful to think of shift left testing like this: Test early and in parallel with development, instead of the traditional waterfall model of development followed by QA testing. 

    Shift Left Testing Challenges in a Microservices Environment

    By structuring an application as a collection of loosely coupled, independently deployable services, a microservices architecture enables frequent delivery of large, complex applications.

    While shift left testing practices bring many benefits, shift left testing also comes with three challenges that organizations must address when implementing it across a decentralized, microservices-based stack:

    1. Building and managing each microservice separately reduces development time. However, implementing shift left testing at the microservices level means shift left practices must be applied to each individual microservice.
    2. Microservices developers can choose the programming language and service dependencies best suited to their development needs. However, as a result, shift left testing practices need to be implemented and managed across different programming languages and dependencies.
    3. To ensure that microservices function as expected, teams usually run integration tests, end-to-end tests, and performance tests that assess functionality in the context of the larger stack. Shift left testing is especially challenging in the microservices context because of the complexity of testing against the stack.

    Microservices, each with their own SDLC pipeline

    Pipeline Scaling Considerations

    Several classes of automated shift left tests can be integrated into the DevOps pipeline, including unit tests, integration tests, performance tests, and security tests. Different microservices will have different testing needs.

    The key is to apply shift left testing practices to each microservice while maintaining the flexibility and decentralization of those microservices. The ideal workflow system and DevOps pipeline implementation must simplify pipeline creation while enabling development teams to manage the pipeline and requiring minimal input from DevOps and platform teams. In other words, shift left practices must be tailored to the specific needs of the microservice in question to make the testing process scalable and easy for development teams to manage on a continuous basis.

    As just one example, GitHub Actions and similar CI/CD pipeline tools enable development teams to use GitOps-based practices to configure, manage, and administer the entire pipeline for their microservice while leaving infrastructure testing and testing that relies on or affects other parts of the system to DevOps teams.

    Test Environment Considerations

    Test environments are environments where unit tests, integration tests, performance tests and security tests can be run on an isolated collection of microservices and configurations. The nature and quality of these test environments directly correlate with the effectiveness of shift left testing.

    A cloned production or staging environment can be used for testing when the number of microservices is low and overall complexity is relatively low. However, once an organization reaches a threshold number of microservices or a threshold level of complexity, runaway infrastructure costs and operational burdens make such environments untenable.

    In response, some organizations rely upon smoke testing (minimal viable stability testing) and mock-based integration testing early in the SDLC. While such tests can prove useful in the near term, deferring essential testing to a later stage in the SDLC often introduces test signal and validity problems in the staging environment and can lead to delivery delays and serious issues with software quality.

    So how can shift left testing be used successfully in a microservices environment? The solution involves the use of scalable, ephemeral test environments.

    Ephemeral Shift Left Testing Environments

    The availability of high-quality test environments is a common bottleneck for organizations looking to scale shift left testing practices. Using a few high-fidelity staging environments for signal testing can slow the release process. More to the point, it tends to slow down the entire SDLC and increase stack instability over time.

    Ephemeral test environments offer a fundamentally different approach. These sandbox environments use isolated environments that enable users to test independently without impacting other users. In particular, they enable isolation while sharing resources (e.g., external cloud dependencies and hosted databases) using application-level multitenancy instead of the resource duplication approach in traditional environments.

    The idea is that microservice test environments can be used to test changes against a much larger stack of services. Because multiple tests can be conducted at the same time, test coverage and resource efficiency vastly increase.

    High-quality sandbox environments can spin up in seconds and scale to tens of thousands of microservices. In fact, each pull request, branch, or commit can have its own test environment without significantly increasing infrastructure and operational costs. Any change, no matter how small, can be tested in isolation to ensure performance and stability against the entire stack.

    Shift Left Testing Solutions for Microservices

    A microservices-based organization looking to scale shift left testing across the organization must first address several challenges. Considerations include the need to apply testing practices to separate microservices, the need to accommodate different programming languages and service dependencies and the need to test against the stack.

    The right tooling and automation are needed before shift left testing practices are scaled across the organization. In addition, a cultural shift may be required in order to help developers understand the ways in which shift left testing can expedite and improve the core functionality that they are developing.

    To apply valuable shift left testing practices in a microservices environment, organizations should consider using ephemeral sandbox environments. Using the right set of tools in a sandbox environment can enable development teams to test microservices throughout the SDLC. In doing so, organizations can ensure those services follow security best practices and perform exactly as intended, both independently and against the stack–all while reducing costs, speeding time to market and improving software quality and user satisfaction.

  • 5 Tips to Consider When Introducing Infrastructure Automation

    5 Tips to Consider When Introducing Infrastructure Automation

    Although we discuss infrastructure automation often, the fact is that a lot of manual processes still exist throughout DevOps. Whether it’s rewriting scripts to spin up new servers, updating cloud-native policies or configuring computing environments, manual toil is an all-too-common reality. To make matters worse, engineers commonly create lengthy scripts that work in isolation but are too bulky to be reusable by other developers working with slightly different constraints.

    More infrastructure automation is the key to increasing deployment agility, but setting aside time to automate your daily workflow can easily take a back burner, especially when developers are already behind on their long list of to-dos. So, for the time-pressed, where should they start? What exactly should be automated? And how can we effectively reuse and share automation to help other areas of the business?

    I recently connected with John Laffey, technical product marketing director, Puppet. Laffey has thirty years’ worth of tech support and engineering operations knowledge under his belt and was “doing DevOps before it was called DevOps.” He shared with me five helpful tips to think about when introducing more infrastructure automation into your workflows. Below, we’ll expand on each of those points and consider other best practices around automation as the industry moves toward platform engineering.

    1. Automate Things That Interrupt Your Day

    First, Laffey recommended starting with the things that are easy to fix. These are the tedious actions that interrupt your day. Perhaps a disk runs out of space and troubleshooting takes 15 or 20 minutes. Minor interruptions like this might seem like part of the job, but they definitely add up over time.

    2. Automate Things You Can Reuse

    Secondly, try not to automate things for an ultra-specific purpose, said Laffey. Instead, focus on building automation that is more generic and can have more value to others. This follows the DevOps mantra of teamwork and collaboration. One-off automation that is too verbose or specific to a certain condition won’t apply well across groups.

    3. Automate Stuff That Doesn’t Change Much

    Next, automate the infrastructure that isn’t prone to change. These are fairly reliable things that you’re sure will be part of your systems for the foreseeable future. Because if a component does end up having to be updated often, supporting the automation could be more hassle than it’s worth. “Don’t trade fixing problems for fixing automation,” said Laffey.

    4. Start Small

    Here’s some sage advice applicable to most new projects with a big scope—start small. Laffey recommended starting by automating the easy stuff that has proven immediate value. These tasks should be modular and do one specific thing, like rebooting a machine or draining a log file. “Automate things in chunks that you can put into a toolchain.”

    Too often, teams try to automate everything all at once. But by taking big bites, you can quickly get off track and potentially add to technical debt. If you keep things modular, you’ll find it easier to change and reuse this automation.

    5. Share and Be Open

    This last tidbit is more of a cultural change—don’t code so that you’re the only one who knows about it! Sometimes, engineers don’t necessarily want the world to know about the automation they create. They might worry about automating their job away. Yet, Laffey said, these worries are misplaced. There will always be more work to do, and automating toil away can free up time to help you focus on more exciting projects.

    So, Laffey encouraged DevOps and SREs to be open about the automation they create. Ensure they’re well documented and stored in a centralized repository to allow other departments to reuse your code. This can avoid the duplication of having five different versions of the same script written in different languages within the same organization!

    Next Step: Move From Scripts to Industry-Standard Tooling

    It can be challenging to argue for investing time and energy into infrastructure automation, especially when engineers are hard-pressed to keep the lights on. Yet, small gains in automation can make huge waves in the long term.

    Eventually, said Laffey, you’ll want to move away from using scripts within your toolchain, which he describes as a low-end DevOps maturity. But as you update your process, you’ll want to replace these scripts with more modern industry-standard tools like Terraform or Puppet’s DSL. He recommended slowly revamping scripts in a toolchain as you update them one by one. “Start where you are and avoid a monolithic approach.”

    In essence, platform engineering, Laffey explained, is wrapping DevOps principles at a higher level to make them more efficient. And we will likely see an increased emphasis on infrastructure automation as platform engineering gains more mindshare. “It’s going to become a necessity.”

  • Migration to AWS Cloud: Do’s and Don’ts

    Migration to AWS Cloud: Do’s and Don’ts

    What if one day, you were told to migrate to AWS/GCP/Azure/any other cloud provider? It would likely introduce some stress, to understate the case, right?

    I’ve seen quite a few migrations in my career and, in this article, I would like to share some thoughts to help DevOps engineers, architects, managers or anyone else who might be involved in such a migration. I’m focusing on migration to AWS mostly because I have a specialization in this subject and because it’s fairly common, but I think these experiences are applicable to any other cloud provider. For this case, I’ll cover migration from on-premises to a cloud because these types of migrations are much more difficult (in my opinion) than cloud-to-cloud migrations. I’ll also share my own perspective on do’s and don’ts of migrations. But please filter them through critical thinking and remember that everybody has their own background and experience. What’s good for one might be bad for another.

    What is a Cloud Migration?

    Let’s start by defining what migration is. How do you know if it’s a migration? I like the following definition: 

    Cloud migration is the process of moving a data center’s capabilities to IaaS providers.

    “Capabilities” is the keyword. A business does not migrate servers/virtual machines/hardware/data or anything else. A business migrates its capabilities.

    There are seven migration strategies (or seven Rs): Relocate, rehost (lift and shift), replatform, repurchase, rearchitect/refactor, retain and retire.

    For example, I’ve seen situations when a company was creating a new product that would replace an existing one hosted on-premises. The product would be designed to be well-architected and cloud-native from the very beginning and would be hosted in a cloud. Is that migration or greenfield project type? From the definition above, we can say that this is a migration. This is a rearchitecture strategy for migration.

    What if we buy a new SaaS subscription instead of some software we host on-premises? This is also a migration: A repurchase strategy.

    Let’s dig deeper into these seven migration strategies:

    Relocate. We host some workloads on-premises and move them as-is into the cloud. This is possible in limited cases; when we migrate cloud-agnostic workloads or when a platform might be cloud-native. Examples: Relocate a Kubernetes cluster from on-premises to the cloud or relocate VMware machines with VMware Cloud.

    Rehost. This is also known as lift and shift. We convert virtual machines/servers on-premises to virtual machines in the cloud. Examples: Migrate on-premises virtual machines to EC2 instances with CloudEndure, or create EC2 instances, install the software and apply the same configuration as on-premises.

    Replatform. We migrate a workload to a cloud-native platform without rearchitecting systems. Examples: Migrating PostgreSQL to AWS RDS or Docker containers to ECS.

    Repurchase. We replace some custom/legacy system with SaaS. Example: Replace custom on-premises email system with SendGrid or replace CRM with Salesforce.

    Rearchitect. We change your application’s architecture to be cloud-native and use managed cloud services. That also includes rebuilding from scratch. Examples: Change the code of your apps to upload files to S3 instead of local storage or containerize an app and migrate to ECS.

    Retire. We decide that a piece of the workload will be eliminated if not needed. Example: Log servers are no longer needed as migrated applications use CloudWatch.

    Retain. We leave it on-premises. Usually, it’s temporary. Example: A huge Oracle database with a lot of functions and customization is not moved to AWS because of the massive costs. It’s decided to keep it on-premises until it’s replaced with another solution in the modernization phase.

    I like the following picture that shows the high-level stages of each migration type. It may be a helpful cheat sheet to approach migrations.

    Picture Source: https://aws.amazon.com/blogs/enterprise-strategy/new-possibilities-seven-strategies-to-accelerate-your-application-migration-to-aws/

    Migration Don’ts

    I don’t recommend the following when performing a migration:

    Migrating too quickly. A business might think, “We have to migrate to the cloud quickly. Let’s do lift and shift right now!” It’s better not to start right away. Are you sure that rehosting strategy will really be faster than some other strategy? In one instance, we tried to migrate on-premises VMs, but it didn’t work because the OS on the machines was quite old. Some OS-level dependencies didn’t work on AWS’ hardware. This required a lot of configuration work, custom code and testing and the initial estimates were completely wrong. It eventually turned out to be a replatform. 

    Assess what you have, analyze it, analyze the business case and think. During this engineering work, we should stop and think before we act.

    Migration without clear goals. Why are you migrating? If you can’t answer the question, you probably need to find an answer or not migrate at all. I’ve seen situations in which businesses wanted to migrate to the cloud to reduce costs. They performed a lift and shift migration without doing a total cost analysis. Only after the fact did they realize that it cost even more than when it was in a colocation center. This migration may be considered unsuccessful.

    To avoid this, define clear goals and define the target state that achieves these goals. If costs are the driver, make the proper calculations and plan accordingly. Cost optimization will be the main thing to focus on when you plan your target state. If availability is the reason, measure it, define SLOs and focus on availability in a target state.

    Black box migration. Don’t treat workloads you migrate as a black box. It is almost critically important to know what exactly you migrate, what technical requirements it has, what dependencies it has and how it works. That information will allow you to choose the correct strategy for workload migration. If this is not done, there’s an excellent chance that you will choose the wrong path and the result will be lost. Capturing information about your systems is the most important task when planning.

    Don’t forget well-architected. Migration is stressful. That’s why people tend to try to get through it quickly on as short of a timeline as possible, focusing on the only thing: Making everything work in the cloud ASAP. They set aside well-architected framework pillars. This is a mistake; it may lead to a different set of negative consequences of different severity. Don’t ignore well-architected framework pillars, or you won’t know about potential issues or risks until they have already materialized.

    Use one strategy for everything. It’s always easy to make a single overarching plan for all workloads. For example, deciding to do lift and shift and then rehost everything in the cloud. But what if some things could be retired? What if some components could be replatformed with little effort? You will save significant resources by analyzing all of the components of the workload and deciding on a unique strategy for every component. Collect data and plan properly.

    Cloud Migration Do’s

    Do perform a cloud readiness assessment. If a person migrates to a new location, they should consider:

    • That the climate will be different
    • The language spoken in that location may be different
    • The cost of living will be different and perhaps more expensive

     When a company migrates to the cloud, they should ask questions including: 

    • Have we considered all the risks?
    • What must be changed for the workload to actually work?
    • Do we have the human resources required for all of the work?
    • Do I have the budget to accomplish everything? 

    The Cloud Readiness Assessment is a long-form interview about varying processes in the company that identifies whether a company is ready to live in the cloud. It’s based on the Cloud Adoption Framework (CAF) and allows us to understand what should be changed so that a migration can happen successfully. AWS has a Cloud Readiness Assessment tool which covers all of that. I urge you to use this process for all workloads being considered for migration.

    Create a business case. Describe the business case. What are the drivers for migration? What problems are resolved with the migration? What goals should be achieved by migration? Be specific: If you intend to reduce costs, then calculate the sum. If you intend to improve availability, then define a clear SLO. That will be the number-one driving force for your migration. All other artifacts should be based on it.

    Automate discovery and data collection. Use automated data collection and analysis to create a portfolio for migration. Don’t base your analysis on manually created spreadsheets or verbal communication. Let machines collect data about inventory and usage statistics. The more precise data you collect, the better. AWS Migration Evaluator service will help you to do this in most cases.

    Portfolio management. Create a portfolio of workloads to migrate, analyze them, choose a migration strategy for each, choose some for the pilot/first wave migration, estimate timelines and prepare a plan. Execute the migration after that. Repeat until the migration portfolio is empty. Share the data. Have subject matter experts review it.

    Found CCoE. This is part of the CAF mentioned above. Before executing the migration, create a team of people responsible for creating a strategy and defining a good standard of living in the cloud: A Cloud Center of Excellence. It should include people with a variety of subject matter expertise including management, operations, platform, people, finance and every other department consuming or providing services in the cloud. These people will prepare a landing zone for the migration, hard policies and softer policies for the landing zone; preparing the company for life in the cloud. Don’t fear revisiting any policy.  Make changes as new factors come to light.

    Well-architected framework reviews (WAFR). Do well-architected framework reviews (WAFRs) during migration waves and after them. That will allow you to avoid foolish mistakes and to get a high-level overview of what you have done from a technical perspective.

    Modernization plan. Prepare a modernization plan along with the migration plan. Migration isn’t the end of the story. To be effective in the cloud, you will have to do well-architected reviews and modernization. The first modernization is likely to be huge; that’s why it’s important to plan and to schedule budgets and resources in advance. Failure to do this will lead to increased operational costs, not decreased costs.

    Summary

    If we want to reduce the risk associated with a migration and to make it less stressful, we do not rush. You may believe that rushing will save you time and other problems will be managed later, but experience tells us that this is wrong. The most likely outcome is that you will end up resolving all the issues encountered in the most stressful situation possible: When you cut over to your new cloud-based production environment. Instead of resolving issues in a well-considered way and in advance, the choices will be made under duress. Proper engineering principles must be applied here. Picking up a bridge and moving it from one river and plunking it down on another never works. The key to migration success is to measure, analyze, design and plan. Migrating a workload to the cloud is hard, but I hope these do’s and don’ts will make it easier.

  • Looking to the Future of Developer Experience

    Looking to the Future of Developer Experience

    Developer experience is where it’s at! Developer experience (DX) means taking a user experience approach to the developer journey. DX (also abbreviated as DevEx) is commonly applied to areas like documentation, sandbox environments, SDK ergonomics and more. By improving a developer’s experience with a tool, software providers can increase stickiness and retain more users.

    Nowadays, having a sleek developer experience is a competitive advantage—especially since developers often have the discretion to purchase new tools. And amid an ongoing skills gap, maintaining quality DX for internal software ecosystems is also important to retaining happy engineers and avoiding burnout. All these factors mean developer experience is becoming a C-suite objective. So, where is DX heading? How will the developer workflow change in the near future?

    I recently met with Ash Arnwine, director of developer relations, Nylas, to ponder what’s on the near horizon regarding developer experience. According to Arnwine, DX is still a very new concept. Yet, it has become an integral aspect of positioning modern software, especially for public-facing SaaS, and will undoubtedly drive purchasing decisions in the year ahead. Below, we’ll share some predictions of how DX will evolve within four general categories: onboarding, building, deploying, and maintaining.

    State of Developer Experience

    First off, what is the state of developer experience within the tech world at large? Well, Arnwine is quick to note that the concept is still relatively nascent. “Developer experience was something that we didn’t talk about that long ago.” Yet, the idea is growing in mindshare as developers exercise greater control over the software life cycle.

    In fact, the 2022 StackOverflow survey found that 66% of professional developers have at least some influence over their organization’s purchases of new technologies, up from 56% in 2020. Furthermore, the way developers learn is changing. The new generation of programmers is primarily self-taught, relying on coding bootcamps or other online resources. An impressive 70% of developers learned how to code from online resources.

    What we end up with, said Arnwine, is a growing developer population with a much wider range of skills than in the past. And since no one is there to hand-hold them, the technology itself must be usable and intuitive—the onus is on the software provider to craft frictionless self-service workflows. And with developers’ growing influence in purchasing, DX becomes a competitive advantage and an executive talking point, said Arnwine.

    Onboarding

    Developer experience is a broad topic, but it’s most often associated with the time it takes to get started with a tool. This is often referred to as ‘Time to First Hello World.’ But although it’s an important step, the metric might doesn’t always carry much significance. Time to value, on the other hand, said Arnwine, takes this a step further and is a better gauge of success.

    For example, onboarding to a web API would likely involve the signup process, creating an account, getting API keys and perusing the documentation—all before a first call is sent. According to Arnwine, there are techniques to apply at this stage to help developers move faster throughout the onboarding process to reach value sooner.

    Here are some ways in which developer onboarding is becoming more streamlined:

    • Automatically generating keys upon signup.
    • Providing a terminal within the developer dashboard to test sample code.
    • Handing developers the code they need to be customized to their specific stack.
    • Using specifications as a single source of truth for documentation.

    Building

    What can SaaS companies do to help improve DX while building? Well, one way is to enable auto-complete within IDEs. When developers onboard with a new tool, they are often handed an SDK and reference documentation. But, it doesn’t add value to send programmers back and forth between a browser to look up method names. Instead, Arnwine believes developer tools should be “baked into the IDE in a way a modern developer would expect.”

    One example of auto-complete on steroids is Fig, a next-gen command line that supports auto-complete for over 500+ popular CLI tools. Taking this a step further is GitHub Copilot, an AI assistant that can take natural language prompts and generate pretty complex code suggestions. The industry appears to be heading toward more AI-fueled low-code software development—a trend that could significantly improve a developer’s experience during the construction process.

    However, one potential downside of increased software development automation is ensuring the AI suggests correct code snippets. Thus, Arnwine said he anticipated possible troubleshooting and debugging problems. Furthermore, SaaS providers will have to vet systems that generate glue code to ensure it’s accurate and up-to-date with the latest versions. His advice to SaaS providers is to meet developers where they are, but do their best to maintain a quality experience that they can control.

    Deploying

    For a long period of software development history, IT administrators were the only ones able to deploy software. The old-school method required getting a hosting provider, installing Ubuntu and plenty of other legwork. Nowadays, however, software deployment has become far more drag-and-drop. DevOps has become more democratized in recent years, and we will likely see greater usability emerge in the years ahead.

    One way to increase developer experience for deployments is through more custom automated routines via GitHub Actions or similar event-based systems. And plug-and-play pre-built actions could reduce the barrier to using such systems. Such connectivity is necessary to enable a docs-as-code approach, said Arnwine.

    Maintaining

    Another area that could use a serious developer experience upgrade is dependency management. Software dependencies go through a life cycle. They are versioned, updated, retired and sometimes replaced outright. As such, keeping track of an evolving surface area of third-party dependencies can be a headache. To make matters worse, software supply chain threats have complicated the integrity of many open source dependencies consumed by most applications.

    Although it’s imperative for organizations to maintain parity with dependencies, a single person is rarely tasked with continuously checking dependencies and monitoring for security threats. Therefore, more automation for dependency management and security monitoring is necessary. For example, Arnwine uses Snyk, which sends a weekly report on the state of his dependencies.

    Final Thoughts

    Assuming you have the killer functionality to back it up, an investment into developer experience can help a developer tool stick out in a crowded marketplace. But of course, improving the DX means many different things depending on your context. It could be a subtle user interface tweak, adding relevant code samples or making your documentation more intuitive.

    On the other hand, a negative developer experience can turn away potential users and cause frustration throughout the application support life cycle. This can instill a poor reputation among programmers.

    To review, in the near future, we’ll likely witness the developer experience evolve in the following areas:

    • Enhanced methods that streamline the signup process.
    • Auto-complete and AI-driven code suggestions that understand a developer’s stack and meet them where they work.
    • More intuitive, automated deployment models.
    • Increased automation around the maintenance of dependencies.

    While developer experience is a growing concern, it shouldn’t be the only measurement of a SaaS — the ongoing value it brings developer consumers and end users is just as (if not more) important to consider.

  • AWS re:Invent — Top 4 Things We Learned This Week

    AWS re:Invent — Top 4 Things We Learned This Week

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week, four things that caught my eye from Amazon Web Services’ re:Invent 2022 conference: SnapStart turbocharges Lambda, Graviton3E ARM HPC SoC, AWS continues hiring, and AWS’s origin story. (more…)