devopsy.kinsta.cloud

Category: DevOps Practice

Making DevOps Smoother
Being a DevOps developer isn’t just about technical expertise necessary to build a product. It is also about learning the operations that facilitate the entire process. That means working with a fast-paced, high-performance, cross-functional team using Agile methodologies. But because of these sometimes-conflicting demands, it can sometimes get rocky.

Here, we’ll offer six tips for DevOps developers to ensure a smoother development process.

Set Measurable Goals

While working toward a specific goal, precision is paramount. Defining the problem accurately leads to a faster solution. For instance, “Configure deployment to a new region” is too vague compared to “Configure the deployment of all components of service X for North America.” The more specific aim of the latter makes it easier for the parties involved to identity the problem, make a plan and resolve the issue quickly.

Most DevOps developers use the acronym SMART while setting goals in a sprint.
1. Specific: Define exactly what needs to be done.
2. Measurable: Highlight the parameters and values necessary to indicate completion.
3. Achievable: Define how the task will be accomplished and decide who will be responsible for doing it.
4. Realistic: How realistic is the goal based on the above details?
5. Timely: Set the date and time when the task has to be completed.
While setting goals for a particular sprint, keep the above five points in mind to make sure that the tasks are precisely defined.

Document Everything

Proper documentation during the entire course of development is valuable to everyone. In many cases, DevOps developers join the project after it has started, and having them review documentation created by other developers is the easiest way for them to get up to speed. Not only that, but documentation also helps with debugging, troubleshooting and guiding the clients.

Documentation of the process is often seen as boring and a waste of time. However, thanks to online documentation tools, logging information has become a lot simpler. Using a document template with good formatting makes things easier down the line.

Communicate and Collaborate

Essentially, the efficiency of a DevOps development team boils down to how well they communicate and collaborate within the team and with the external stakeholders. Gaps in communication will slow down and possibly derail the entire development process. This is especially important as more and more teams are going remote and popping into your coworker’s physical workspace to ask a quick question is no longer possible.

In a remote working environment, communication can be slower. Sending a text, waiting for a reply, scheduling a call, etc., adds a lot of steps to the process as opposed to simply pointing at a screen and asking a question. Although there are a lot of visual collaboration tools and more integrations for video conferencing tools, it is still not the same.

Therefore, it is important to set firm communication and collaboration rules and make sure expectations are set from the beginning so that team members and stakeholders know what to expect. One of the easiest ways to do so is by having specific tools for specific channels of communication. Typically, there are three types of communication tools that are necessary:
- Team communications: Messaging apps like Slack and Microsoft Teams.
- Stakeholder communication: Basecamp, Workzone or regular email.
- Video conferencing: Can be used for internal and external communication. For example, Zoom, Google Meet, etc.
Automate Repetitive Tasks

If there are certain recurring tasks that happen at a local level, you can use Bash or PowerShell. Bash is older and used by a lot of IT and DevOps teams. PowerShell, developed by Microsoft in 2006, is newer and has replaced the command shell in Windows environments.

Although both Bash and PowerShell are cross-platform, generally Bash is preferred by Linux and macOS users whereas PowerShell is preferred by Windows users. Ultimately, it comes down to personal preference. No matter whether you choose Bash, PowerShell or a newer automation tool, automation will save a lot of crucial development time and remove toil from a developer’s workload.

Choose the Right Tech Stack

Every team is different in terms of their requirements and the tools they prefer. “DevOps tools” is a term that covers everything from server orchestration tools, cloud platforms, code management and many other categories. In simple terms, DevOps tools are instruments that make it easier to write, test, host and deploy code. There are many DevOps tools that a development team can choose from. Here are three of them which are worth considering.
1. Git: Git is a version control and distributed source code management. It makes it easier for developers to collaborate, experiment and merge code on the main branch.
2. Cloud IDEs: Fully hosted cloud integrated development environments (IDEs) protect your progress in case your local system fails. GitHub Codespace, Cloud9 and Glitch are the most popular ones.
3. Containers: Containers simplify the build, test, stage and deploy pipelines. They can either be locally hosted or fully hosted on the cloud. Containers help to identify and fix issues before they are deployed.
While choosing your tech stack it is important to keep the user experience and user interface, usability, scalability, available integrations and pricing in mind. Your tools should fit into your process, not the other way around.

Value DevOps Culture

The work culture is essential for the cumulative success of the entire DevOps team. The way the team interacts with each other defines the way they work. When a team is provided with a sufficient amount of freedom to make its own decisions, it will come up with innovative ways to make the overall process more efficient. Overcoming the divide between “dev” and “ops” in its own way will empower the team to do its best work.

With that being said, remember that a culture is something that develops over time organically, not forced upon people.

Wrapping Up

Developing software while collaborating with IT operations doesn’t have to be hectic or frustrating. By following these simple tips you can improve your entire team’s productivity. We hope that this article has provided you with some insights that will help you smooth out any development hurdles.

Happy developing!
May 17, 2022
Why Over-Permissive CI/CD Pipelines are an Unnecessary Evil
Balancing the speed of innovation with security is an age-old problem in software development. This is no different in modern DevOps. Development teams want a frictionless path to deploy new code but often struggle with access permissions that are linked to services. The CI/CD pipeline makes it easier for developers to quickly introduce new features and updates, in an agile way. As application changes move through the CI/CD pipeline, automation can allow changes to happen very quickly, without creating downtime or delays on the customer side.

Permissions of CI/CD Servers

When it comes to the permissions of CI/CD servers, DevOps often don’t want to deny access. After all, the wide access permissions are part of allowing the speed of innovation, and development teams are regularly—and often understandably—cautious about putting least-privilege or other access rules in place.

But there are some security risks to consider. Any developer being granted admin permissions to deploy new code is open to vulnerabilities, weaknesses and misconfigurations. Cloud sprawl has created a lack of visibility into whether or not the infrastructure is properly set up and connected with logging tools. How does the development team know if things are getting properly patched?

The balance is extremely sensitive. DevOps need the speed of change and the ability to innovate freely, but security teams can’t handle the growing risk. Something has to change that can allow security to shift left and get involved earlier in the CI/CD pipelines rather than getting in the way.

Reducing the Risk Without Impacting Innovation

Here are some suggestions for accelerating the speed of innovation while maintaining a healthy security posture:

Map threats and secure connections. Conduct a threat modeling exercise to map threats to the pipeline. Remember that every connection to the CI/CD pipeline is a potential point of compromise. Regularly patch and scan all services that connect to the pipeline and block the devices if they fail to meet security policy requirements.

Learn to love logs. When a security issue arises in production, developers, IT engineers and security engineers need to react quickly and efficiently to solve the problem. Logging is critical for enabling fast and coordinated responses. Waiting on manual sharing of sensitive data can slow down reaction times. But when every stakeholder can get the data they need from logs, access to information is no longer the weakest link in the security incident remediation process.

Tighten access control and enforce strong admin permissions. Be specific about who has access to which data. Clear and enforced separation of duties should ensure no one person or account can access more than necessary in the continuous software delivery pipeline. Improperly credentialed users or bad actors can’t establish control over the entire pipeline and mess with the code or processes. Moreover, they won’t easily be able to penetrate deeper into other corporate systems and data.

Leverage flags to manage risks. Developers often use feature flags (or feature toggles) to add new features to applications while mitigating the risk that those features will introduce performance or stability issues. Although most developers likely don’t think of feature flags as a security tool, they can be used for that purpose, too. New features are prime vectors for security vulnerabilities, especially if developers haven’t thoroughly tested them yet.
Many security teams do not realize that just because their network is private, they’re still sharing pipelines with third parties, including anyone from maintenance to the team deploying your mobile app. By embracing these tools, developers can shift left, shoring up defenses early in the process, without adding a blocker to the pace of innovation coming from DevOps CI/CD pipelines.
May 16, 2022
Why Data Lineage Matters and Why it’s so Challenging

Change introduces risk. It’s one of those foundational principles of software development that most of us learned very early in our careers. Nevertheless, it always seems to keep cropping up in spite of those repeated life lessons. Our inability to foresee the impact of changes, even small ones, often leads to negative outcomes.

As the complexity of interconnected IT systems has increased, this problem has grown worse. It’s harder and harder to predict how even a small modification might impact systems upstream or downstream of the change. Data lineage accomplishes that, but it’s extraordinarily difficult to do data lineage well.

Why Data Lineage Matters

In the old world, data resided in silos with relatively few connection points between them. Today, data flows multi-directionally among multiple systems. Many of those systems share data with external organizations, which means that information subsequently flows to internal systems there, as well. Consequently, it’s easier than ever to overlook the impact of a single change on multiple other systems.

Imagine, for example, that your company provides a mobile app that enables customers to quickly and easily request a new service call. In the past, your customers could specify which product variant they owned. Your design team has determined that since you already have that information in your CRM database, you no longer need to collect it from users of the mobile app.

You’ve been tasked with simplifying the app, so you remove that input field and replace it with a query that pulls the customers’ product information from your CRM. Theoretically, the resulting work tickets should include all of the information that your field service personnel need to get the job done.

Unfortunately, there’s a problem; some of the customers’ product records in your CRM system contain null values. Your mobile app is querying that information, but it’s delivering invalid results to your ticketing system. Your field service personnel no longer know how to prepare for their next service call.

That might sound like a relatively simple problem of data quality—one that might have been foreseen if someone had been more thorough about vetting the CRM data. Even that might not have solved the problem, though, if null values in the CRM database were not introduced until later. If customer data from a newly acquired company is merged into your CRM system, for example, you might not have product information tied to those new records.

Sometimes these issues are more difficult to predict, and they may not be apparent right away. Imagine what happens when your marketing team decides to redefine its customer segmentation criteria. A new data field is added to designate the customer category; then it’s populated with values for all customers and the old field is deprecated.

Unbeknownst to the marketing department, though, the company’s top executives are still looking at the old data. They’re using dashboards that rely on data from the deprecated field. New customers are being added every day, but no one is entering any values in the deprecated field. As a result, the executive team is looking at distorted data. Suddenly, the C-suite is tracking very different KPIs than the rest of the organization. What’s worse, no one even notices that there’s an issue until weeks or months after the fact.

Next, imagine a similar scenario, but instead of populating an executive dashboard, the source system is feeding data to the AI algorithms that drive customer recommendations on your e-commerce site. When AI is trained on flawed data—or if the data changes materially in some way—it can dramatically impact the effectiveness of that technology. In this case, it distorts your customer recommendation engine.

Tackling the Challenge With Data Lineage

Data lineage tools provide a systematic means of understanding the impact of changes by presenting a complete roadmap of potential implications, both upstream and downstream.

This amounts to risk reduction and cost control. If we can predict how changes might impact other systems, we can avoid creating problems like the ones we described earlier. That leads to less wasted effort, faster time-to-market and lower costs. We’re all familiar with the principle that says it’s faster and cheaper to solve a problem when you catch it early in the development process; data lineage provides a critical missing link, potentially ferreting out problems before they even reach QA.

There are benefits to data governance and regulatory compliance as well. When the European Union passed the General Data Protection Regulation (GDPR) back in 2016, it set off a chain of new requirements for systems that house personally identifiable information (PII). If a customer asks you to delete their data, you’re legally obligated to comply with that request. But what happens upstream and downstream of that change? Will it break anything? Or will some of the customer’s PII remain elsewhere in your systems, rendering you non-compliant with the law?

Data lineage addresses these questions by automating the process of detecting upstream and downstream effects. It provides clear auditability that can help determine where changes originated and how the data arrived at its current state.

Column-level data lineage is especially difficult because it requires that SQL queries be parsed to discern exactly what changed and how. This means tapping into database logs, but because there are so many database vendors—each with their own unique SQL dialect—parsing that information for various databases can be particularly challenging. As each of those SQL dialects evolves over time, data lineage tools must constantly be refined to accommodate those changes.

An example column-level lineage view that shows all the tables and columns that are involved in computing the sales table used for reporting in an imaginary e-commerce business selling craft beers.

As interconnected systems extend their reach even further and the flow of information accelerates, column-level data lineage will become more important. Today, it’s an extraordinarily useful tool that saves time, effort and money. Soon, data lineage will be indispensable.

May 16, 2022
Observe, Inc. Dives Deeper Into Observability

Fresh from picking up an additional $70 million in funding, Observe, Inc. announced this week it has added multiple tools, dubbed Data Universe Maps, to its observability platform to visualize all datasets and surface underlying event data.

The company also added pre-built applications for Kubernetes, Amazon Web Services (AWS) and Jenkins environments with editions for Google Cloud Platform (GCP) and Microsoft Azure planned for later this year. Observe Apps are intended to make it easy for IT teams to ingest data that is surfaced via pre-built dashboards that will generate alerts when issues arise.

Finally, Observe, Inc. has added dashboards that provide transparency into its usage-based pricing model. IT teams can now more easily determine how many credits are consumed by both users and specific datasets to help contain costs.

Observe, Inc. CEO Jeremy Burton said those tools are crucial because in the absence of tools to control data costs, most organizations will soon discover that observability is unsustainable.

Of course, it’s not clear yet how many organizations are moving beyond monitoring metrics to embrace observability and launch queries to discover the root cause of performance issues. Many organizations don’t have the level of expertise required to structure such queries. However, as more organizations struggle to investigate and remediate issues impacting customer experience, interest in observability platforms is rising, noted Burton.

Burton said thus far, as a startup, Observe, Inc. has expanded its customer base by a factor of three to now include 50 organizations, which has resulted in a five-fold increase in the number of monthly active users of the platform. The platform is now ingesting more than 40TB of data each day, with more than 25 million queries each day launched across nearly one trillion rows of more than 10 petabytes of data. Capital One is included among the investors in Observe, Inc.

At the core of the Observe platform is a graph engine that makes it simpler to uncover the relationships between machine data that is ingested into the platform.

Going forward, Burton said Observe intends to add support for distributed tracing using the open source agent software created by the OpenTelemetry project that operates under the auspices of the Cloud Native Computing Foundation (CNCF).

Change within enterprise IT organizations happens slowly. Many will be using legacy monitoring tools alongside observability platforms for years to come. However, there is an opportunity to rationalize monitoring tools as the capabilities of observability platforms continue to expand.

In the meantime, IT environments will continue to become more complex as more modern applications based on microservices and constructed using containers are deployed. These applications, over time, are generally more resilient than monolithic applications; given all the dependencies between microservices, determining the root cause of a performance issue can be a major challenge. As a result, it may be just a matter of time before this new class of applications ultimately forces the observability issue in the enterprise.

May 13, 2022
Top 3 Requirements for Next-Gen ML Tools

As the machine learning market matures, new tools are evolving that better match data science and machine learning teams’ needs. Vendors, both open source and private, have been quick to introduce new products that better meet requirements making it easier and faster to develop models and enable collaboration.

These new offerings range from cloud-based platforms that make it easy to build and deploy models to specialized software that can speed up the training process. As an example, the existing field of ML experiment tracking is a multi-billion dollar market with a crowded amount of vendors: Comet ML, Weights & Biases, Neptune AI, Databricks’ MLflow and more. These tools will need to evolve to move beyond meeting only the basic requirements of ML teams like experiment tracking, visualizations and model training to address collaboration and technical requirements so these teams can get models to market faster.

The types of teams that are driving modern requirements for data science are the bet-the-farm types, where it’s all about finding new technology and solutions to drive efficiencies and get models to market faster for their organizations to maintain a competitive edge. Going forward, we’re seeing the following as requirements for ML teams as they evaluate tools to add to their MLOps stack.

An Open, Modular Approach

Data scientists and ML engineers use multiple tools to do their jobs, from Jupyter notebooks for model coding to DVC for experiment and data versioning, to automation tools like AirFlow to model visualization tools. Solutions will need to be open to integration with other tools instead of a walled garden that many ML tools, like those offered by cloud vendors (e.g. Amazon SageMaker, etc.) are now.

Familiar Developer Experience

ML developers and engineers are familiar with standard DevOps tools and workflows like GitOps (think GitHub or GitLab) and live in CLI-oriented solutions. ML tools should integrate tightly with these existing tools instead of complicating the developer experience with a separate ecosystem. And as a part of the developer experience, workflow is also important. Data scientists can run hundreds of experiments using their preferred CLI-based solutions but, often, all those experiments are streamed and posted into existing ML tools. A more optimal approach would be to offer data scientists the experience of running experiments locally and letting them commit only the most important experiments for further discussion and collaboration so the team isn’t overwhelmed.

Strong Connection to Source Code

This approach makes a stronger foundation for something like tracking ML experiments. As ML teams race to get models to market and continually improve them, being in lockstep with DevOps teams to deploy those ML models into production is critical. An example would be a developer tracking five experiments with changing hyperparameters. For these experiments, they may have a link to a Git repo for a particular version of source code. But what if the source code changes or data set changes are not committed or reported in their existing ML solution? Then, the connection is lost and existing metrics would be based on a previous version of source code or data that’s no longer accurate. That lost connection slows down time-to-market and makes it harder to optimize models.

As seen with the above examples, ML teams juggle myriad tools and workflows as they develop models for various applications and services. The requirements discussed will help these teams to collaborate better and build models faster in an increasingly heterogeneous MLOps landscape.

May 13, 2022
How Waterfall Methodologies Stifle Enterprise Agility
In this age of sweeping digital transformation and accelerated change, organizations are recognizing that depending solely upon the drawn-out process of waterfall methodology is no longer sustainable if they are to remain competitive and relevant.

Enterprise infrastructure must incrementally embrace evolution from monolithic architectures in response to the escalating pace of technological innovation, ever-changing consumer needs and demands and the failure of a long-running single project focus.

EA Must Continuously Adapt

Traditional enterprise architecture (EA) frameworks struggle to manage the impact of disruptive technologies and systems that are agile and flexible. An architecture design approach that embraces continual evaluation and iteration is better equipped to navigate change and capitalize on emerging trends and innovations. Adaptability, in this context, may be defined as the fluid ability to modify, restructure and recreate the architecture in relation to changes within and external to the enterprise. It involves the early identification and rapid response to new product concepts, important trends, events and consumer demands; developing a “from the outside in” perspective.

This adaptability, or ‘evolvability,’ quotient enables the flexibility and agility to deliver services and products reliably in this new development norm. Enterprise leaders are quickly acknowledging the need to inject a more open and agile format into their enterprise’s processes, at least in some measure, to succeed in today’s unpredictable environment.

The Waterfall Model Constrains Enterprise Agility

Traditional organizational architecture can impose limitations on an enterprise’s ability to successfully reach its digital transformation goals. The up-front model, with a focus on one long-range project, can slow productivity and choke creativity. While planning is needed as agility scales, the detailed technology life cycles with large timeline projections are no longer effective or profitable in meeting the business mandates that drive enterprises forward. Enterprise leaders are increasingly abandoning the five-year architectural plan for one that is designed to evolve with the ever-changing software development environment.

Enterprise architects must now develop and promote adaptive methods that support agility in order to appropriate the value of new technologies like AI, machine learning, big data, IoT and intuitive tools that enable advanced analytics and enterprise-wide collaboration. A less intentional architecture, decomposed into smaller units, can be managed by autonomous cross-functional teams that are accountable to peers and managers with shared strategic objectives, bringing all fields into a coherent whole.

A continuous architecture design with no end state, guided by iterative changes, will fluidly address the changing needs of the enterprise. A shift from a project-oriented to a product-oriented management mindset leads to faster learning cycles, shorter time-to-market, higher quality and increased revenues.

The Attributes of a Flexible and Adaptive EA

An open architecture designed to evolve in an ongoing manner and enable agility will include many beneficial characteristics and principles. Flexible and adaptive EA:

Provides product-centricity that connects software development to business goals;

Explores multiple solutions simultaneously with shorter analysis, testing, experimentation cycles and progression to eliminate weak solutions;

Empowers autonomous, cross-functional teams with clear accountability roles and alignment to a shared mission statement and effective information sharing;

Enables flexible and efficient resource allocation driven by changing demands or activity levels;

Strengthens an evolving roadmap that shows vision and targets, open to the whole enterprise, to allow input and questions for collaboration, cohesion and transparency;

Facilitates incremental changes to software architecture in a continuous manner;

Provides a continuous integration and delivery model with automatic testing for ‘good enough’ MVPs to keep the business moving forward.
An adaptive EA allows for timely decision-making as agility scales, ensuring that the right products are delivered at the right time in alignment with business initiatives. When design accountability is distributed across the enterprise, consensus on complex decisions involving diverse stakeholders is more easily attained.

The business rewards of transitioning to an adaptive architecture should be an easy sell by examining and tracking the monetary, time and customer success gains. The evolution toward adaptive enterprise architecture can reap numerable benefits. In addition to fostering an enterprise culture of continuous improvement, it accelerates and improves development cycles, journeys and outcomes for stakeholders and customers.

Adaptive EA addresses the inherent uncertainties and complexities associated with dynamic business changes. It can deliver the rapid incremental changes, capabilities and benefits that a waterfall model inhibits. The agile and open methodology, empowered to continually evaluate and modify, provides visibility into project requirements and feasibility to reduce risks typically experienced with a traditional design and waterfall methodology.

Making the Transition to an Agile EA

Transforming enterprise architecture to a more adaptive and agile model will require planning and investment. A necessary first step is to assess what the enterprise goals and objectives are in making the transition. To what degree is the organization ready and what needs to be in place in order to move forward?

Investing in an enterprise-wide training program of agile technologies, principles and methodologies will develop a cohesive and cooperative mindset that overcomes resistance to change and optimizes transition success.

Of utmost importance is the development of a culture of communication, from the C-suite down across all stakeholders. A shared vision must be communicated and encouraged within a collaborative framework to cultivate enterprise-wide buy-in and acceptance of changes from waterfall methodologies to current roles and practices. In the migration to agile, there can be many obstacles, but by adopting an incremental approach with frequent reassessments an enterprise can ease the transition.
May 12, 2022
Red Hat Adds Managed Ansible Service on Azure

Red Hat today announced general availability of a managed instance of the Red Hat Ansible Automation Platform on the Microsoft Azure cloud.

Announced at the Red Hat Summit 2022 event, this offering makes it possible to deploy the control plane for Ansible on the Microsoft Azure cloud as an alternative to existing support for Amazon Web Services (AWS) or an on-premises IT environment.

In addition, in version 2.2 of the Ansible Automation platform, Red Hat is previewing support for content signing technology that automatically validates content to improve overall security.

Thomas Anderson, vice president of Ansible for Red Hat, said this managed service enables IT teams to take advantage of integrated Microsoft billing in addition to the latest features and content provided for the platform and other capabilities that have been added to optimize Microsoft Azure environments.

The open source Ansible platform has been gaining traction as an automation framework that enables IT teams to declaratively automate tasks. It requires less programming expertise than rival automation frameworks, which Anderson said is the primary reason so many third-party vendors have adopted Ansible to automate a range of IT tasks.

The challenge many organizations encounter today is managing multiple islands of automation across multiple clouds and on-premises IT environments. Ansible presents IT teams with the ability to begin unifying all those islands under a single framework, said Anderson.

It’s not clear to what degree DevOps teams that have embraced rival automation frameworks might be willing to replace them with Ansible. However, as more IT teams embrace automation to make up for shortages of IT personnel, it’s apparent that Ansible is being used across a wide range of IT disciplines spanning everything from DevOps to network management.

Regardless of the approach, the need to automate as many tasks as possible is becoming critical as IT environments grow increasingly more complex. Many of the scripts that IT teams created on their own simply don’t scale. More challenging still, they are not often documented, so every time the IT professional that created that script leaves the organization it becomes difficult to maintain that scripting code.

In the longer term, of course, automation frameworks will be increasingly infused with machine learning algorithms capable of learning the unique nuances of individual IT environments. It’s not likely those algorithms will replace the need for IT professionals any time soon. However, rather than resisting automation, more IT professionals have concluded that the tasks required are simply too challenging to manually perform. Many will conclude they simply prefer to work for organizations that have a consistent approach to IT automation that makes it feasible for them to manage a wide range of IT tasks.

One way or another, IT automation is becoming a standard capability that now goes well beyond the scope of a few isolated DevOps workflows. The issue is finding a way for IT professionals with varying skill levels to collaborate across a common framework that is both easily accessible and highly extensible.

May 10, 2022
Jamstack Will Revolutionize How Websites Are Built
Today, every organization wants a website that loads quickly and looks sharp. The reasons go much deeper than cosmetic motivations, though—even a slight delay in loading can cause users to exit a page prematurely and site performance plays an important role in search engine rankings. Meanwhile, threats like ransomware and phishing are everywhere and are constantly evolving, requiring organizations to look for more ways to improve site security and protect user data from bad actors. Then, of course, there’s the need to quickly and easily scale websites to meet sudden spikes in demand. The need to adapt quickly has become especially apparent over the last two years as we have come to rely on the internet in new and profound ways compared to pre-pandemic times. Fortunately, a relatively new but popular approach to web development—Jamstack—is taking websites to a nearly infinite level of scalability and performance, serving up only what you need, when you need it. To understand the value that Jamstack brings today, it’s helpful to take a look at how we got here.

Throughout the history of the internet, web and database servers were the primary infrastructure for site hosting. When websites got more complicated, another layer, called the application tier, would take on more complex aspects of the site (like an API, for example). As traffic increased, web servers were scaled out to add capacity with the help of load balancers. Scaling in this manner wasn’t always easy or cost-effective, given how unpredictable consumer demand and spikes in traffic often are. Later, content delivery networks (CDNs) came into the picture, taking a load off web servers by pushing static content to the edge. But, for most traffic, the origin servers would still have to be ready to serve site visitors. In this approach, website development and management were often costly and difficult to scale, and sites faced constant security risks.

Enter Jamstack. With Jamstack, all site content sits on a CDN, ready to be served. Web pages are pre-rendered instead of being rendered in real-time (as was the case in pre-Jamstack days). As such, content is closer to the end user, pages load faster and sites can easily scale up or down to handle unexpected shifts in traffic.

Jamstack represents an exciting and fundamental shift in how we have traditionally approached web development—one that is inclusive of the needs for performance, security, availability and scalability and at a much lower cost to the developer. It’s no surprise that Jamstack has taken the web development world by storm over the last couple of years; in 2020, the number of sites developed with Jamstack jumped 85% from the year prior. Here, I’ll share some basic tips for getting the most out of Jamstack.

Adding Dynamic Features to Static Sites

Jamstack starts with the idea of a static site—pre-building as much of a site as is practical—and incorporates dynamic elements to achieve fast websites with fewer attack vectors. The term, which was coined in 2015 as JAMstack, derives from:

J = JavaScript, the website’s programming language

A = APIs, which enable dynamic content on an otherwise static page

M = Markup, which is the HTML and CSS code that provides formatting instructions to the browser
By combining these three elements, Jamstack allows developers to quickly create and maintain websites that can be served efficiently to users. A static site renders well for mobile, web and video while relying on APIs for dynamic features. High page speed is achievable by pre-building as much of a site as possible and then using a CDN to deliver the page to users around the world. When a user visits a website, they see a pre-rendered page delivered by the CDN with no dedicated servers needed.

Notably, the page’s static nature does not require the content to be static. Leveraging third-party APIs can enable dynamic content such as search, payment processing and real-time data. This modular approach allows for flexibility and avoids vendor lock-in—swapping out different APIs as technology changes and new tools become available is easy. The ability to integrate with serverless functions also lends itself to Jamstack sites becoming more dynamic.

Without the need for web application servers and database servers, Jamstack pages scale well as visitors increase, improving both the experience of the visitor and the organization’s bottom line.

In essence, a Jamstack site has no backend that needs to be managed by a developer. The static nature of the page means there is no link between the website and a database, which reduces the security attack surface. Since all site elements are delivered from the CDN or an API, potential attackers aren’t able to hack web application servers or database servers.

When an organization creates a website with this approach, there are no servers to maintain and no staging environments to create, so there is less of a need for complex DevOps resources. Jamstack’s simplicity means there are fewer moving parts than in a traditional site and less room for error. Because of this decoupling between the frontend and backend, Jamstack sites are faster to launch and more reliable.

Another advantage of the static setup is that reverting to an earlier version of a page is simple. This is possible through the concept of atomic deploys, where the entire site is updated at once, creating a new version. With Jamstack, developers have more freedom to experiment and avoid the tiring trial and error of traditional sites—if something does not render as intended, switching the entire site back to a previous version is painless.

Assembling a Jamstack Toolkit

Jamstack’s tools, processes, and best practices are evolving rapidly, and an important part of diving in is remaining open to experimentation and tracking emerging developments. Getting started is simple since Jamstack integrates with existing workflows and many organizations have established relationships with vendors already in place for key technologies, speeding up the decision-making process. These technologies include:

CDN: To deliver pages from locations close to users—a requirement for achieving fast performance.

Static site generator (SSG): To generate the site using raw data and templates—automating the process of coding pages and ensuring they are pre-built and ready for users.

Content management system (CMS): To act as a content repository. A CMS for a Jamstack website is described as “headless,” which means that content is stored outside of the codebase and delivered via APIs for seamless display across different devices.
Expanding the Value of Jamstack

We are still in the early years of Jamstack’s maturation, and not all deployments are equally effective. For example, some Jamstack platforms are hosted out of centralized data centers coupled with a CDN, which slows performance relative to delivering sites directly from the edge. Some Jamstack hosts also charge more to add developer seats, which discourages collaboration among large teams and can result in unpredictable pricing.

Regardless, Jamstack is one of the best technological solutions to come to developers in recent years and one which will surely influence the future of web development. Not only does it make websites faster, more secure, and easier to scale, but it builds on many of the tools developers already know and love—empowering them to achieve their maximum productivity.
May 10, 2022
Does GraphQL Introduce New Security Risks?

The GraphQL query language is an excellent tool for increasing the ease of data sharing. The premise is that you request the fields you need in a single bundled request, avoiding multiple network calls. Due to its usability, GraphQL is a growing area of development. Yet, as with any new technology, it comes with caveats.

REST APIs function as a request-and-response paradigm in which the receiver accesses one resource at a time. On the other hand, GraphQL exposes many fields at once, requiring additional rate-limiting and time-outs to avoid nefarious queries. Additionally, the ability to traverse the entire schema could be abused by attackers to expose intricate data linkages. There are also some usability features with GraphQL that could be abused if misconfigured.

GraphQL is a friendly alternative to REST. But how do we handle the security implications of GraphQL? I recently chatted with Mike Benjamin, vice president of security research, Fastly, to dive deeper into the security repercussions of adopting GraphQL. Below, we’ll consider the unique insecurities associated with GraphQL and suggest specific steps to limit the impact of these risks.

Four GraphQL Security Vulnerabilities

1. Introspection

GraphQL provides built-in documentation that tells developers what the fields are, providing a schema of everything you can query. You’ve essentially given the entire dataset schema to a developer, meaning they can perform introspection to capture attack surface information. Hackers will most commonly look for injection or mutation capabilities that they can exploit, explained Benjamin. Or, they might seek to leverage the interlinkages between data to map a way through to classified information.

To mitigate this problem, Benjamin cautioned against over-exposing all documentation publicly. Instead, he suggested using a restrictive security model for external viewers. “Provide appropriate user documentation in a different method,” he says. This may end up looking more like the standard REST API view of documentation.

2. Field Suggestions

Another usability benefit in GraphQL could also be a potential security concern—field suggestions. Field suggestions occur when you don’t enter the correct field name or enter a partial input and the system returns a hint. You might type in 0 or user, and the system makes a suggestion to fill out the rest of the field.

This user interaction improves the developer experience, but, unfortunately, it could be abused by a hacker, said Benjamin. Thus, he recommended disabling hints after initial development. Or, if you decide to keep it in, he recommended placing a limit on its use. For example, if you try this more than five times in a second, the system could flag it as a bot.

Right now, this attack vector is pretty limited, but Benjamin anticipated that, over time, hackers could automate this introspection with mass scanners to divulge more information.

3. Denial-of-Service-Oriented Attacks

Without the proper guardrails, all web APIs are prone to denial-of-service (DoS) attacks. Such an attack may pummel a service with a million queries per second to shut it off completely. In the case of GraphQL, these queries can dive very deep in a nested data relationship. A social network, for example, might expose friends of users, friends of users’ friends and so on.

Or, hackers might utilize GraphQL’s bulk query ability to perform reconnaissance, brute-forcing the enumeration of data to query hundreds of user IDs to see which are valid. If you’re allowing unlimited data retrieval bundled within a single request, you could be inviting hundreds of bundled queries that could expose large swaths of the underlying database in one fell swoop.

To counter these actions, Benjamin recommended limiting the testing depth. This could involve placing limits on how long a query can run, which would cause a time-out for requests that require suspiciously high degrees of processing. Another option is to limit how deep the bundling can get, said Benjamin. For example, perhaps the system restricts going three levels deep into a graph database, as that could expose millions of interconnected endpoints.

4. Lack of Object-Level Authorization

Another vulnerability that GraphQL endpoints may be prone to is a lack of object-level authorization, which hackers could leverage to get at a datapoint through its edges, said Benjamin. Since the idea is that you can access any data from one all-encompassing endpoint, authorization rules are often too abstracted away from the actual data. Queries may be able to access nested data or query relationships between data, too, which may unintentionally divulge information.

Benjamin recommended placing authorization closer to your objects or data to rectify this issue. Use authorization-level controls that align permissions structure and roles with users. “Everyone needs a common authorization model,” said Benjamin.

Future Evolution: GraphQL Security Matures

As I’ve written before, the greatest strength of GraphQL is also its greatest weakness. Using GraphQL with default settings is a security trade-off in favor of usability. But according to Benjamin, we’re seeing more GraphQL servers disabling some introspection features, which he sees as a very positive move and a sign of maturity of the base configuration.

GraphQL might suffer from similar threats that all APIs are vulnerable to, such as XSS, injection, DDoS attacks and broken authorization. However, Benjamin stressed that the added complexity of GraphQL makes it a unique case. “The most important thing for folks to consider is that GraphQL is more complicated than a REST endpoint—much more complicated,” said Benjamin. “GraphQL is unique in that it has linkages of data which contributes to complexity. It takes a little more time to think about how that’s confirmed, how it’s presented and how it would be attacked.”

In other words, with great power comes great responsibility.

Although exploits in public GraphQL APIs are still rare, we should stay vigilant. Benjamin encouraged developers to continue reading about, understanding and thinking about these attack vectors when designing their interfaces, as these risks will become more prominent as GraphQL usage grows and hackers become more familiar with these specific frailties.

May 10, 2022
Checkmarx Report Highlights Need for AppSec Collaboration

A research report published by Checkmarx finds the same basic malicious software developed using multiple programming languages as cyberattackers industrialize their malware development processes.

Checkmarx, a provider of code scanning tools, shared examples of malicious packages written in multiple programming languages. These example packages share the same indicators of compromise that have gone undetected for years. A “junkeldat” PyPI package for Python applications, for example, that has been compromised also shows up in a Ruby version of the package.

Tzachi Zornstain, head of supply chain security for Checkmarx, said much like any other development team that uses multiple programming languages, it appears cybercriminals are also sharing techniques. These examples highlight the need for development teams to share security intelligence even if they build applications using different programming languages, he added. There is a tendency for development teams using one programming language to ignore application security issues that appear to only impact applications written in another programming language, noted Zornstain.

The fact that the packages remained undetected for such a long period of time is due—at least in part—to the lack of information sharing in the ecosystem, said Zornstain.

Cyberattackers are, of course, trying to take advantage of the implicit trust many development teams have in open source projects. However, now that more cybercriminals are making a concerted effort to compromise downstream software components, it’s imperative that every component be vetted, said Zornstain. In fact, organizations should not use any code provided by strangers, he advises.

The core issue is that many open source projects are maintained by a small number of programmers contributing their time and effort to building components that others are free to use. Many of them argue that the onus for making sure software is secure is on the organizations that decide to deploy that software. Nor is it their responsibility to keep track of cybercrimimals distributing malicious versions of their software.

Unfortunately, many IT vendors and large enterprise IT organizations that rely on that code are, unfortunately, not contributing anything meaningful back to the project, either in terms of financing or helping open source maintainers find and remediate vulnerabilities. Many of those same organizations, however, are assessing whether the open source software they employ is, from a security perspective, actually sustainable in the absence of those contributions. As a result, it may be only a matter of time before these long-simmering open source software security concerns boil over into a larger crisis.

One way or another, DevSecOps best practices will need to be applied to application development at a deeper level. Organizations can’t assume that the software components they are relying on have been scanned for vulnerabilities by someone else. It’s simply too easy for a development team to make a mistake. The need for higher levels of security may lengthen the development process, but the alternative—malware infestations wreaking havoc—is far less desirable.

May 9, 2022