devopsy.kinsta.cloud

Tag: data

Using Microsoft Azure as a Backup Repository

Every year, the amount of data that needs to be backed up increases. There are two ways to store backups – using on-premises facilities or trusting cloud-based solutions. Having an on-premises storage infrastructure can be costly for a company. This is because of the high prices for storage equipment and the expenses to maintain this equipment, e.g., hiring a storage engineer. Depending on a company’s rules, Microsoft Azure could be an alternative to traditional ways of keeping data. If the amount of data is relatively small (less than 4 TB), there is no problem with making use of this solution; you can either use Azure Blob as a CIFS or NFS storage, or you can create your own virtual machine by adding a disk to store your data and then configuring it as a file server. However, this blog post focuses on cases where space limits are more than 4 TB.

Microsoft Azure allows you to create VMs and add disks with a maximum capacity of 4 TB for each disk. It is suitable for application servers but very often doesn’t work if you require a big file or backup server.

However, we can exceed this limit by using Windows software RAID array. The idea is to use RAID5 as the software RAID which contains arrays starting from three disks.

Why RAID5? Because it has a good read speed limit. It is also economical to use X disks, where X starts at three and you only lose one disk space for the total RAID result space. Furthermore, this should give you redundancy if one of your disks were to fail. However, if we build a RAID array in Azure, we already have the virtual disks as blocks of our array and they are protected by Microsoft Azure Cloud. Thus, the chance of failure in our RAID array (which is based on disks that are already in the RAID array) falls significantly.

Here’s how to create an Azure-based backup and replication repository server.

1. Log in to your Microsoft Azure account or register a new account at https://azure.microsoft.com/en-us/

2. Create a new Windows Virtual Machine:

In this example, we choose Windows 2008 R2 SP1, since we don’t require any advanced services, just software RAID5.

3. Type in your VM name, username and the password according to MS Azure rules of complexity.

4. Choose your VM size. Please note that the VM type depends on the number of attached disks. Thus, if you require more than 8 disks (over 24 GB in RAID 5), the only option is to choose DS2_V2 Standard:

5. The settings for available storage are some of the main options for our future repository:

At this point, you have created a new VM; now it is time to add disks. On the Disks page, you can add as many disks as you need for your future disk array. In our case, we will build software RAID5, which will require seven disks in total.

We need to have our SMB share available from the internet, so we have to configure the Azure Firewall settings.

In the Networking section, click the Add inbound port rule button and create a new rule allowing you to make port 445 (SMB) accessible to everyone (for testing). Alternatively, you can only make the port accessible to the backup software instance’s external IP as in the Source field (for production).

The next step is to connect to the VM via RDP using the IP and credentials you created during the VM creation process.

Next, you need to create a RAID5 volume. In Storage Manager, you`ll see all 4 TB disks you attached. Right click Disk Management > All Tasks > New Raid 5 Volume:

Select all available 4 TB disks to add them to your array and move them to the right column:

Follow the standard wizard; choose a letter for the disk and perform a quick format by NTFS.

Next, create an SMB share on a new logical disk and add write permissions to the future repository.

At this stage, you can use your entire 24 TB volume as a remote repository in your backup system.

If you use the NAKIVO backup to Azure solution, just add a new repository using the server’s IP path to your share as well as the credentials you set for the SMB share:

Then, create a new job using the new repository as the backup target and run it.

Conclusion

The amount of data organizations need to back up is growing exponentially. Therefore, it is important to store that data in the most economical way. Using a data backup system like NAKIVO and RAID5, you can make sure your VMs are reliably protected and that there is enough space to store them in Azure.

May 16, 2023
Nobl9 Adds Tools to Make SLO Queries More Efficient

At its SLOconf event today, Nobl9 added a Query Tester tool that validates whether queries made against the data its service level objectives (SLO) management platform collects will deliver meaningful results. Nobl9’s SLO management platform collects data in real-time from data sources such as New Relic, Datadog and Dynatrace to help customers meet SLOs.

In addition, a Query Checker tool will now automatically put any new SLO into a testing state to automatically validate that the query against that SLO will run correctly.

The company has also improved the precision of its calculations to provide DevOps teams with more accurate, reliable and actionable insights into the performance of services and added a Metrics Health Notifier toolset that provides insights into anomalous data being collected from various sources as well as what data might be missing. Nobl9 has also added a Custom Query Delay tool that gives DevOps teams more control over when data is pulled along with access to data source logs.

Kit Merker, chief growth officer for Nobl9, said the overall goal is to improve the overall resiliency of its platform for managing SLOs.

Finally, Nobl9 also announced that its platform is now available for the Google Cloud Platform (GCP) in addition to existing support for Amazon Web Services (AWS). As part of that GCP support, Nobl9 made available an educational site that provides example of how DevOps team can take advantage of the large language models (LLMs) that Google is making to improve SLOs or generate an SLO that is automatically converted into a set of YAML files.

A survey of 314 IT professionals conducted by Dimensional Insight on behalf of Nobl9 found 69% have implemented SLOs, with more than three-quarters of those respondents crediting SLOs for preventing business disruptions. A total of 80% also noted their organization has an increased focus on system reliability, with just over a quarter (27%) saying SLOs saved their organizations a half million dollars or more. A full 95% said SLOs are also driving better business decisions.

However, 97% also reported it is difficult to manage SLOs because of a lack of reporting, lack of application and environment support and interoperability issues.

Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing SLOs across a Git-based workflow. Nobl9’s platform based on that specification has now been integrated with 24 data sources, said Merker.

SLOs, of course, are not a new idea. They have been employed as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications.

Ultimately, each IT team needs to provide some sort of objective benchmark that assesses their overall effectiveness at delivering application services. SLOs-as-code are intended to make it simpler to gather the metrics that confirm whether service levels are being achieved. Those SLOs are then tracked and integrated across any set of application services a DevOps team cares to track.

May 15, 2023
Four Trends Defining the Observability Market in 2023

Observability is a market that has grown exponentially over the past few years. It started as a buzzword in late 2015 and early 2016, and since then we’ve seen dozens of insurgent companies rush in, eager to stake their claim in a new (albeit loosely defined) market. Incumbent companies also rebranded their wares, hoping to capture some of the momentum that comes from the new term.

Venture capital saw the market opportunity, too. By my conservative estimate, observability companies raised nearly $2 billion over roughly the last two years alone (I’m including some cybersecurity companies, which I’ll explain later).

In the seven or so years since its inception, the observability market has fragmented and specialized into categories such as application, data, network and edge observability. On the artificial intelligence and machine learning side, there is model observability and AIOps. Connecting it all are observability pipelines.

While more businesses than ever know about observability and have built it into their tech strategy, category awareness still has a long way to go. And though the dollars were flying fast and furious a couple of years ago, the economic cycle has turned against risk, and growth funding rounds are harder to come by. It’s time to step back and examine the key trends defining today’s observability market.

Staffing Shortages: A Blessing and Curse to Observability

Many companies in the observability space, regardless of their subcategory, represent net-new work for IT and security teams. A new application performance monitoring (APM) platform may offer a marginal improvement around APM’s golden signals, but that marginal improvement is a substantial lift for the end users tasked with implementing and deploying the new platform. While software may get sold, overworked ITOps and SecOps teams may decide the effort isn’t worth the perceived ROI, leaving the software on the shelf.

On the other hand, observability products that reduce toil for operators will continue to outpace market growth rates. IT departments face acute shortages in skilled staff, leaving companies with little choice but to seek out software that automates high-effort, low-value tasks or otherwise offers a force multiplier to existing teams. Subcategories like AIOps and observability pipelines can address those needs.

Consolidation is Needed to Simplify the Market

Another looming hurdle for the nascent observability market is feature similarity between companies in a subcategory. Two subcategories where this is apparent are application and data observability. In the application observability space, several companies were founded in the wake of the availability of extended Berkeley Packet Filter (eBPF) technology. eBPF allows you to extend the capabilities of the Linux kernel at runtime, and a common use case is to trace applications running in user space. While this is a great technology with several uses, there are simply too many companies with the same idea, the same go-to-market strategy and same limited feature set.

Data observability is another subcategory riddled with clones. Data observability started as a reaction to operational brittleness in the modern data stack, and there are only a few companies with marked differentiation here. I agree with Matt Turck’s assessment that data observability products will drift into the well-defined (but arguably less sexy) data quality category.

CFOs are Looking to Reduce Tool Sprawl and Cut Costs

The challenges with raising growth rounds have been well documented. VC funding represents the supply side of the finance puzzle and IT budgets are the demand side. The demand side is getting squeezed as CFOs clamp down on budget growth. Deals are taking longer to close and often involve more parties eager to reject a budget request from another team because it means more money for their team.

A common refrain from CFOs today is that any new spending must be more than offset by corresponding savings somewhere else. This puts the products representing net-new work with little marginal utility at risk. Observability vendors today need a strong consolidation (e.g. can reduce tool sprawl, serve multiple teams, etc.) and cost savings story to survive in a difficult budget climate.

Cybersecurity remains a budget bright spot. According to a recent survey from IDC, 27% of respondents stated cybersecurity budgets were immune from budget cuts. With changing SEC rules highlighting the need for board-level cybersecurity expertise and reporting of breaches, coupled with the new National Cybersecurity Strategy, the priority around preserving these budgets is clear.

Cybersecurity is Pure Greenfield for Observability

A few short years ago, SecOps teams, threat hunters and incident response teams didn’t have enough data to create an accurate picture of their environment. Endpoint data was sparse and incomplete, cloud-based application deployments were still uncommon, the network remained between the office and data center, and so on.

Today’s SecOps teams, threat hunters and incident response teams have too much data to understand their environment. There are too many alerts and too little context, making finding the signal through the noise a difficult, if not impossible, proposition. From a compliance perspective, much of this data must be retained for longer periods, driving up storage costs.

This is where the principles of observability benefit cybersecurity teams. Much like site reliability engineers need to trace an error well after it occurred, cybersecurity professionals need to trace a breach or find threats over huge volumes of data. They need robust exploration tools, rational data storage and routing options, and, most of all, the ability to do this while maintaining budgets. This is where observability and cybersecurity converge.

Seven years young, observability remains a nascent market despite its rapid growth. Though the market will face growing pains, the opportunities and upside are bright.

May 15, 2023
Why We Need to Rethink Data Pipelines

Data is the fuel that powers modern business. But as demand for data surges, so does the pressure on data leaders and practitioners to deliver it. Businesses need resilient data pipelines that deliver critical insight for real-time decision-making to users on demand. However, against the backdrop of today’s chaotic modern data ecosystems, this is much easier said than done.

DevOps teams are struggling with a complex combination of legacy and diverse technology systems across multiple environments. The resulting siloed systems and an ever-shifting data supply chain cause significant data integration friction. This friction keeps businesses from creating seamless and resilient pipelines to drive digital transformation and other business outcomes. Businesses need a better way to take the pressure off data leaders and DataOps practitioners and enable line-of-business teams to do more of the “last mile” data collection and analysis themselves. To achieve this, they need a unified, end-to-end platform designed to build resilient pipelines.

Data Demand Sparks Friction

The acceleration of digital transformation has created a demand for data that far exceeds supply. This has become especially true as competitive advantage gets harder to carve out and worrying macroeconomic headwinds gather. In today’s climate, all areas of the business are demanding data, but it’s hard to meet every request all the time.

Research that polled data leaders and practitioners shows nearly half (48%) of admin and operations and customer service departments request data at least weekly, followed closely by accounting and finance (44%), IT and digital (43%) and sales and marketing teams (40%). And the strain is showing. Over half (59%) of data leaders said changing priorities have created significant data supply chain challenges.

The problem is one not just of capacity but complexity. Building pipelines from source to destination requires rules to integrate, transform and process data. But when data is siloed across cloud, legacy and mainframe systems and stored in inconsistent formats, creating bespoke data pipelines to fulfill departmental requests is a huge challenge.

Legacy systems also bring another challenge. Over half of data leaders say it’s so difficult to unlock data from legacy systems like mainframes that they simply don’t bother. That will ultimately undermine the value of the cloud analytics tools that many business leaders see as a single source of truth for decision-making–especially as legacy systems often hold decades of vital business insights.

This patchwork approach creates extra toil for DevOps, DataOps and technical teams and means that over two-thirds (68%) of data leaders said they are being prevented from delivering data at the speed the business needs.

The Scourge of Broken Pipelines

For many organizations, building pipelines is also a labor-intensive job requiring a high degree of manual effort to produce hand-coded, one-off solutions. The resulting pipelines are brittle and vulnerable to being disrupted whenever there is a shift in the environment–such as adding new data sources. In fact, two-fifths (39%) of data leaders admit their pipelines crack at the first sign of trouble, and 87% have experienced a break at least once a year. More than one in 10 say it happens at least once a day.

The most significant cause of breakages is due to bugs and errors being introduced during a change (44%) while infrastructure changes such as moving to a new cloud (33%) and credentials altering or expiring (31%) are also creating disruption. These broken pipelines immediately impact the corporate bottom line. They also force technical teams into a vicious cycle of firefighting. Data engineers spend, on average, almost a third (31%) of their time troubleshooting and recoding broken pipelines–time that could be better spent on value-adding tasks.

But the negative impacts aren’t only felt operationally. The breakage of any pipeline can also lead to bad decision-making. For example, a supply chain director working with old data may over- or under-order goods or a financial trader making stock picks on out-of-date intel may lose money for a client.

Unleash the Power of Data

For organizations looking to overcome these challenges, the first goal is to reduce the workload on DevOps, DataOps and technical data teams by empowering business units and end users to do more. Research shows that 70% of technical data leaders are responsible for the last mile of data collection and analysis while 86% of them would prefer lines of business teams, such as marketing or finance, to be empowered to do this independently. However, this can only become a reality if organizations invest in the right data integration platform. It must be able to build, run, monitor and manage smart data pipelines at scale from a single console–across all cloud and on-premises environments. And it must be able to deliver resilient pipelines built to withstand continuous change.

Only by eliminating data integration friction and enabling self-service analytics for lines of business can organizations accelerate digital transformation. Doing so will unleash the power of data across the enterprise and reduce the burden of overworked data leaders and practitioners.

May 9, 2023
A Supportive Engineering Culture is Key for Recruiting and Productivity

Skilled engineers are in high demand, as their ability to build new products quickly is extremely valuable for businesses competing in a tight market. Strong engineering talent allows businesses to increase productivity and meet customer demand faster, which is essential for long-term success. However, in the current economic context, companies are being forced to reduce spending while trying to achieve the same results. This means engineers have fewer resources available to achieve their goals–increasing the pressure on their role. At the same time, engineering talent remains a valuable asset; in the UK, for example, just under half (49%) of engineering businesses struggle to find workers with a sufficient skillset for the job.

So, businesses must find a way to attract and retain talented developers while ensuring they feel supported through challenging times. A vital step to achieving this is creating a workplace culture that promotes developer satisfaction. Below, I explore how businesses can create a positive working environment for engineers, make transparent decisions and ensure their teams feel supported – to create a culture where developers want to stay and supercharge business performance.

The Key Pillars to Creating a Positive Culture

Leaders of all stripes are tasked with ensuring that their employees are supported, engaged and motivated to do their best work. However, the route to achieving these outcomes can sometimes seem vague and elusive.

Research shows that there are several key pillars for creating a strong organizational culture. The first is psychological safety. Developers must have the freedom to take risks without the fear of being criticized. Software engineering is an industry in which professionals must be creative enough to be able to create something out of nothing, so it’s important that companies ensure developers have the space they need to experiment and try new approaches.

Secondly, dependability is essential. Developers are more likely to produce high-quality work in a timely manner if their employer believes in them and their abilities. Developers often face immense pressure to ship functional code on a daily basis; feeling trusted can give them the confidence they need to take on more responsibility and drive projects forward. Alongside this, it’s the leader’s role to ensure developers have a very clear and structured mission. They must know exactly the goals they’re aiming to achieve, the part they’re playing in reaching these goals and the plan for execution.

Finally, ensuring developers understand the meaning of their work and are fully aware of the impact it has is an important pillar for cultural success. Sometimes, software engineers spend a huge amount of time on small fixes and updates that may seem insignificant. Business leaders must make efforts to communicate the importance of developers’ work and the value they bring to the business. If developers feel valued and rewarded for their difficult work, they will be more motivated to commit themselves to their mission and stay at the organization.

When engineering teams are happy and supported, they are also more likely to speak highly of their organization and encourage other developers to join the team, helping the company build a strong pipeline of engineering talent.

Making Transparent Decisions Using Concrete Data

Transparency is another crucial element to ensure engineering teams are happy and motivated to do their best work. Developers can’t do their jobs effectively if they don’t have insight into the workflows on their teams or don’t have context into decisions that impact their ability to perform effectively.

With tools such as software delivery intelligence platforms, developers can see the positive impact of their contributions and how they’re helping the business succeed. These tools also enable engineering managers to rely on concrete data to celebrate developer progress on previously invisible work and show how their work contributes to wider business success. This helps developers gain more autonomy to achieve the company’s goals, and eventually create better business outcomes.

For example, Manulife adopted an engineering analytics tool to create an engineering culture that is based on trust and transparency. The concrete data generated is used to analyze engineering processes and highlight talent that may have remained hidden without a data-driven approach. Data also helps managers decide who within the team deserves to be promoted based on precise insights into everyone’s contribution.

Use Data to Help Overcome Complex Challenges

The mission of developers is constantly evolving, with new technologies, programming languages and methodologies changing very quickly and making the job more complex over time. It is up to engineering leaders to help their teams remove potential barriers that are blocking them from being successful in their roles.

With data-driven insights, engineering leaders can better support and uplift their teams. Data about engineering workflows allows leaders to understand exactly where bottlenecks exist and remove them quickly before slowing their projects down. This means that employees can learn what worked–or didn’t work–from one project to the next, as well as develop the necessary skills to remove roadblocks to future success.

Timelines and targets for projects set by business leaders can also then be tailored to each person’s abilities and based on concrete data that show where they can boost efficiency to ensure everyone produces their best quality work and remains engaged and motivated.

In the current technology industry, employee culture is one big factor determining where developers wish to work. In a context where engineering talent is in high demand, it’s the job of every organization to create a stellar work culture to attract and retain talent.

May 8, 2023
Loosely Couple Your Schema and Keep Data Separate

In software development, one of the primary goals is to write clean, maintainable and extensible code. To achieve this, developers often strive to design systems that are loosely coupled and separate concerns into smaller, more manageable components. Loose coupling refers to the practice of designing components that are not tightly dependent on one another, allowing them to be modified or replaced without affecting the rest of the system. Separating data into distinct layers, such as a database layer, further contributes to this modularity by allowing developers to focus on writing business logic without worrying about data management. Together, loose coupling and data separation provide a solid foundation for building scalable and adaptable systems that enable developers to concentrate on writing high-quality business logic. This approach also lends itself to platform engineering best practices, a popular methodology that is being implemented or considered by developer teams today.

Loose coupling is an important concept in microservices architecture because it helps to achieve the goal of creating modular, scalable and easily maintainable software systems. In a microservices architecture, individual services are designed to be independent, self-contained and loosely coupled, meaning they can be changed, updated or replaced without affecting other services or the system as a whole.

There are several benefits to loose coupling in microservices:

• Scalability: Microservices architecture allows scaling of individual services independently based on their needs, and with loose coupling, the interdependencies between services are minimized, allowing for better scalability.

• Maintainability: With loose coupling, it’s easier to make changes to individual services without disrupting the system as a whole, reducing the risk of introducing bugs and making maintenance tasks easier to perform.

• Flexibility: With a loosely coupled architecture, services can be developed, tested and deployed independently, enabling teams to work in parallel and make changes to the system without impacting each other.

Platform engineering plays a crucial role in ensuring that microservices are loosely coupled. A well-designed platform should provide a standardized set of APIs, protocols and interfaces for service communication, as well as tools and frameworks for building, testing, deploying and managing services. This enables service teams to focus on developing their services rather than worrying about the underlying infrastructure and communication protocols.

Moreover, platform engineering can also ensure that there is a standardized and centralized approach to security, monitoring and logging, which are critical aspects of microservices architecture. By providing a consistent and standardized approach to these areas, platform engineering can help ensure that individual services remain loosely coupled and can evolve independently, without negatively affecting the system as a whole.

It’s All About Keeping the Schema Separate

In platform engineering, the concept of keeping the schema separate typically refers to separating the schema of a database or data store from the application code that interacts with it. This separation allows for greater flexibility and scalability in the platform, as changes to the schema can be made independently of the application logic. By decoupling the schema from the application code, developers can avoid introducing breaking changes to the platform, and can more easily evolve and scale the platform over time.

Separating the schema also allows for different parts of the platform to interact with the same data store using their own specialized schemas, without interfering with each other. This is particularly important in large and complex systems, where different teams may be working on different parts of the platform concurrently. By keeping the schema separate, teams can work independently and avoid stepping on each other’s toes, while still interacting with the same underlying data store.

In summary, it’s all about the schema – don’t share databases and tables. The separation of data is fundamentally driven by that desire for loose coupling. Microservices should be as independent as possible from each other from design to development to production. Why? Efficiency. Disparate teams can change data structures and will not affect others versus the more traditional approach where we do share and it’s more cumbersome and involves more coordination between teams.

Keeping schema separate in platform engineering is important for several reasons:

● Scalability: Each microservice owns its private schema, which enables the platform as a whole to scale more effectively. As the platform grows and evolves, it may become necessary to change the data schema without necessarily changing the underlying code. Separating the schema makes it easier to make these changes without disrupting the rest of the platform.

● Maintenance: Separating schema from code makes maintenance easier. It allows developers to make changes to the schema without needing to change the code, which reduces the risk of introducing bugs or errors.

● Collaboration: Separating schema from code makes it easier for multiple developers to work on the platform at the same time. When developers work on the code and schema separately, they can work independently without interfering with each other’s work.

● Flexibility: Separating schema from code provides greater flexibility for integrating with other systems or applications. It enables the platform to adapt to changing requirements or to interface with different data sources more easily.

Separating Schema From Code is an Essential Practice

By hiding data from service components from other service components, It provides guardrails and “just works.” While it does provide a form of coupling, it’s through messaging (typically asynchronous). Platform engineering encourages good design patterns from the start versus having to be more aware of these things. It’s a lot easier for especially newer developers that are not familiar with some of these architectural patterns and can get themselves into trouble.

Overall, separating schema from code is an essential practice in platform engineering that provides scalability, maintenance, collaboration and flexibility benefits. It allows developers to build more robust and flexible platforms that can adapt to changing business requirements and technical environments.

April 28, 2023
Don’t Make Big Tech’s Mistakes: Build Leaner IT Teams Instead

One of the biggest narratives in the U.S. concerning the current wave of tech layoffs is the fact that ‘traditional’ sectors have a prime opportunity to snap up new tech teams and IT talent. This has always been a significant challenge for sectors such as banks, insurance and manufacturing that compete with big tech for talent. In 2022, a majority of CEOs (71%) identified skills and labor shortages to be that year’s biggest business disrupter.

And they were right to be worried. According to Salesforce, three in four workers do not have the digital skills needed by businesses to prepare for the future of work. The temptation in this environment for many traditional organizations is to enter the labor market to build larger tech teams to help drive their modernization programs. However, this may be a mistake.

Building large, expensive, bloated tech and IT teams is not always the answer. It’s critical that traditional companies learn this lesson from big tech.

Instead, traditional organizations should consider other avenues to drive their digital transformation ambitions with their lean resources. What does this look like?

What is the Real Challenge you are Trying to Solve?

Since COVID-19 hit, U.S. companies have been forced to reevaluate how they work, with remote working becoming commonplace–even for companies who couldn’t imagine themselves working remotely.

And the challenges have grown as U.S. businesses seek growth to recover lost gains from the last few years. Evanta’s 2022 CIO Leadership Perspectives study found that CIOs rate data and analytics as their second top priority within the IT function. It is clear that business leaders see advancing organizational use of data, analytics and automation as key to reaching enterprise objectives.

For many organizations, the biggest challenge they face is the unprecedented amount of data generated. Critically, more than 80% of all new data is unstructured and messy. This makes it difficult for data-driven companies to accurately review, gather, analyze and rely on what is mission-critical information.

Traditional industries such as health care, banking, insurance or the public sector, have even larger concerns because data is increasingly critical to how they build value and deliver services for customers. So, it is important for their IT teams to create a single source of truth and data governance framework across their organizations to ensure the right people and senior leaders have the most accurate information to make smarter decisions.

For instance, in the manufacturing industry, no-code database platforms can be leveraged to structure daily operational data and visualize insights through digital dashboards for quality assurance, continuous improvement and health safety.

Additionally, at Baserow, we work with treasury customers at some of the biggest international banks that use no-code platforms to keep track of local legislation around the world by centralizing information in real-time databases available to relevant teams, including compliance.

Modern organizations depend on their IT teams to accomplish a lot of their goals. So, it is important to provide them with the tools to focus on more complex development projects. No-code tools free your non-technical teams to address their pressing needs themselves without having to wait for budget or resource availability in the IT department.

Have you Also Considered Upskilling Your Current Team?

We know from the Salesforce survey that the vast majority of U.S. workers lack the digital skills to support their organizations’ ambitions and needs. This suggests that there is a wider issue within most companies at large. Do businesses need to prioritize upskilling their current team instead?

A U.S. Gallup report found that 72% of workers are overwhelmingly interested in pursuing additional digital training to prepare them for an increasingly digitilized workplace. Is there room for no-code technologies to supplement the digital capabilities within companies?

Enterprises are already expanding the responsibilities of application development outside their IT teams as well as increasing the productivity of their existing IT resources by automating and abstracting repetitive developer tasks. Gartner predicted that by 2025, 70% of apps will employ low-code/no-code technology. This trend aligns neatly with the business needs, such as the development of organization-specific applications and tools, because every business’s requirements are different.

Furthermore, no-code/low-code technology enables companies to drive innovation and shorten the idea-to-market cycle times. The technology’s pre-built components make it easier for end-users and workers to intuitively assemble the templates that solve their complex needs.

In Conclusion

U.S. companies that need to fill the IT skills gap should consider how technology can be leveraged to boost productivity and accelerate the speed of innovation while accepting that new technical resources will continue to be expensive and scarce.

These organizations are better off taking the lead in the democratization of IT within their teams. Only then can they keep up with the speed and agility required in today´s competitive environment.

March 27, 2023
How to Supercharge Your Engineering Teams

With the economy still bleak, data and engineering teams that were already struggling to keep pace with business needs are now facing a potential hiring freeze or worse in 2023. That’s why there’s never been a more important time to maximize the productivity and impact of your teams, but the question is how to identify the big changes that move the needle in performance. Incremental adjustments that save a sliver of time here and there are not hard to come by, but what teams need are the breakthroughs that yield giant leaps in productivity and efficiency. Rather than a 10% improvement, it’s the 10x improvements that drive lasting gains.

As an engineer and CEO of a software company, I’m constantly looking for ways to boost the productivity of our teams. Our guiding principle is simple: Maximize the impact per hour invested for every individual. To do that, we must ensure that individuals maximize the time spent flexing their superpowers, minimize busy work and reduce dependencies on others that delay work or tie them up in meetings. While counterintuitive at first, we consistently find that collections of well-aligned, independently executing individuals produce a far greater impact than a large team swarming on a project. Put another way; the individual impact goes up when the number of people you depend on to deliver the result goes down.

I’ve developed an approach over the years that has allowed me to identify several opportunities for step-change advancements in productivity and efficiency, and I’d like to share it here, along with two concrete examples. I will also share an exercise we run with our teams to help identify and execute these opportunities.

First, here are my two main criteria for identifying the projects and tasks where we can effect changes that will have an outsized impact on the company:

● Big resource-consumers: Work that’s essential to the business but consumes a large amount of engineering and other resources is a top candidate for optimization.
● Big impact, easy win: Areas that will have a significant impact on company performance if we can do them faster and with higher frequency are also great candidates—even if they’re not consuming a ton of resources.

Here are two examples that illustrate each of those criteria and where we were able to see massive gains in productivity, quality of service, or both.

Tackling Big Resource-Consumers

At Ascend.io, we provide a platform for building and automating data pipelines, which requires us to build connectors into a huge variety of data sources. At one time, building a new connector required multiple engineers to spend two to three weeks writing more than 5,000 lines of code that spanned four microservices in three different languages. As we added more customers and needed to support more data sources, this quickly became unsustainable.

When we looked at this process, we saw two issues: Engineers were writing custom code for each connector in a way that would be better served by a common framework. Moreover, the process required engineers from multiple teams and disciplines to work together and coordinate their priorities. As soon as you get dependencies that require engineers to coordinate with others and wait on the outcomes of other people’s work, you’re wasting time.

We addressed this by designing a new development framework that exploits commonalities across the connectors we build. This now requires far less unique code and removes the need for multiple engineers to work together. In most cases, one person can build a new connector in days with 100 lines of code. This made us far more nimble and freed up engineering time to do other valuable work.

Achieving High Impact

Like most companies building software, we need to convey the capabilities of our product to new and existing users to ensure they receive maximum benefit from their use of Ascend. With new features and enhancements being released on a weekly basis, there is a lot to keep up with. This is high-impact work, however, and if we can do it in a vastly better way, it drives an outsized impact for Ascend and our customers.

For us, building these automated product tours typically required a product marketer or product manager to design the customer journey, and one or more engineers on a sustained basis to build the actions into the product. This again creates dependencies that slow progress and require significant manual coding.

The breakthrough for us was a tool called Intercom, which allows you to build sophisticated product tours without needing to write code. Our product team can now build more interesting and complex product tours for customers, and our engineers can stay focused on building new features instead of the code needed to showcase them. What’s more, they are able to design, test and deploy these tours independent of our product and code deployment cycle, further freeing them to iterate as fast as possible.

A Thought Experiment

So how do you actually find these opportunities for 10x improvement? I run the following thought experiment with our teams several times a year to uncover these opportunities:

Imagine your workload scales 10x overnight and you absolutely must meet this increased need, but without hiring more engineers or working more hours. How do you do it?

It’s amazing what this exercise can uncover—but there are ground rules and it can take time to see results. Coming up with these big ideas is a skill like any other, and most people are not great at it the first time they try it. Your first iteration may produce no options to pursue—some may have high-impact ideas but with even higher investment requirements, while others may reject the premise outright. That’s normal. So, don’t insist they sit in a room until they dream up that 10x breakthrough, because it will be demoralizing. Instead, plant the seeds by establishing the above framework and then let the ideas start to grow.

More likely, a few days later, they’ll be out for a hike or chatting with an old co-worker and they’ll suddenly have the germ of an idea. That may still not be the winning solution, but someone else will build on it, and then someone else and, finally, you iterate your way to a breakthrough. Pulling the team back together and embracing the “yes, and” approach can be both energizing and highly productive.

Brainstorming like this is fast and free, and you only need one great idea to have a material impact. We run this exercise a few times a year and have come up with several big ideas that were game-changers for the company. Besides the examples above, we also moved key parts of our software development from Scala and Go to Python a few years ago, which is a much more efficient language (as measured by developer time) and allows our engineers to build things much faster.

Software engineering is a complex beast that often spans many teams working on multiple processes in parallel. But that complexity is also your opportunity: Every complex system has room for optimization, especially at companies that moved fast in the beginning to build a product and find market fit. Now is the time to lift your gaze, look around and ask where significant gains can be made. They almost certainly exist, and a 10x improvement could be what allows you to not only survive the turbulent year ahead but emerge with a thriving team and business on the other side.

March 27, 2023
Data APIs: Realizing the Future of Data Warehousing
Data is the currency of business in today’s digital economy, with organizations collecting mountains of data from their customers, products and services. Enterprises are increasingly turning to data warehouses to store this valuable enterprise data and to make it useful and actionable. Data APIs–both GraphQL and REST–are emerging as a tool to power frictionless and high-quality integrations between applications/services and data in enterprise data warehouses. But what exactly is driving this shift toward using APIs on top of data warehouses? In this article, I will talk about the drivers behind the rise of data APIs on data warehouses, challenges with current approaches to building, operating and scaling data APIs and how organizations can address those challenges.

The Rise of Data APIs

Data APIs enable organizations to create powerful integrations between applications without having to go through lengthy development cycles or write complex code. As a result, businesses can deliver improved user experiences faster than ever before by quickly connecting disparate applications and services together using intuitive REST or GraphQL endpoints. Because these endpoints are standardized across multiple systems (e.g., REST or GraphQL), developers don’t have to learn new languages or coding techniques every time they want to connect two different applications together. This makes it much easier for developers to work with multiple systems simultaneously while still ensuring that all applications remain connected via one interface–the API endpoint itself.

Challenges With Current Approaches

Despite all these advantages provided by Data APIs, there still remain a few challenges that organizations need to consider when building out their own solutions. For example, many existing approaches require manual setup processes, which can be extremely time-consuming and tedious for developers working with multiple systems at once. In addition, many existing solutions only allow for limited query functionality, which means that more complex queries may not be possible without additional coding or manual setup processes being performed. Finally, some existing solutions require an extensive amount of organizational resources in order to build out a full-fledged solution—time that could be better spent on more important tasks like product development or customer service initiatives.

How Organizations Can Address These Challenges

Fortunately, there are now tools available that make it easy for organizations to quickly build out powerful data API solutions with minimal effort required from their developers or IT staff members. These tools provide pre-built templates that can be used as a starting point for any integration project while also allowing users to customize their own solution based on specific needs and requirements without having to write any code themselves. Additionally, these tools allow users access to advanced query functions such as joins across multiple tables so they can get the most value out of their databases without compromising either speed or security either. Finally, these tools also include powerful monitoring capabilities so users can ensure that their data APIs remain running smoothly at all times, no matter how much traffic they receive.

Benefits of Using an API for Integrating Cloud Data Warehouses

When it comes to integrating cloud data warehouses into your application or service, there are several advantages that come with using an API as opposed to traditional methods such as manual coding or ETL tools:
- Increased Flexibility: By leveraging an API, you are able to access data from multiple sources quickly and easily without having to manually code every integration yourself. This also allows your application or service more freedom when it comes to accessing additional data sets in the future; all you need is an existing API connection in place.
- Improved Security: Since APIs are designed specifically for secure communication between different systems, they offer enhanced levels of protection against malicious attacks such as DDoS attacks or unauthorized access attempts. You can also add additional layers of security by implementing authentication protocols like OAuth or JWT tokens on top of your existing API connection.
- Enhanced Performance: APIs can be designed with specific performance optimizations in mind; this means that you can create custom solutions tailored towards the exact needs of your application or service which will ultimately result in improved end-user experiences and higher engagement rates from customers who interact with your product or service regularly.
Data warehouses have become an essential part of modern digital businesses due to the sheer amount of valuable enterprise information they store. However, traditional approaches can often lack the flexibility or scalability needed in order for businesses to stay competitive in today’s ever-changing market environment. This is why smart organizations are using data APIs on top of them for quick integration among disparate applications/services along with advanced query capabilities.

By leveraging tools designed specifically for this purpose, companies can easily overcome current challenges associated with building out their own solutions while still getting maximum bang for their buck when it comes down to successful implementation and operation over the long term.
March 23, 2023
How Database DevOps Fuels Digital Transformation

In today’s real-time, hyperconnected reality, data is a core driver behind most digital strategies. User data is constantly gathered and shared across various technology backends and platforms—as is system performance data to inform engineers. Data is pivotal for everything, from generating machine learning models, refining user experiences and even supporting new data-centric services. But although data is a clear backbone to many enterprises, there is no clear winner regarding which database to use.

Large organizations often find themselves supporting different database styles, from relational databases to NoSQL and distributed SQL. And as organizations continue to adopt multiple databases, safely managing the processing and storage of data becomes more challenging to enact at scale.

I recently met with Jakub Lamik, CEO, Redgate Software, to understand the role DevOps, specifically database DevOps, is playing in advancing data-driven digital transformations. According to Lamik, applying DevOps practices for database management can improve performance, increase productivity and help organizations stay compliant. Below, we’ll consider the state of today’s data-driven digital transformations and highlight ways to improve how DevOps is applied to database management.

State of Data-Driven Digital Transformations

The 2022 StackOverflow Developer Survey found the most popular databases to be MySQL, PostgreSQL, SQLite, MongoDB and Microsoft SQL Server. But most often than not, businesses deploy more than one of these in production. Redgate’s 2021 State of Database DevOps report found that 70% of businesses had more than one DBMS in use, and 48% had teams working on three or more different databases.

Engineers operate in increasingly hybrid and distributed technology environments. This isn’t for no reason—in fact, 90% of organizations said multi-cloud is helping them achieve their business goals. Lamik also notes a diverse array of software technologies in practice today, from multiple clouds to decoupled microservices architectures and database types. But the desire to work with diverse toolsets causes fragmentation, which could hinder software delivery efforts. For example, organizations want to deploy the same framework and similar methodologies across different databases, said Lamik, and have it be database- and cloud-agnostic.

Tips to Improve Database DevOps Operations

To increase the pace of software delivery, infrastructure automation is often needed. And like other roles, database professionals are increasingly trying to automate operations and free themselves from mundane tasks. So, what are some ways to streamline database DevOps? First off, Lamik encourages adopting a common approach across multi-database environments, using standard toolsets that are flexible and portable. “The organizations that standardize deployments across different database environments can increase the frequency and reliability of deployments,” he said.

Catching and resolving errors quickly is also becoming increasingly essential to meet objectives. And although DevOps brings many advantages, the CI/CD pipeline has gaps. “Observability supported by intelligent automation is key to closing those security gaps,” said Lamik.

Thus, he encouraged additional oversight and tracking of changes to ensure stability throughout the database deployment pipeline. Also helpful is encouraging collaboration with security professionals (also known as DevSecOps) and integrating security principles into the DevOps workflow to ensure the process is secure by design.

Another element of database security involves the proper observability and classification of sensitive data. As Lamik describes, organizations must be careful to distinguish between data used for production purposes and data used for development purposes. Ensuring sensitive data is cataloged correctly is imperative to avoid production data from being breached or accidentally publicly exposed.

Final Thoughts

In summary, applying DevOps ideals to database management could be a boon for development efforts. For those who haven’t yet considered it, a DevOps transformation will pay dividends, says Lamik. Furthermore, a common management layer could help wrangle multiple database types and help adopt new emerging database types, such as Snowflake and Postgres. Yet, when acquiring new database toolsets, Lamik admitted it may not be a quick transition. He recommends a land-and-expand strategy rather than a top-down approach (which typically isn’t as effective in empowering individual developers).

Commenting on the industry’s recent move toward platform engineering, Lamik sees it as the next evolution of DevOps. Whereas DevOps brought helpful toolsets to streamline deployment and digitization, platform engineering is taking these practices and productizing them to be reusable at scale across teams and departments, he explained.

Also, in the new cloud-native stack, containers are designed to be ephemeral, causing engineers to craft workaround solutions to enable stateful storage on platforms like Kubernetes. Regardless of deployment, data must have proper backup and recovery controls as well. In this new cloud-native paradigm, the role of database professionals is changing, explained Lamik, and “APIs are becoming the focal point of communication.”

March 17, 2023