devopsy.kinsta.cloud

Tag: app development

Steady On. We Still Have a Job to Do

It is all the rage to talk about how significantly large language models (LLMs) and generative AI will change every niche and corner of IT. And change IT it will. But the hype cycle is at its highest right now, and I’d caution against getting too worked up about it. We’re in for a few interesting years, and the spaces I cover (DevOps and security—with a more application-centric perspective but with ops experience) are definitely going to benefit from the various AI bits being worked in. All testing—including AppSec, but all testing—will benefit both in test generation and results analysis/filtering. In fact, I’ll go ahead and say that the question in testing will no longer be, “How much time to we have to devote to it?” and will rapidly become “How many systems’ resources do we have to dedicate to it?” It will still be a long process—AI isn’t speeding up the cumulative time it takes to run thorough testing (be it functional or security). It will make that the bottleneck though, because test generation will at least be LLM-assisted, and will eventually become completely generative AI-created while at the same time, results will get more and more filtering and analysis. A minor result here plus a minor result there will automatically be detected as an extended vulnerability. There are a ton of other possibilities, but these two seem pretty obvious when I look across the testing technology space. More testing and better results analysis are on the horizon—without requiring more staff. Of course, you can do so much more, which would require the organization to add staff to follow up on results, but that’s nothing new; testing finds issues that are interruptive and take man-hours to fix—we all know of bits in the application portfolio that we should probably fix but that aren’t high enough priority to do so, for example. But fixing is different than finding. I have seen demos in security scanners that offer auto-fix options for developers. Run the test during CI, create a ticket and let the developer see suggested code rework right in the IDE. It is early days for this, but if you look, you can find vendors doing it. And that’s really rather cool.

But we’ve all heard the “this changes everything” storyline over and over. 4GLs, XML, Java and JavaScript were all going to eliminate developers. In fact, they all created more developers. AI has a better chance of actually removing parts of our workload than any of these technologies did, but we’re still in “watch and wait” mode. It seems obvious that grunt coding will fall to LLMs … And aside from keeping staff trained to manage source, that is a good thing. We use open source, modules and libraries to do much the same thing today. I see this as equivalent to entry-level writing jobs. We’ll have to transform from the current model, but the current entry level development model wasn’t ideal anyway, so lets make a better one. Testing capabilities will be greatly enhanced, but testing of all kinds was so far behind new code generation at most orgs that the impact will be all positive as testing catches up. AI-based test generation today can generate thousands of permutations for a given piece of code, and analyze the results to just have analysts look at the truly disconcerting or confusing test results. And the question will become, “How many servers do we want to dedicate to run this?” Which is different, but just means we have more tests running to make a more solid and/or secure product and environment.

So, in summary, is change coming? Yes. And that is not significantly different than the last couple decades where change has become IT’s constant companion. For most of you, the way you do your job today is not the same as the way you did it even two years ago. ML/GAI/LLMs are just another iteration. The apps still have to be built, deployed and maintained. So until HAL comes along, keep kicking it. You’re keeping it together, and who knows what the future brings.

September 6, 2023
Organizations Prioritizing Java Application Modernization

A survey found that modernizing existing Java applications is a higher priority than other projects for 87% of respondents, with more than 80% of these applications expected to be re-engineered over the next five years. The survey polled 450 IT professionals working in organizations in the U.S. and United Kingdom that have more than 500 employees.

Conducted by the market research firm Vanson Bourne on behalf of Diffblue, a provider of a generative artificial intelligence (AI) platform for refactoring applications and generating unit tests, the survey also found enterprises plan to invest nearly half (45%) of their software development resources on modernization of existing applications versus developing new ones. A full 90% of respondents expected to rely on automation to accomplish those tasks, the survey found.

Specifically, survey respondents reported they planned to rely on automation to improve security (50%), increase development velocity (45%) and improve developer experience (42%). Well over half of respondents (58%) said they had already been working on these projects for more than 12 months.

If organizations don’t modernize Java applications, more than half of respondents (51%) said too much time would be spent on unproductive coding, while 50% said development velocity would be adversely affected. The biggest obstacles cited are choosing the right technical approach (45%) and securing budget (42%).

More than half of organizations (59%) have created a dedicated modernization budget to support their efforts. Nealy half (48%) said funds are being provided by business teams for projects, while 49% said general operating budgets have increased.

Diffblue CEO Matthew Lodge said it’s clear there is a huge commitment to application modernization. The reality is the majority of work IT teams do involves applications that have already been deployed. The challenge they face is that the effort required to modernize these applications is still far too dependent on manual processes, he added.

Overall, the survey found 96% of respondents considered Java applications extremely or very important to their organizations. On average, 65% reported additional Java applications are still being developed. Only 4% said they planned to migrate away from Java as part of a modernization effort.

Over half of the respondents expected to use Java 11 (53%) or Java 17 (56%) in their applications, compared to 19% using Java 8. More than a quarter (27%) expected to use a forthcoming Java 21 release. A full 100% expected to use the Spring framework to build these applications, with more than half (54%) planning to upgrade to the latest version that supports Java 17. A full 97% said unit testing is an “extremely” or “very important” element of their application modernization effort.

After experiencing something akin to a mid-life crisis, Java has itself been modernized with the release of Java 17, noted Lodge. The Java community is now adopting many concepts that may have been pioneered in other programming languages, he added.

Regardless of whether organizations are modernizing existing Java applications or writing new ones, the one thing that appears certain is the venerable programming language isn’t about to disappear any time soon.

February 14, 2023
Playwright: A Modern, Open Source Approach to End-To-End Testing

I was excited when I started writing my first end-to-end tests years ago. The idea was promising; create an automated test suite that spins up a browser and mimics your user’s behavior. Test your website end-to-end. Frontend, backend, your database—you name it. All these automated tests would cover everything! Unfortunately, it wasn’t all roses and sunshine. The implementation was more complicated than expected and the results were not as stable as I had hoped. On one hand, I expected that learning a new testing framework would take time. But on the other hand, investing time and money just to end up with flaky tests and countless false positive alerts was not acceptable.

Luckily, testing frameworks evolved, and new libraries make end-to-end testing more approachable and, most importantly, less flaky. One of the more recent solutions is Playwright. Let me tell you more about it!

Write Your Test Code in Four Different Languages

While Playwright’s default choice is Node.js, it offers support for three other languages: Java, .NET and Python. You’re in good hands if your team uses one of these languages because no one wants to learn a new language just to write some tests, right?

Cross-Browser Tests on the Back of the Chrome DevTools Protocol

Contrary to other solutions, Playwright doesn’t use the WebDriver protocol. It leverages the Chrome DevTools protocol to communicate with Chromium browsers (Chrome/Edge) directly. This approach allows for more direct and quicker communication. Your tests will be more powerful and less flaky.

But Playwright doesn’t stop at Chromium browsers. The team behind the project understood that cross-browser tests are essential for an end-to-end testing solution. They’re heavily invested in providing a seamless experience for Safari and Firefox, as well, and even Android WebView compatibility is in the works.

Testing your sites in Chrome, Edge, Firefox and Safari is only a configuration matter. And this saves time and headaches!

Fast Tests Due to Auto-Waiting

It’s not only about automating multiple browsers, though. If your tests are hard to write because you have to place countless “sleep” statements everywhere, your test suite will take hours to complete and become a burden.

To avoid unnecessary waits, Playwright comes with “auto-waiting.” The idea is simple: Instead of figuring out when a button is clickable by yourself, Playwright performs actionability tests for you.

Let’s look at an example:

“`
await page.click(‘[data-test=cta]’)
“`

There’s nothing fancy about clicking a button, right? Except in Playwright, there is. This innocent “click” call includes additional convenience functionality. When you call “click,” the framework checks and waits for a matching DOM element to be:

– attached to the DOM
– visible
– stable (it’s not animating or moving)
– able to receive events (not hidden or obscured by other elements)
– enabled

All these checks guarantee that your target element is ready to roll when you want to interact with it and, more importantly, it’s clicked as quickly as possible. Don’t worry about DOM elements. Just interact with them and let Playwright figure out the rest—no more wait statements!

Auto-waiting is built into all of Playwright’s interaction methods so that you don’t slow down your test suite. Your tests will be easier to write and run as quickly as possible.

But that’s not all! While I use Playwright via the command line, it also comes with handy tools to make end-to-end testing easier.

Valuable Tools to Create, Debug and Analyze

Playwright provides additional commands and visual tools to make the writing of end-to-end tests as accessible as possible.

Record Scripts With Codegen

To give you a head start with your first test, use Playwright’s test generator.

“`
npx playwright codegen playwright.dev
“`

Only a single CLI command is necessary to spin up two new windows: An interactive browser and the Playwright Inspector. The fun part is that every interaction in the browser window will be recorded and shown in the inspector.

If you want to create a Playwright test that navigates somewhere, clicks a button and fills out a form, these tools are perfect for creating an initial setup.

Replay the generated script, adjust your selectors and assertions, and you’re good to go!

Congratulations! You just created your first end-to-end test, and I bet it didn’t take you more than a few minutes to get there.

Debug Your Scripts While Running Them

When your test suite grows, you’ll find yourself in situations where you must debug complex scripts. For these cases, Playwright provides a handy debug view.

Enable debug mode by explicitly calling `test –debug` or defining the environment variable `PWDEBUG=1`.

“`
npx playwright test –debug
PWDEBUG=1 npm run test
“`

Playwright then opens a browser and the inspector walks you through your test case, similar to your editor’s debugger. Additionally, it highlights the touched elements with a little red circle so you can see what was clicked or filled.

This approach is perfect for debugging tests during local development. But what about debugging failed tests that run in CI or on a different machine? There’s a solution for that, too!

Analyze and Debug Previously Failed Tests

In an ideal scenario, you’ll run your tests automatically on commit or deployment in your CI/CD pipeline. But debugging failed tests from a different environment can be a massive challenge.

Luckily, the Playwright framework allows you to enable a trace mode. Trace mode records all script actions while screenshotting and keeping the website’s state.

Then, if your tests fail on your remote servers, use the provided Playwright tooling to load a generated `trace.zip` file and inspect the test to find out what failed. Method calls, a timeline and screenshots will be available—it’s time travel debugging at your service!

A Refreshing Approach to Synthetic Monitoring and Testing

These are only a few things that make Playwright my testing framework of choice today and the team behind it develops fast and ships new releases every month.

If you want to learn more about the Playwright framework or to chat with the maintainers, there’s a very active community Slack channel with many friendly people.

So, do you still avoid end-to-end tests because they used to be slow and flaky? I hope that I could convince you to give it another try. Because there’s nothing better than knowing that you didn’t break things while developing and shipping software.

September 8, 2022
A New Development Model: Bringing SaaS Apps to Data

The last few decades have seen multiple major shifts in how applications are built and deployed, and now we’re seeing the start of a new shift that opens up enormous opportunities for all kinds of developers. Instead of bringing data to applications, developers are bringing their applications to the data.

Let’s back up. In the 1990s, as data grew too large for local machines, applications moved with the data to data centers—a model more like the mainframes that preceded them. Then, in the 2010s, those same applications migrated from data centers to the cloud, bringing the data with them again. Now we’re seeing a third major shift, where customers take back control of their data and centralize it in the cloud. And developers are picking up on this trend, building their applications to not control their own data, but instead to connect to the customers’ data, letting them have and control a single copy.

In a traditional SaaS application, the SaaS vendor stores and processes data in their own platform, even if that platform is running on one of the major public clouds. A traditional CRM (customer relationship management) tool, for example, will use its own data platform to store all of the data that all of its customers are managing with the tool. When a customer wants access to their data, often to join with other data sets, they either have to export the data (via APIs or CSVs) or upload the other data set into the tool and use whatever analytics capabilities the tool has built. This is painful for customers and also painful for the developer who has to support use cases that are further afield from traditional relationship management.

In this new world, the customer runs and the applications all connect to the same data platform, storing and processing their data there. This means that joining two data sets is a single query away, with no movement of the data required. It also means that the customer owns and controls the data, making security and governance hurdles much easier for developers to overcome. And, because the customer pays for the data platform, the developer doesn’t have to mark up the costs and is able to command significantly higher margins.

While it’s early, this shift is already underway. Fast-growing startups like Supergrain, Anecdotes.ai and Rudderstack, to name a few examples, are taking full advantage of this shift and leading investors have picked up on it as well.

Why Now? Access to the Best Data Platforms Available

There are two key trends that are leading to the emergence of these connected applications: The wide adoption of cloud data platforms and the prioritization of control over data by customers.

In the last ten years, a new class of highly scalable and cost-efficient data platforms has emerged, allowing enterprises of all sizes to adopt technology that was previously only available to the most sophisticated vendors for use inside their own products. Previously, SaaS vendors had to roll their own complex data platforms that were very specific to their use case and charge high prices to cover the cost of running those platforms. Today, we’re not only seeing SaaS vendors migrate to cloud data platforms but also enterprises adopt the platforms themselves. This means that, for the first time, every enterprise has access to the best data platforms available.

We’re also seeing a huge change in how enterprises think about and secure their data. Partly because of regulations, and partly because of publicity around data breaches, large and small companies are paying significantly more attention to how secure, how controlled, and how well governed their data is. When your data is spread across dozens or more different SaaS applications, it’s nearly impossible to guarantee its security and access controls. Centralizing that data into a single platform, and connecting your applications to it, allows for a single point of control and security for every piece of data.

What’s In It For Developers?

It’s clear that connected applications are a big win for customers: A centralized source of truth with strong security and control. But this model has enormous benefits for application vendors as well.

First, it changes the go-to-market dynamics by letting you tap into work that customers have already done to centralize their data. According to George Xing, co-founder and CEO of customer engagement platform Supergrain, CEPs built with traditional SaaS models suffer from several drawbacks including long implementation times, poor reporting, and higher costs. The connected app model makes Supergrain more competitive, he said.

“Instead of moving your customer data to a third-party data platform, we bring the marketing software to your data warehouse, unifying source-of-truth customer data and marketing product experiences into one cohesive system,” Xing said.

It also accelerates time to market. Developers no longer need to build and maintain a data platform for their customers and can focus instead on their core product differentiators. They also get to leave responsibility for data security and compliance to the customer and their data platform.

With the customer managing and paying for storage and compute directly, vendors can significantly increase their margins, charging for the value they provide, rather than trying to mark up the compute they’re consuming.

What’s Next for SaaS?

The incredible growth in the amount and types of data available to companies is only accelerating this shift in how applications store and operate on their customers’ data. We’re going to see more and more SaaS vendors support connected application deployment models, but it won’t replace vendor-managed models. Instead, there will be opportunities for both models to flourish depending on the customer and specific application. We expect that many vendors will offer both models, just as we’ll see others go even further, bringing their entire application—including its UI, logic, authentication, etc.—fully into the customers’ data platform.

The opportunity for companies that take advantage of these new models is enormous, and we’re in the very early days. But the applications that take full advantage of this shift will be the winners of the next generation of data apps.

August 24, 2022
There are Few Enough Silver Bullets

I was working through this week’s blog this morning, and it was laser-focused on a narrow topic. I had examples of why too much of a good thing is bad, how absolutism about methodology is hurting the majority of organizations out there, and how to get past this issue to keep improving what IT does for the organization.

And I was nearly done when I decided that there are far too many areas of IT that suffer from absolutism, so I’d offer general symptoms and steps that work to get out of any of them.

What do I mean by ‘absolutism’? Well, it used to be—and, in a couple of areas, still is—true that people would say, “We are an X shop.” And mean it. If you were a Cisco shop in the 1990s or 2000s, that is all you bought. Competitors were not allowed; nor was consideration of competitors. This applies to methodologies as well as product categories. In fact, today it applies to various methodologies more than products.

And it’s bad for the organization, no matter where it comes from. While with vendors, it streamlines purchasing and offers the (in)famous “single throat to choke”, it also causes inefficiencies in costs (from your perspective, they are a monopoly, and some large vendors gained infamy by trying to take advantage of that fact), and causes people to do things in the least efficient way because that’s what the chosen methodology/solution requires.

Sometimes, we manage to do the dance and avoid “One True Way”—take the move to cloud and continuation to multi-cloud, for example. While cloud vendors do not offer customers much leeway in pricing or priority (we customers are all cattle, not kittens), multi-cloud still offers us the opportunity to use the cheapest or best-suited cloud vendor without much drama.

Many times, we are not so effective at avoiding the idea of a silver bullet. Take Agile. If there was a One True Ring of this decade in IT, I’d want it to be containers, but it would actually be Agile. Most IT departments zealously guard that Agile is the One True Way. Early-stage new product development is astoundingly well suited to Agile, and we needed it—if just for that part. Late-stage product maintenance, where changes are not feature-laden monuments to systems complexity, are also really well-suited to Agile and we should be happy to have it. In between? Large updates that can span data centers—or even continents—with a geographically dispersed workforce and hundreds (or more) of hours of development to get one sub-feature completed? Not really a great fit for Agile, and while many of us have said that for years, here is hoping things are changing. As I’ve said before, as maligned as waterfall was, it did the job for decades and handles large complex projects well. Maybe we need a replacement more in line with the times; one that incorporates what we’ve learned from Agile. I’d love that, actually. But what we don’t need is “We’re an Agile shop.” It is less useful than when it was said about products (at least if you only chose a single product, you got single sourcing and streamlined purchasing. With methodologies, you don’t even get that).

Note: Agile is an example, not the point of the post. Avoid “One True Way” across your organization. Root it out and examine it. Sometimes, a single solution with a single specialized skillset is the perfect answer—at least for now. Far too often, “We are an X shop” is someone’s preference codified into corporate policy.

To get past One True Way, you have to be there showing its weaknesses to management. There is some skill overlap when allowing multiple approaches or products, but if management starts prattling about that, simply point out the number of development and/or CI/CD environments the org is currently supporting. To show the overhead, track hours spent. Staff shortages have been endemic to IT for years. Make certain you know how much time is spent in overhead for the product or process, and try to map that to what could have been achieved with that time. It’s not perfect, but it will be illustrative. And don’t stop. Do your work, but point out every time that One True Way makes the job harder or more expensive.

You are nailing it. Don’t let things like Agile keep you from churning out great apps for the company. I’ve been on teams where the time spent in process overhead was dragging down the ability to deliver quality. Fight that. Keep kicking it, and use the best tools/processes for the job.

June 8, 2022
Update Those Ops Tools, Too

We’re all pretty well settled on the idea that the rate of change in IT has been on the increase over time. From the advent of VMs to Agile to DevOps to containers, the movement has been faster—and the same is true for internal development.

I’m going to approach this topic from three perspectives instead of my usual two. Kind of like breaking the fourth wall in cinema. We’ll discuss this movement just a bit from the perspectives of my two preferred bits, DevOps and DevOps. But for this blog, we’ll also mention DevOps—the infrastructure, tools and processes that enable the other two.

For DevOps, languages—both updates to those currently in use and the addition of new ones—bring with them a nearly automatic update to tools used. Favorite libraries/modules, IDEs, build tools—these are all looked at and updated more frequently today than at any point in history, by my estimation. For teams that do not change languages or libraries much, source code analysis (SCA) scanning is increasingly helping to take a closer look at all those included tools and filter them based upon impact (normally security impact, but other impacts on occasion, too—like project activity and stability).

For DevOps, the toolchain is still new enough that evolution is causing regular change, and that change is bringing with it new tools and utilities to solve the problems of this existing infrastructure. In fact, I’ve long thought the DevOps part needed a bit of a cooldown on the change and new tools/utilities front, personally.

And finally, DevOps has gone a bit sideways in this regard. Every person—whether you call them ops, admin or DevOps—responsible for troubleshooting machines and apps has a collection of tools. But the divergence of platforms has created a divergence of tools. It is possible to work without a dedicated set of tools—and, indeed, many DevOps engineers do. Others have a folder. Meanwhile, there is another dedicated set of DevOps engineers that have a PCMCIA or USB full of useful tools that is always with them. Some have both. Of course, what tools are included in all of this varies widely also. It is an interesting artifact of our growth in IT that developers tend to share or borrow the best libraries/modules/APIs across an organization, but DevOps is more independent. As long as DevOps team members are getting the job done, no one is all that worried about which tools they used to troubleshoot. They do tend to share if asked, but with little physical evidence of what tools they used, it doesn’t come up often.

So, what am I getting at? Look for ways to enable DevOps tool sharing on an ongoing (or at least regular) basis. An internal social media channel dedicated to the vendor your teams use to communicate, brown bags, a maintained rating system “Why I think fdisk rocks” type of thing, etc. The more power there is at the fingertips of every DevOps member you have, the better. Given our history, building an “approved” list is likely a bad idea, but having that central communication location that security can also peruse to look for red flags is not a bad idea. Of course, they are likely too busy, but worst case, such conversations can help in forensic analysis later. Not being cynical, just living IT in the 21^st century.

And keep kicking it. More and better tools will help keep all those sweet apps and servers clipping along, so there is really only upside here. Give your teams the tools and revel in your considerable uptime.

May 11, 2022
Bringing z/OS Systems into Today’s Modern Application Development

Bringing z/OS Systems into Today’s Modern App Development

Modern application development is all about automation, addressing the concerns of developers and the operational side of the house, and supporting the common CI/CD (continuous integration/continuous delivery) approach needed to keep apps up to date.

Developers typically make use of a variety of open source and other tools to aid in their work. While most of these tools were created for traditional server or cloud-native environments. Some of the most powerful tools are available to run natively in z/OS environments.

One example is Ansible, an open source community project for app deployment, configuration management and orchestration. It is available for z/OS.

The IT automation engine works by connecting to your nodes and pushing out small programs called “Ansible modules” to them. These programs are written to be resource models of the desired state of the system. Ansible then executes these modules and removes them when finished.

Simply put, Ansible allows developers to automate z/OS applications. With Ansible, developers can use existing JCL, REXX and z/OSMF to automate z/OS-based software. This lets the developer take a common approach to hybrid applications and integrate z/OS into an enterprise automation strategy consistently.

A great advantage is that developers can work in Ansible and Python. These are readily available skills in the marketplace and they can be applied to z/OS applications.

Another common tool is Jenkins, an open source, server-based tool that builds and tests software continuously, supporting CI/CD. Jenkins integrates with a broad range of source code management systems such as Git and Subversion. Jenkins’ capabilities can be extended to different platforms and systems using plugins. One such z/OS connector is the IBM z/OS Connector, a plugin for connection of Jenkins CI to IBM z/OS, including integration of IBM SCLM as SCM.

With tools such as these, developers can build a CI/CD pipeline that, for example, calls open source scripting that you have running on Z. It might move some files from A to B, or it might call a compiler. It helps someone familiar with these tools carry out these tasks and incorporate them into their processes. If they are running on the z/OS platform, developers can interface and make calls right on the platform. For example, an app may take advantage of existing mainframe Rex scripting without having to rewrite it.

Accounting for Cultural Differences

In today’s always-on, instant-access-to-everything world, employees and customers demand frequent updates and new features to existing applications and services, as well as new and better offerings all the time. Meeting these demands is a competitive differentiator.

Bringing commonly used open source development, workflow automation and CI/CD tools to the z/OS environment helps businesses speed the development of new applications to meet changing demands of their customers. It allows a business to leverage the data and applications on its z/OS systems and create innovative new applications.

It’s also about providing the tooling to DevOps that they are familiar with and providing tools to build z/OS applications in the same way they build other applications. Many are building their applications off-platform with the same tools. Having tools that run on z/OS is helpful. It makes it easier. Businesses can make fewer application changes. So, there’s less risk.

If a business has 20, 30 or even 40 years’ of legacy applications running on z/OS, those applications have proven to be secure, stable and reliable. An open source application development tool for z/OS offers a way to retain those qualities and modernize application development for building pipelines and instituting DevOps principles and practices that are well-known to the new generation of developers.

June 23, 2021
Out of the Silo: Big Data and DevOps Meet in the Middle

Given the myriad benefits they provide, it seems crazy that big data and DevOps have been implemented independent of each other

If you have been working from home during the recent lockdowns, you probably don’t feel like you are making many new connections, but when we come to look back on this time in a few years, it might prove to have been one of the most effective networking sessions of the past decade.

Nowhere is this more true than in the twin worlds of big data and DevOps. It seems that being forced to work from home, far from locking engineers in both disciplines deeper within their silos, has brought them out of their shells. In recent months, we’ve seen an outpouring of speculation on how DevOps and big data can work together, and even a number of impassioned manifestos on why it is time to bring DevOps ideas to big data.

In this article, we’ll look at why this crossover is suddenly so popular, how these two disciplines can engage in a symbiotic relationship with each other, and how they are already doing so.

The Pandemic and the Silo

For consumers, the experience of lockdown has accelerated internet usage trends. While some of the changes that have recently swept the online world were easy to guess, the speed at which they have taken hold has been astounding. It’s therefore no exaggeration to say, as the New York Times put it recently, that the virus has changed the way we use the internet.

These consumer trends are driving changes throughout the software development industry, forcing firms to adapt to rapid release, testing and update cycles.

One of the biggest changes in the way that developers collect data over the past decade has been the source of the data. This year, and largely due to the COVID-19 pandemic, we passed a notable watershed: Mobile devices and smartphones now account for more than half of all internet traffic, resulting in an increased data flow that boggles the mind and creates issues of its own. This increased data load from consumer mobile devices, rather than from corporate data acquisition systems, is forcing teams to work more closely together.

In the words of Bill Detwiler, editor in chief of TechRepublic, in a recent CBS interview: “The COVID-19 pandemic has accelerated the blending of data analytics and DevOps, meaning developers, data scientists, and product managers will need to work more closely together than ever before.” For most firms, the way in which these consumer trends are fed into the development life cycle is via analytics teams. This has meant that, during the pandemic, analytics teams were generating actionable insights at an increased rate, and this has been the driving force behind the newly intimate relationship between big data and DevOps staff.

Waterfall vs. DevOps

To understand the way in which big data and DevOps teams can best work together, it’s instructive to visualize the way in which developers, operational staff and data analytics agents work.

A decade ago, most software developers were using some form of waterfall development methodology. As the name suggests, this approach was essentially linear—the specifications for software were set in advance, and after this point multiple teams worked in tandem to deliver a continuous “stream” of software updates.

The DevOps framework changed that by encouraging communication and collaboration between operations and development staff. The idea, in short, was to ensure that software underwent continual testing—not just in terms of usability and security, but also against the requirements of the operations staff who were charged with maintaining it.

Now, we might be facing a shift of similar scale. Many firms have now adopted big data frameworks, but these systems generally have been seen as a way of harvesting and collating market insights. This was always a waste of potential for big data because as long as it was regarded as part of marketing, it’s utility in other parts of an organization were difficult to discern.

Not any longer, it seems. The changes caused by the pandemic—both in consumer behavior and in the way that organizations communicate internally—have given developers an appreciation of the value of the data that analytics teams hold.

Pioneers Show the Way Forward

It’s also worth noting that big data and DevOps teams working in tandem is not a utopian vision; rather, this way of working was already being explored within leading companies even before the pandemic.

In retrospect, in fact, it seems crazy that these two teams ever worked independently. If you’re a transportation company, the ability to track your loads on the road and the health and safety of the cargo that they’re carrying has become mission-critical. If you’re in the armed forces and using drones on the battlefield to conduct and report reconnaissance in real-time flyovers, the ability to adapt software parameters on-the-fly can be the difference between success and failure.

The military is, in fact, the place to look for examples of how big data and DevOps will shortly become one process. Big data in the military has long been more advanced than that in the civilian sector and is beginning to inform everything from IT predictive analytics to real-time usage analytics.

The Future of Big Data and DevOps

There remain, of course, significant challenges to be overcome before the promise of DevOps working with big data teams can be realized. Perhaps the most thorny of these is the fact that computing performance is crucial for big data in a way it hasn’t been for development: The latter teams may need to build capabilities to make use of the insights generated by big data teams.

Still, communication can only be a good thing, and if the pandemic has forced some of us to emerge from our bunkers and start to work together, then at least one good thing will have come from it.

December 4, 2020
Safeguarding Secrets Within the Pipeline

How can an organization write source code to use secrets without directly referencing them and risking them being leaked to the public?

2020 closes out a decade full of countless code leaks, a topic we talk about quite often. In fact, just recently, Intel became the latest victim after 20GB worth of internal documents were leaked and uploaded to a public sharing site.

Intel finds itself in good company as it joins a long list of 2020 victims whose confidential source code has involuntarily gone public.

Many, if not most, of these victims share one common fatal flaw: They hard-coded secrets (logins, tokens, etc.), then published their source code to a trusted cloud computing service platform such as an online Git repository.

It would be naive to lay blame solely on the platform, however. It has long been a reliable rule of thumb that anything you put online—gated or not—is always at some level of risk. With this in mind, one should never, under any circumstances, include secrets in their source code. Stealing source code is damaging enough; the keys to your kingdom should not be included.

That’s easier said than done, of course. How do you write source code to use secrets without directly referencing them? In this blog, we’ll show you an overview of how you can utilize your continuous integration and delivery tools to inject secrets into your code at build/runtime without actually including them in the code itself, thus placing a veil between your code and your secrets.

Injecting Secrets

One method for publishing source code reliant on secrets without actually including the secrets in the code is to use placeholders within the source code which, at build time, are replaced with the actual values needed.

Microsoft Partner Technology Strategist Primoz Kocevar details such a method in his Medium post. The general idea is that secret variables are provided to a Docker image so that no secrets are found within the source code itself.

An environment variable, set up beforehand, stores the secret. This variable is defined in a Docker file, which is used to create an image. During the build phase, secrets are then passed via the Docker image through the variables specified in the source code. Once complete, the application and image can be deployed.

If you happen to run your images on Kubernetes, you can also leverage its own feature for securely storing secrets. This method allows you some additional options, such as using a Kubernetes environment variable or ConfigMap within a Kubernetes pod to store secrets for injection at build time.

By the way, this approach to secret injection is by no means exclusive to Docker or Kubernetes. Though we highlight them here as specific examples, the same method can be employed via virtually any similar solution you rely on for deploying pods or containers.

Injecting secrets via clever use of environment variables and configuration files in your microservices platform of choice is certainly a viable option, but we don’t think it’s the best option. Consider that this approach requires one image per each secret. This can get cumbersome fast if you have a development environment with one secret and a production environment with another secret, each requiring different images.

Leveraging Your CI/CD Tools

First, a quick primer for the uninitiated: Continuous integration (CI) and continuous delivery (CD) tools and practices are an essential part of the larger software supply chain. Continuous processes are those that enable developers to publish code changes easily and frequently without degrading the product’s reliability.

Continuous integration tools and practices establish uniform processes for creating, testing and building source code, integrating with any required external systems along the way. Continuous delivery covers the activities that follow, streamlining and automating the delivery of source code to infrastructure, especially when multiple multiple environments are involved.

Many CI/CD tools provide their own built-in solutions for securing sensitive data. You can leverage these features when storing secrets until it’s time to inject them during the build.

For example, Jenkins provides its own credentials store and its own automatically encrypted Secret field type. (A word of caution: This alone doesn’t prevent a user from accidentally printing a secret in the console log.)

Another CI tool, CircleCI, offers a number of options for protecting secrets, such as secret masking to prevent secrets from appearing in terminal output and contexts, which limit access to certain variables to only specific user groups.

There are, of course, as many additional options as there are tools available.

We can leverage such features to enhance our previous example: secrets stored in your CI/CD tool can then be injected into our Docker run command (or your platform’s equivalent command) during deployment. In other words, your CI/CD tool of choice injects secrets into the image, which in turn injects the secrets into the corresponding source code’s variables. This method ensures our secrets never appear in the actual source code.

A word of caution: Even utilizing this approach, your secrets are still only as safe as the tool you use to secure them. The Bleeping Computer article we linked earlier in this post points out how weakly configured tools themselves pose a threat: “The leaks have been collected by Tillie Kottmann, a developer and reverse engineer, from various sources and from their own hunting for misconfigured DevOps tools that offer access to source code” (emphasis ours).

It’s important to stress the universal rule that there is no such thing as perfect security. Consider that encrypting your secrets within one of these tools offers little protection if access to those tools is not sufficiently restrictive. We recommend cautiously limiting who is granted what level of user access to each tool, applying the least privilege principle.

Protection Post-Deployment

Now that we have built and deployed our secret-free code, what’s next?

If we were to stop here, we would have overlooked the fact that any developer who has access to our containers can see our secrets. The downside to utilizing a microservice infrastructure is that there are many opportunities for security gaps to appear between all the moving parts, from the moment the first line of code is written to when it all goes live, which can easily go unnoticed.

Containers and pods need to communicate with each other and other endpoints, thus an attacker can leverage this by compromising one container or pod then moving laterally to others, rendering any secrets stored within ripe for the taking. The bigger your distributed infrastructure, the more difficult this is to prevent, because of the inevitable complexity of security policies required to do so.

Thankfully, many tools offer options such as built-in network/firewall policies between containers/pods and endpoints, an option to encrypt data at rest or a mechanism for enforcing the least privilege principle, such as via restricted security groups.

Some tools may also allow you to configure a Key Management Service (KMS) provider to encrypt your secrets. A KMS provider employs an envelope encryption scheme for data stored within etcd. The added benefit to this approach is that the decryption key is stored remotely on the KMS as opposed to on the same host.

Conclusion

Regardless of what tools you have in play along your development pipeline, secrets must always exist outside of the source code and should be (securely) injected into the code as it passes through the pipeline.

Wherever secrets ultimately live, be it in a remote storage location, a container or pod or within a CI/CD tool, they must always be encrypted and never stored in plaintext.

Finally, all the obfuscation and encryption in the world will not protect secrets from a lackadaisical user access policy. The least privilege principle should always be in play and user security groups should be strictly managed to control who has access to the systems responsible for securing your secrets.

All that said, mistakes still happen, and none of the tools discussed here perfectly ensure that no developer will nonetheless accidentally hard-code secrets into your source code. It is with that in mind that we advise adding another layer of security by automatically monitoring your source code repositories for secrets.

September 15, 2020
7 Forces Driving the Low-Code Movement
Why low-code? Why now? Here are some of the driving forces behind the burgeoning low-code movement

If you’re keeping an eye on the tech tooling landscape in 2020, you may have noticed the buzz around low-code. Countless low-code platforms are being funded and coming to the market. In fact, Forrester estimates that spending in the low-code category will exceed $20 billion in 2022.

But, arguably, visual programming models are nothing new; FileMaker is 35 years old, for crying out loud. WYSIWYG web editors are just as old. Providers including Outsystems and Appian have similarly supported visual code generators for decades. So, what’s with all the low-code hype? Why low-code, and why now?

Well, a new generation of low-code is emerging, spurred by some unique forces. I recently talked with Peter Nelson, vice president of Claris, an Apple subsidiary, and maintainer of FileMaker, on the current market factors driving advances in the visual programming space.

Low-Code, High Interest

Below, we brainstorm top forces stimulating the low-code trend, from a pressure to automate internal processes to a lack of developer talent and COVID–19. This list is not exhaustible, but hopefully offers a window into why the low-code movement is gathering steam.

Rapid Development

There is a pressing need to automate internal processes to maintain business agility.

Release velocity is quickening. Across the board, all companies face the need to automate their internal operations. Companies require workflow automation for form generation, data retrieval and database-driven functional apps, among other things.

The low-code industry had become a bit stale up until about five years ago when things started to change rapidly, Nelson said. “Recently, we’ve seen an explosion in the low-code no-code space.”

The emergence of next-gen low-code tooling runs parallel with the emphasis on rapid, continuous-release strategies. New UX layers that appeal to citizen developers empower more actors within an organization to embrace DevOps and enable quicker deployments.

Lack of Developer Talent

Businesses lack programmers with proper skill sets.

Code.org tracks 666,534 open computing jobs nationwide, yet only 71,226 computer science students graduated into the workforce last year. That means roughly 89% of open positions cannot be filled with the talent on hand.

IT faces a labor shortage. Drag-and-drop visual models enable problem solvers to quickly string together a workflow, avoiding tracking down the right IT department or PHP developer.

Nelson feels low-code can rectify the labor gap and act as more than just a temporary bandage. “You aren’t sacrificing long-term viability,” he said. “These aren’t temporary solutions that you need to grow out of.”

Cloud Integrations

Nuances of cloud integration workflows are challenging to maintain.

Think of all the cloud-based business utilities an average user consumes: Slack, DocuSign, Dropbox, Google Drive, Salesforce, etc. If a problem solver is automating a unique internal process, they’re likely creating an event-driven workflow between these apps. This means cloud integrations have become very important.

Web APIs do offer a standard communication format between cloud-based microservices. However, each connector has its nuances, and manually programming actions between them can be time-consuming. Not to mention the implications of supporting third-party dependencies such as authentication time-outs, versioning breaking change and uptime monitoring.

“Integration within no-code did not exist 10 years ago,” Nelson said. Fast forward to 2020, and ProgrammableWeb cites 23,000 public APIs. Low-code platforms have emerged as a glue between properties, solving the integration hurdles.

“You can build anything with the right legos,” he noted. Similarly, workflow automation requires stringing together the right components. “You don’t need to be a developer to do this.”

Hybrid Modes

Hybrid styles mean platform-agnostic development layers become more important.

Hybrid cloud, multi-cloud, multi-device. The world is accessing data from multiple sources and storing it in multiple places. Thus, business applications must be flexible in a platform-agnostic world.

A 2012 Cisco Systems study found 95% of organizations allow employee-owned devices. In 2016, Systonic found 87% of businesses rely on workers to access their smartphones. Now in 2020, a fully remote economy clocks in to work on their personal devices from home.

In this remote paradigm, web-based collaboration is an absolute necessity. Users should encounter the same digital experience regardless of the device used or where they’re located. However, even though users are remote, not all servers are remote.

“We’re slowly shifting toward cloud-native,” Nelson said. “But on-premise[s] still takes the lion share.” With concern looming over container-based cloud deployments, some high-security institutions could remain on-premises for some time.

Thus, business workflows require platform-agnostic capabilities that can talk with both on-premises and cloud-native servers. Low-code platforms are acting as an agnostic layer, catering to these hybrid modalities.

Artificial Intelligence

Low-code democratizes access to machine learning and AI.

“Today, you can learn how to stand up a database and interact with it from a YouTube video,” Nelson noted. “However, you can’t integrate AI, or scale it easily. The problem statement has changed.”

Low-code platforms abstract the back-end complexity of AI, enabling object recognition, language analysis, predictive analysis, risk scoring and other abilities. Some even can train new machine learning (ML) models.

Low-code enables companies to leverage powerful AI without much investment. For example, Outside Insight recently reported how low-code platforms are democratizing AI, while consulting firm Delloite assembled TrueVoice, a voice analytics platform built on a low-code platform using behavioral and emotional AI.

“If you can attach machine learning to the automation of business processes, I think you change multiple industries,” he said.

COVID–19

COVID–19 amplifies the pace of digital transformation.

Digital transformation has always been there. But now, you can’t ignore it. “COVID isn’t actually changing anything, it’s accelerating things,” said Nelson.

In the wake of COVID–19, companies may wholly depend on virtual business. For example, Take Allstate Insurance, which now receives 90% of claims using virtual tools, up from 11% in pre-COVID times. Low-code helps these brick-and-mortar companies quickly onboard to new technologies.

To supplement traditional business models, some industries will take digital transformation even further, becoming legitimate tech companies with externalized SaaS. For example, take speaker designer Dolby’s transition to online media transformation as a service.

Reusability

Next-gen low-code emphasizes reusability and externalization more than predecessors.

Anyone who follows the Bezos mandate of externalization from day one knows how valuable internal infrastructure can be. The tools you build should always be reusable by anyone on the network.

So, if you’re introducing an abstraction layer to build new software, you don’t want to shoot yourself in the foot. Low-code has a potential downside of inhibiting code ownership and stunting the ability to externalize valuable infrastructure.

Whereas older low-code platforms may have caused these concerns, Nelson views modern platforms as more malleable. He noted that with low-code widgets and data accessible via APIs, “internal processes, management work, by definition, are exposable.”

A Low-Code Revitalization

Low-code, no-code, some-code, workflow automation; call it what you will. Citizen developer platforms have taken on much interest (not to mention much funding) in recent years.

The need for increased accessibility to advanced workflow automation seems to underpin the low-code movement. But what else is powering recent transformations? To review:
- Rapid automation: Companies need to create frictionless business processes easily and rapidly, and change them on the fly.
- Developer scarcity: The workforce lacks qualified candidates. Low-code fills this gap in the market.
- Integrations hurdles: Internal apps involve many cloud integrations. Low-code can support these connections, enabling users to link event-driven workflows visually.
- Hybrid requirements: Users demand ubiquitous network access no matter device or cloud type. Low-code can act as an agnostic layer.
- Democratization of AI: AI empowers analytics and business intelligence projects, yet machine learning is difficult to implement. Low-code is abstracting this complexity.
- Reuse and externalize: Newer low-code platforms are better designed to expose data and services.
All this adds to a confluence of forces, giving rise to a low-code tidal wave. “This is the time where everything is coming together and is possible,” said Nelson.

What do you see as spurring our current generation of low-code?
August 27, 2020