devopsy.kinsta.cloud

Tag: engineering

Managing Engineering Teams During Economic Downturns

The tech industry has faced strong economic headwinds over the past year or so. We’ve seen thousands of layoffs at public tech companies, along with weak stock performances and a decline in venture capital funding. Although some analysts see light at the end of the tunnel, the current slump doesn’t seem like it’s quite over yet. As such, some IT leaders are learning to cope with a slashed budget and a leaner engineering team.

Simultaneously, engineers commonly desire more flexibility from their employers, both in terms of remote working opportunities and flexible hours. In fact, only 15% of developers are working 100% on-site, according to The State of Tech Hiring 2023 survey, which surveyed 14,000 professionals. In addition to being open to flexible working conditions, some organizations are examining how their internal tools could be improved to increase developer experience and productivity. Amid a downturn, leaders might need to consider these new strategies and many others to maintain steady progress.

I recently met with Amanda Richardson, CEO of CoderPad, to discuss tips to lead your engineering team during this downturn. According to Richardson, finding the right blend of contractors, remote work opportunities, and helpful tooling is necessary amid the current climate. Leaders can also adopt many creative methods to encourage productivity and motivate developers. Below, we’ll overview four ways technical leaders can build and sustain their teams.

1. Consider Hiring Contractors Vs. Full-Time Developers

Recruiters often struggle to fill highly technical positions like full-stack developers, backend developers or DevOps professionals. If a company can’t fill full-time roles, contractors are another valid option. According to Richardson, contractors provide companies with more flexibility because they are easier to hire and cut. These mercenaries could swoop in to pick up the slack by solving common tedious problems, like Salesforce integrations or less-sexy administrative work, she said.

Of course, contractors don’t solve every need, and they’re not bought into the company’s vision. That being said, more and more companies are leaning toward contractors to help fill technical gaps. In fact, 63% of recruiters now say their company hires contingent workers for tech needs, up from 42% last year, the report found.

2. Target Remote Work

Work-life balance has become a top requirement among today’s engineering workforce. For example, remote working options rank as the third-highest priority when looking for work, right behind salary and flexible working hours. Many engineers work better without the interruptions and noise of a traditional office space, explained Richardson.

Yet, the desire to work remotely is at odds with the policies of some large tech companies, who are attempting to push developers back into the office. Interestingly, this may present an opportunity for small-to-medium enterprises to embrace flexibility and get a leg up on the competition. “Give the people what they want,” said Richardson. “If you want to retain talent, remote work is what you must do.”

3. Use Tools to Fill in the Gaps

Another method for filling technical gaps amid a downturn is improving efficiency with new tools. For example, platform engineering is a strategy that aims to develop reusable, self-service platforms to manage internal infrastructure. Centralizing common automation and builds could bring more consistency to deployments and reduce grunt work, freeing up engineers and making them more productive.

Of course, we’ve also seen a lot of interest in new generative AI technologies, as well as development automation using low-code/no-code platforms. According to Richardson, leveraging AI and new automations presents a newfound opportunity to empower the workforce. For example, ChatGPT can solve interesting challenges in the prototyping phase, like generating a mock database or co-authoring support materials. Technical experimentation is a critical aspect of any team, and new AI capabilities could help bolster and support the operational sides.

4. Retain a Strong Culture

Lastly, maintaining a vibrant culture is another strategy to help retain talent. Richardson suggested internal hackathons and coding challenges to bring teammates together in an organic way. An internal hackathon could also be organized to rethink internal processes, providing an outlet for engineers to suggest innovative solutions. There’s also the basic stuff, like daily standups or brown bag lunches, that can create a space to share ideas. “It takes a good leader to figure out what’s going to engage their team,” said Richardson. (One business owner I interviewed actually condoned cooperative gaming during work hours to build community and has seen moderate success with this approach).

Final Thoughts

Of course, the above suggestions are easier said than done, and it will take a leader with a keen understanding of their unique engineering team’s attitudes and requirements to find their own version of success. On that note, talent comes in many shapes and sizes and traditional education isn’t everything when assembling a development team. In fact, a full 59% of developers do not have a university degree in computer science, the CoderPad report found, indicating that leaders may have to be more flexible when it comes to recruiting.

In general, following the above tips could be helpful when managing technical talent. Enabling the company to hire contractors and remote workers increases your potential talent pool, and investing in new tools and company culture could help retain workers. Considering half of today’s developers are considering a job change in the next year, leaders should act now to avoid high turnover before it negatively impacts their team’s output and morale.

April 3, 2023
How Developer Productivity Engineering (DPE) Enhances Software Delivery
Developer experience (DX) is in the zeitgeist, and for a good reason. Digital demands are increasing, yet the market faces cautious IT budgeting, workplace burnout, and high resignations. The idea is that quality developer experience can help keep the remaining talent happy and increase productivity. And organizations are considering many strategies for increasing DX, such as creating more usable interface interfaces, maintaining intuitive documentation and abstracting complexity with more automation and low-code.

Another way to increase productivity is by slimming down the time it takes to test and deliver software. There are many potential hurdles in the way of deploying new code, and a cumbersome pipeline can easily take a programmer out of the flow. The practice of streamlining this toolchain is known by some as Developer Productivity Engineering (DPE). DPE focuses on improving developer productivity through accelerating DevOps processes, like CI/CD pipelines.

I recently met with Hans Dockter, founder and CEO of Gradle, to understand the core tenants behind DPE. According to Dockter, the dialogue with the toolchain is a developer’s bread and butter, and optimizing this exchange is vital for quickly validating new ideas. Below, we’ll seek to understand what DPE is and how it might increase the pace of software delivery. We’ll also consider some methods for implementing DPE in practice.

Roadblocks in the Development Toolchain

Engineers are always experimenting with new feature development. “A developer needs to have a dialogue with their toolchain to see if their hypothesis makes sense,” explained Dockter. Yet, there are many potential barriers to instant feedback. For example, a toolchain may be comprised of code compilation, static code analysis, unit testing, performance testing, security testing and other checks. Running new changes through a CI/CD pipeline can take anywhere from a few seconds to minutes or hours. In his work, Dockter has even encountered teams that deal with a 24-hour feedback cycle time.

Long wait times can be taxing as they demand users to switch contexts. Neuroscientists say that multitasking depletes glucose in the brain and leads to exhaustion. Similarly, Dockter argued that long waiting periods with the toolchain can take developers out of the flow and slow down the pace of software delivery. “You cannot have a good developer experience without good productivity.”

Furthermore, he added that engineers often encounter untrustworthy signals from their toolchains. If testing suites produce too many false positives, for example, you can lose faith in the tooling altogether. This equates to a bad experience and may result in developers overlooking important signals that do matter.

Understanding Developer Productivity Engineering (DPE)

DPE is all about improving developer efficiency. And quickening the feedback cycle is one method to enhance developer productivity. According to Dockter, DPE applies a few strategies to streamline this feedback loop and shorten the timeframe:
- Optimizing which tests are run: First, Dockter recommended only running tests relevant to the change at hand. Optimizing test selection can reduce bulky builds and significantly reduce the time it takes.
- Caching: Caching is integral to any web application, so why not apply similar heuristics for the development pipeline? This could involve caching code that is not changed. “Changes are often incremental from one feedback cycle to the next. A lot of output is produced that can be reused,” said Dockter.
- Predictive test selection: To automate test optimizations, Dockter recommended predictive test selection to understand which tests are sensitive to which areas of code. This would utilize an AI trained on release mechanics to help pinpoint what components the changes modify and, thus, what tests to run.
- Test distribution: Another way to accelerate the pipeline is by distributing testing processing. Using multiple machines to test different areas simultaneously could greatly speed up the routine. Test distribution “is a modern approach to test parallelism that accelerates test execution by taking existing test suites and distributing them across remote agents to execute them faster,” wrote Marc Philipp and Hans Dockter in The Developer Productivity Engineering Handbook.
- Monitor the toolchain: Lastly, it’s good to continuously monitor failure analytics to discover where the toolchain itself is causing trouble, said Dockter. On the side, CI/CD pipelines are not immune to security vulnerabilities and should be treated with the same security standards as other infrastructure.
DPE principles have already been applied at DoorDash, LinkedIn, Netflix and other organizations.

Benefits of Developer Productivity Engineering

In the last few years, DX has moved from niche to mainstream, said Dockter. In fact, DX has even entered the C-suite, as certain companies have embraced the role of chief developer experience officer CDX). And with the emerging discipline of platform engineering, we’re seeing an even greater focus on streamlining internal DevOps workflows.

And although platform roles may have been viewed as second-class engineers in the past, said Dockter, they are held in high esteem nowadays. “It’s as important as the other engineering work we’re doing,” he said. Developer productivity engineering seems to align with these principles, helping organizations manage the effects of an exponentially expanding code base. In the end, an investment in DPE, like platform engineering, could bring more satisfactory working conditions and accelerate the pace of software development.

“DX is a transformative opportunity, but most companies invest incrementally. The economic benefits are transformative for companies developing infrastructure to get more output from their developers.”
March 30, 2023
How to Supercharge Your Engineering Teams

With the economy still bleak, data and engineering teams that were already struggling to keep pace with business needs are now facing a potential hiring freeze or worse in 2023. That’s why there’s never been a more important time to maximize the productivity and impact of your teams, but the question is how to identify the big changes that move the needle in performance. Incremental adjustments that save a sliver of time here and there are not hard to come by, but what teams need are the breakthroughs that yield giant leaps in productivity and efficiency. Rather than a 10% improvement, it’s the 10x improvements that drive lasting gains.

As an engineer and CEO of a software company, I’m constantly looking for ways to boost the productivity of our teams. Our guiding principle is simple: Maximize the impact per hour invested for every individual. To do that, we must ensure that individuals maximize the time spent flexing their superpowers, minimize busy work and reduce dependencies on others that delay work or tie them up in meetings. While counterintuitive at first, we consistently find that collections of well-aligned, independently executing individuals produce a far greater impact than a large team swarming on a project. Put another way; the individual impact goes up when the number of people you depend on to deliver the result goes down.

I’ve developed an approach over the years that has allowed me to identify several opportunities for step-change advancements in productivity and efficiency, and I’d like to share it here, along with two concrete examples. I will also share an exercise we run with our teams to help identify and execute these opportunities.

First, here are my two main criteria for identifying the projects and tasks where we can effect changes that will have an outsized impact on the company:

● Big resource-consumers: Work that’s essential to the business but consumes a large amount of engineering and other resources is a top candidate for optimization.
● Big impact, easy win: Areas that will have a significant impact on company performance if we can do them faster and with higher frequency are also great candidates—even if they’re not consuming a ton of resources.

Here are two examples that illustrate each of those criteria and where we were able to see massive gains in productivity, quality of service, or both.

Tackling Big Resource-Consumers

At Ascend.io, we provide a platform for building and automating data pipelines, which requires us to build connectors into a huge variety of data sources. At one time, building a new connector required multiple engineers to spend two to three weeks writing more than 5,000 lines of code that spanned four microservices in three different languages. As we added more customers and needed to support more data sources, this quickly became unsustainable.

When we looked at this process, we saw two issues: Engineers were writing custom code for each connector in a way that would be better served by a common framework. Moreover, the process required engineers from multiple teams and disciplines to work together and coordinate their priorities. As soon as you get dependencies that require engineers to coordinate with others and wait on the outcomes of other people’s work, you’re wasting time.

We addressed this by designing a new development framework that exploits commonalities across the connectors we build. This now requires far less unique code and removes the need for multiple engineers to work together. In most cases, one person can build a new connector in days with 100 lines of code. This made us far more nimble and freed up engineering time to do other valuable work.

Achieving High Impact

Like most companies building software, we need to convey the capabilities of our product to new and existing users to ensure they receive maximum benefit from their use of Ascend. With new features and enhancements being released on a weekly basis, there is a lot to keep up with. This is high-impact work, however, and if we can do it in a vastly better way, it drives an outsized impact for Ascend and our customers.

For us, building these automated product tours typically required a product marketer or product manager to design the customer journey, and one or more engineers on a sustained basis to build the actions into the product. This again creates dependencies that slow progress and require significant manual coding.

The breakthrough for us was a tool called Intercom, which allows you to build sophisticated product tours without needing to write code. Our product team can now build more interesting and complex product tours for customers, and our engineers can stay focused on building new features instead of the code needed to showcase them. What’s more, they are able to design, test and deploy these tours independent of our product and code deployment cycle, further freeing them to iterate as fast as possible.

A Thought Experiment

So how do you actually find these opportunities for 10x improvement? I run the following thought experiment with our teams several times a year to uncover these opportunities:

Imagine your workload scales 10x overnight and you absolutely must meet this increased need, but without hiring more engineers or working more hours. How do you do it?

It’s amazing what this exercise can uncover—but there are ground rules and it can take time to see results. Coming up with these big ideas is a skill like any other, and most people are not great at it the first time they try it. Your first iteration may produce no options to pursue—some may have high-impact ideas but with even higher investment requirements, while others may reject the premise outright. That’s normal. So, don’t insist they sit in a room until they dream up that 10x breakthrough, because it will be demoralizing. Instead, plant the seeds by establishing the above framework and then let the ideas start to grow.

More likely, a few days later, they’ll be out for a hike or chatting with an old co-worker and they’ll suddenly have the germ of an idea. That may still not be the winning solution, but someone else will build on it, and then someone else and, finally, you iterate your way to a breakthrough. Pulling the team back together and embracing the “yes, and” approach can be both energizing and highly productive.

Brainstorming like this is fast and free, and you only need one great idea to have a material impact. We run this exercise a few times a year and have come up with several big ideas that were game-changers for the company. Besides the examples above, we also moved key parts of our software development from Scala and Go to Python a few years ago, which is a much more efficient language (as measured by developer time) and allows our engineers to build things much faster.

Software engineering is a complex beast that often spans many teams working on multiple processes in parallel. But that complexity is also your opportunity: Every complex system has room for optimization, especially at companies that moved fast in the beginning to build a product and find market fit. Now is the time to lift your gaze, look around and ask where significant gains can be made. They almost certainly exist, and a 10x improvement could be what allows you to not only survive the turbulent year ahead but emerge with a thriving team and business on the other side.

March 27, 2023
Five Great DevOps Job Opportunities

devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

LinkedIn

M-Logic
Dallas, Texas/Chicago, Illinois
Security Engineering – DevOps
$150,000 to $160,000

SimplyHired

Cisco Systems
Waltham, Massachusetts
DevOps Engineer
$134,200 to $197,700

Indeed.com

AT&T
McLean, Virginia
DevOps/Python Specialist
$111,000 to $140,000

Monster.com

Capgemini FS
Charlotte, North Carolina
AEM Full Stack Developer
$110,000 to $130,000

Dice.com

Booz Allen Hamilton
McLean, Virginia
Lead DevOps Engineer
$99,300 to $212,000

March 27, 2023
Five Great DevOps Job Opportunities

devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

Monster.com

Citibank
Tampa, Florida
Application Development Technology Lead Analyst
$154,000 to $179,000

Dice.com

ZealTech
San Jose, California
DevOps Architect
$150,000 to $200,000

SimplyHired

Musicnotes
Remote
DevOps Engineer
$85,000 to $100,000

Indeed.com

Snowrelic
Remote
DevOps Engineer
$82,486 to $173,694

LinkedIn

Oak Ridge National Laboratories
Oak Ridge, Tennessee
DevOps Engineer
$78,000 to $137,000

March 13, 2023
The Database of the Future: Seven Key Principles

The database of the future will be more than just a database. Instead, it will be a platform that focuses on the user experience and abstracts the technology. Looking at the evolution of application development, an enormous amount of effort has made it easier to build and deploy applications. Similarly, databases need to evolve for ease of use with a delightful developer experience and an exemplary user interface that surfaces features and functionalities that engineers will actually use.

Here are seven principal features needed to chart the development of the future database.

1. Serverless by Default

Serverless is unquestionably the most significant paradigm shift in software engineering in the last five years and the one that’s been realized the least. There aren’t that many people at this point that are all in on serverless. It’s the shift away from buying a server to a service that does what you need and scales with your growth.

Serverless by default is the only way to ensure the database of the future will be easy to operate while being consistently reliable. Serverless is evolving from running functions in the cloud to becoming the standard for fast-paced development teams that don’t want to worry about vCPUs or node pools but focus on outcomes.

The companies that are doing it the best still deploy servers, but they are leveraging products that scale independently, regardless of how many servers they have. Their component vendors can increase the number of resources allotted to them, and they only pay for what they use, and the architecture of the solution scales in that way. They’re not buying a single server and pushing it as hard as they can. They’re telling one platform, ‘here’s my code,’ they’re telling another, ‘here’s my data,’ and then that scales up based on usage much, much farther than buying a single server.

When people think serverless, they think JavaScript and Lambda, but there’s serverless PHP out there, too. It’s not just the JavaScript community. Other software paradigms are going for this. We will see more and more people pulling this out when they need to do some quick code and they want to ship to production as quickly as possible and scale as quickly as possible.

2. Easy to Operate

The most significant advancements in DevOps have been freeing developers from complex operations that overwhelm their day-to-day operations. Engineering teams today are organized around product and business goals, not maintenance and servicing. But databases have yet to catch up. Product teams still rely on experts to perform overly complicated database operations tasks.

The database of the future must fit into everyone’s DevOps workflows. It must be usable for any developer and leveraged by every team within a technical organization. The database needs to be operable by everyone and no longer require database admins to manage backups and capacity planning or worry about disaster recovery.

Developer-centric features like database branching, database developer environments, autoscaling and usage-based billing will be commonplace.

3. Reliable

Reliable simply means it’s harder to hit the scaling limits. Not only does it mean that the database is there, but that you get the right information when something is going wrong, and you can collaboratively help scale.

Nothing scales infinitely. There are not enough computers in the world to scale something to infinity. You’ll always be able to query a database and pull too much information out of it too fast. No matter how many cars you get or how much horsepower you have, you can’t go a million miles an hour. A good database must let you know when you’re hitting those limitations and help you move past them, whether at a small scale, medium scale or large scale.

4. Interoperable

Future databases will be accessible to every imaginable client–from smart refrigerators to fleets of autonomous delivery vehicles. But this client-agnostic approach requires planning for many more transactions and connections than most current databases could ever manage simultaneously.

In the future, databases can handle unlimited connections through smart connection pooling, query replication and intelligent query routing. Databases will also need to serve a wide set of connections and clients. Although we are already seeing many databases offering web APIs and new connection types, as these get tested at scale, we will begin to see them adopted and used to power the next generation of applications.

5. Globally Distributed, Locally Available

Today, most businesses and applications don’t need a global database, which often introduces unnecessary complexity and operating costs. However, there are many reasons the future database will need to be globally distributed, as they must be located where you need them the most.

The key to this is knowing what your requirements are. Many people look at multi-region and think it’s faster reads, failover, scaling and breaking the speed of light. Ultimately, it’s all a series of tradeoffs. For example, when you read from a local replica, you trade off consistency because you have asynchronous replication. But this is really leveraging the global internet to deliver a better outcome where it’s necessary.

Being compliance-ready with regionally stored data enables you to scale rapidly to a new region and provides a great user experience by quickly serving locally available data. High-transaction, low-latency applications such as mobile games already need this capability.

But there’s one caveat: Global data distribution cannot break principle number two—being easy to operate. Tools will need to be elegant and have straightforward interfaces as their main goal is to make operations easier, not harder.

6. Intelligent

The future database is intended to be used by people and will be able to tell them how it needs to be used and how to make it better. All the tools you buy today can only give you the data you need to make some educated guesses. The future database will replace that with concrete recommendations, then self-healing actions.

This starts with basic data collection and observability and moves to, “let’s record every query that someone makes and use some decision engines and some AI and recommend how they can shard automatically.” It will be able to say, “I’ve recorded your database use for the last 30 days; this is how you should shard for optimal performance. Click the button, and we’ll do it.”

The future database will say, “You’re scaling too far for the single machine at a per-user level. You must break it down into two or break it down into four, and we figured out how to do it for you, so you don’t have to figure that out. We’ve sharded your data on your behalf.”

7. Linearly Horizontally Scalable

Future databases will also need to be designed for future scale. Only a handful of companies on the planet have hit a scale that tests the limits of current technologies, but as information grows, more companies will get there. The next databases will do that by automatically scaling both vertically and horizontally within seconds. Many existing databases require a maintenance window, replication and a failover before they can be scaled up or out. But downtime is unacceptable, and scale should not come at the cost of performance. And it doesn’t have to. The database of the future will scale intelligently and without a blip on the status page.

In short, the consumer should be able to get what they need when they need it, with no risks. It should be easy to say, “I’ve doubled the number of servers, I’ve doubled the amount of throughput. I’ve quadrupled the number of servers and the amount of throughput.” This is what sharding provides. Good sharding happens automatically at a super high scale, and you’ll only pay for what you use as you go along. The future database will do all of this without users having to turn a knob.

In conclusion

It will take the entire database industry to agree on these principal features, adopt a user-focused approach and build future database platforms designed to make engineers more successful by delivering superior products and services to their customers. I envision a day where every company no longer worries about its infrastructure and can simply focus on its business. I want development teams to take back control over their development workflow. Using a database should be as easy as writing code.

January 25, 2023
Prioritizing Product Security With DevSecOps

Building software with strong security can no longer be an afterthought for organizations. The need for a reliable cybersecurity posture has proven vital amid the constant attacks we’re seeing across industries, all over the world. Product-specific security is top of mind for most companies, and a successful product security practice requires integrating with the ethos of the engineering team—DevSecOps. Consider a three-fold strategy, which has proven successful again and again in the industry.

It consists of first creating a culture of security, with all stakeholders understanding the priorities, strategies, and business value. Second, teams must integrate security with the software development life cycle (SDLC). It’s not enough to shift left–organizations must shift their thinking from shift left to shift everywhere. And lastly, monitoring product systems at all times is non-negotiable. In a 24/7/365 world, there must be teams to watch over security 24/7/365. Part of the engineering deliverable becomes providing data to these teams so that they can have all eyes on the products, always.

A Successful Practice Starts With the Culture

In building a strong culture, the goal is to make the entire engineering team a part of the mission. This includes putting a product security team in place to support engineering needs as well as providing continuous education with resources available for addressing any eventuality.

While every organization has its own unique needs, considerations and approaches to creating a product security team, these five roles are highly recommended:

1. Technical Product Manager (TPM): Responsible for driving security priorities in planning
2. Security Architect: Sits on the Architecture Review Board and ensures architecture and design have security processes incorporated
3. AppSec Engineer: Responsible for implementing the security development life cycle
4. Adversarial Engineer: Hammers on the attack surfaces to identify weaknesses
5. Security Champions: Engineers from each Scrum team with an interest in supporting their teams with security fundamentals; they are instrumental in scaling out the overall product security effort

Creating and executing an engaging security education program is also pivotal to winning over the hearts and minds of employees. Consider initiatives such as in-depth security boot camps, vulnerability-of-the-month presentations, capture-the-flag (CTF) contests and security consultation programs.

Integrating Security Into the SDLC

From a high level, all SDLCs consist of planning, development and release, and each phase should have its own set of security controls.

Planning: The product security TPM is responsible for working with the product management team to determine security priorities–derived from sources like risk reduction, executive priorities, customer requests, security findings and regulatory requirements. Determining what is most important is key.

Development: Much of the heavy product security lifting comes during the development phase, which includes implementing threat modeling, security baselines and automated scanning.

Threat modeling: The process of identifying potential threats in a system. Once identified, the threat modeling team can create mitigations, and results are integrated into penetration testing to verify that the threats did not make it into implementation. It’s most efficient to test models during the architecture phase before coding begins to avoid rework and Band-Aids. For example, a company may have a policy disallowing access to port 22 from the Internet. Threat modeling automation can detect violations in the design and notify the owning team asking them to close the port. Tools that support this use case include IriusRisk and ThreatModeler, as well as Microsoft Threat Modeling Tool, which is a free option but does not include automation.

Security Baselines: Teams should create secured images that are fully patched, meet security policies and have mandatory security agents preinstalled. Doing this for both OS images and containers allows developers to get a level of security for “free” and avoid ongoing security reviews. Baselines should automatically update and flow into CI/CD systems.

Automated scanning: To keep up with Agile processes, scanning should be automated and have a low number of false positives. In fact, it is better to report on fewer issues than kill credibility with a lot of noise. Continuous tuning will increase the fidelity rule sets and helpful tools include static application security testing (SAST), software composition analysis (SCA), and dynamic application security testing (DAST). Automation is critical to securing applications that use a CI/CD development process and providing security right at the code push is invaluable–developers get immediate feedback, not a ticket weeks later when they likely have forgotten important context. Integrating these tools into build pipelines is a solved problem.

For example, the below YAML demonstrates automating source scans for embedded secrets (e.g., passwords and keys).

sast:secrets-detection:
extends: .sast:secrets-detection
stage: deployment:development

Release: When teams reach the release phase, adversarial testing becomes an important component. Penetration testing can be in-house, outsourced or both. An internal red team will have a better understanding of the technology and, therefore, have a broader scope than external security partners. For example, they test across product boundaries, test processes and people and test live production systems. For a mature product, consider a bug bounty program. This will invite white hat hackers to find weaknesses in exchange for cash. But “mature products” is essential; otherwise, the number of payouts will become overwhelming.

24/7 Monitoring

As mentioned earlier, 24/7 monitoring is key, especially for any cloud-based technology. A dedicated security operations center (SOC) can help, scrutinizing for threats against the company, employees and customers. The SOC’s responsibilities are much broader than those of a product security team, and they allow product security folks to sleep well at night. Security teams should configure systems, load balancers, public cloud platforms and networks to feed data directly into the security information and event management (SIEM) system. The SOC can then respond to custom automated alerts that identify any data anomalies that are being streamed to them.

As products–and technology overall–become more advanced, the way organizations secure them must do so as well. The perfect security approach varies from business to business, but empowering IT, engineering and security teams is a beneficial first step to identifying, optimizing and integrating security for both employees and customers.

December 14, 2022
Solving the Culture Conundrum in Software Engineering

The one crucial ingredient underpinning our transition to a digital-first economy, more than any other, is software. That’s why software engineering has become one of the world’s most in-demand professions. In the UK alone, the median salary for the role has increased 13% year-over-year as businesses look to stay ahead of the competition and attract the very best IT talent. But the high salary often isn’t consistent with where software developers stand in the business; many organizations still view them as suppliers rather than as crucial players in their success. Based on the current speed of transformation, we know that demand for software engineering is only going to increase. So business and technology leaders need to urgently consider vital cultural changes to help better integrate software development teams within the wider organization. But what exactly should they be?

The Development Dilemma

Agility has quickly emerged as a competitive advantage as businesses strive to deliver on customer expectations quickly and effectively. That’s heralded the rise of DevOps, which aims to write high-quality digital solutions more efficiently and have the ability to continuously update in a secure and low-risk way.

The practice is already accelerating feedback loops in the software development of digital solutions on a wide scale, meaning new software is brought into production as fast as possible and with reduced risk. As a result, today’s successful businesses are now releasing software updates multiple times a day. The figure even runs well into the tens of thousands for tech giants like Amazon and Netflix. However, other businesses are struggling to keep up with the speed the market demands, largely due to a lack of clear communication, ways of working and integration between software developers and the wider business. It’s a dilemma that needs immediate attention if businesses are to remain at the competitive edge.

Developers are too often left to deal with huge monolithic pieces of work, which require them to write code for months on end without it being road-tested. This inevitably leads to mistakes that are found only once the updates have already been released to market creating a ripple effect throughout the organization. It’s an all-too-common scenario that demonstrates business leaders’ clear lack of clarity and understanding about the colossal value adopting a DevOps culture can deliver for the wider business, including its customers.

Take financial institutions looking to introduce a new offering, for example, such as a new mortgage product. Without a collective approach to development, getting the backend systems up and running for a product can take up to six months. But modularising the options and breaking down the functionality can shorten timelines to weeks.

Business leaders must look to create unified, cohesive and cross-functional teams that work together on projects that are broken down into simplified and localized components that can be easily integrated together without a negative impact on each other. It might seem like a small strategic change in working practices, but its impact on the speed and efficacy of software development can be enormous.

Revisiting Personalities and Processes

The role of the software engineer has changed; it is no longer about writing code in isolation without much regard for or knowledge of how it benefits the business. Developers work better when they have clarity about the direct impact their work will have on achieving business goals and on the bottom line. It’s down to business leaders to communicate these challenges and goals (in other words, understand the “why”) to help software developers understand what they’re trying to achieve. But doing so in a way that moves towards a better working culture requires a new approach to building and managing software development teams.

The first step is casting aside the negative stereotypes many have of software engineers and celebrating the intellectual and cultural diversity within their teams. Diversity of personnel brings a diversity of personalities, which is crucial to creating more inclusive cultures that accept and welcome all characters with open arms. While this may seem obvious, what is often overlooked is the impact diversity can have on stimulating and increasing innovation.

Improving the diversity of development teams helps unlock multiple ways of responding to and analyzing the challenges brought forward by business leaders. After all, each individual has an array of different soft and hard skills based on their learning, development and experience. So, encouraging individuals to bring their characters and personalities to their roles is crucial; it ensures each and every stakeholder in the development process is empowered to contribute, feels inherently valued, and is able to challenge existing assumptions, helping deliver a more robust end product.

Recognizing and harnessing this spectrum of skills also helps bridge many of the barriers to development that exist within businesses. Some software engineers, for example, might struggle to communicate technical updates in a way the wider business can understand, but tapping into another individual’s strong communication skills can help unite teams–and entire businesses-around one, holistic digital strategy.

Channeling Change to Deliver Better Outcomes

Clearly, cultural change is needed to embed software engineering at the core of modern business. But revisiting project management practices is also imperative to continually deliver software that provides value to the business, the world of scientific management and Waterfall delivery methods is no longer appropriate for the development of digital solutions. With such a heavy reliance on digital services, businesses—and, consequently, developers—need to be able to make changes with minimal effect. So, eschewing monolithic projects and encouraging cross-functional teams is essential. Not only does it reduce the need for sweeping software updates, but it negates the risk of a project failing to meet the initial brief, which could ultimately impact the entire business.

By reviewing organizational processes, and bringing the wider business and software developers together, we can create teams that deliver digital solutions effectively and quickly, so that every user, whether internal or external, reaps the benefits of a more sophisticated approach to software engineering.

November 3, 2022
Software Quality is the Heartbeat of the Best Organizations
JPMorgan recently announced it was hiring 2,000 engineers, despite the gloom in global economic markets. Is this not an odd risk for an organization to take, given the demand for (and cost of) software developers today? What’s happened?
JPMorgan’s hiring drive is no mystery. Virtually every company in the world is leveraging software, with many transforming themselves into software organizations to drive new efficiencies, widen profit margins and create new markets. Software has been disruptive, so much so that the current era has been dubbed the “third industrial revolution.” Consumers are benefiting from switching providers at the click of a few buttons and most consume services through software. Bad software experiences today mean rapid customer churn and competitive disadvantage.

Quality Should be the Heartbeat of any Organization

In spite of quality’s importance, testing is still perceived as a secondary discipline and a check-the-box exercise. This reflects practices developed for assembly lines during the original industrial revolution. A product gets assembled and testers verify its quality as it leaves the production line. Testing’s goal in this context was to identify problems before they reach end-users. This same approach has been applied widely to software development and is still used by “waterfall” organizations.

Yet, software development has been revolutionized by new methodologies and practices. Software today is constructed at incredible speed. Notice I’m using the word “constructed” and not “built”—the best developers leverage libraries or APIs to assemble software using existing components. They then add a few pieces of custom business logic to define the intellectual property of an application. Many organizations don’t even code software, instead customizing off-the-shelf software with vast configurations to modify functionality.

The assembly-line approach to software development is no longer appropriate. When testing is left late, defects are found before a release or even in production. Release cycles are longer and products poorer quality. Yet, many organizations still stick religiously to this approach, with testing teams operating in distinct silos.

The Role of the Tester is Evolving

The best organizations in the world are changing how software is built. Testing is no longer a check-the-box exercise at the end of the assembly line. It is transforming into a discipline that revolves around evangelizing and driving quality throughout an organization.

Quality should be baked in as early as possible, touching every artifact produced. Ideally, this should occur from the start, as a requirement is formulated, all the way through to the software being shipped. The idea of siloed teams working on different pieces is outdated. For quality to thrive, teams must collaborate.

Modeling offers one approach to driving quality and collaboration throughout an organization. Its journey starts with a requirement. These are typically formulated by business analysts or product owners. Having been a product owner, I found that this usually involves lots of scribbles on paper before communicating to stakeholders on a whiteboard. The preliminary requirement then gets reviewed and refined.

At some point, the requirement will be converted into walls of text in a project management tool. Modeling intervenes at this stage, adding visualizations that make requirements more consumable, less ambiguous and easier to understand.

The word “modeling” may seem a scary concept, but it need be no more than a flowchart that encapsulates a requirement. The benefit of modeling is not only that it provides a digital collaboration asset; models can also generate coverage-focused tests before a line of code is written. Modeling creates requirements that are quality-driven and quality-driving.

The Journey of Quality-Driven Requirements
- Models Provide Data Upfront
Developers today require rich data to develop, while testing is only as good as the available test data. Yet, many organizations still feed slices of production data into developers and test cases.
This approach is problematic for several reasons. First, it often risks legislative non-compliance. Second, production data features large volumes of low-variant data. Data may not then be available when developing unusual-but-important scenarios. Imagine an overdrawn account with several beneficiaries: What should happen when a transfer is attempted?

Modeling defines data explicitly in requirements. It allows engineers to understand a requirement explicitly, including the relevant data. From the modeled definitions, correct data can then be masked and subset, or generated synthetically. Data can further be delivered on-demand and at scale using database virtualization.
- Models Provide Understanding
Once the requirement and necessary data are available, code can be developed. Using text-based requirements, misinterpretations often lead to incorrect functionality. Visual models instead align closely with business logic, and we often see developers enriching models with additional context so that models align with the actual software implementation.
- Models Generate In-Sprint Tests
Once the code is developed, the same tests which were created earlier from the requirements can be executed against new functionality. This facilitates reuse and means the test cases are ready to be executed immediately.

Some tests might further be selected for automation. In this instance, automation can be overlaid onto the requirements model, reusing code to generate scripts. These scripts then become part of the standard regression pack as new features are added to the product.

A Change Request Comes In

With iterative development, change requests are common. This typically creates havoc for quality, since it is difficult to understand the impact of a change, and time-consuming to update artifacts like tests and data.

Traceability is key to overcoming this challenge. Models can connect to external DevOps tools to provide end-to-end traceability, from a requirement, all the way to code and tests. The impact of a change can then be identified directly. Models can be updated, automatically synchronizing test data, test cases, and automation to bring everything up to date.

Testing for the Fourth Industrial Revolution

Software is playing a vital role in virtually every organization, as already recognized by organizations thriving today. Baking quality in early is the only way to ensure quality software. The role of testing must change on a large scale. Models offer a way to achieve a “shift left” and collaborative approach to quality while testing earlier and more rigorously by automating and optimizing test and data creation.
October 27, 2022
The Definition of Done: Why it’s Important for Success

Establishing clear communications and expectations across software organizations is essential to the success of any project. A company is less likely to experience unexpected delays or other unpleasant surprises if everyone is on the same page. To ensure all team members are synchronized and to eliminate unnecessary ambiguity, engineering teams should create a “Definition of Done” to guide projects.

What is the “Definition of Done”?

In Agile software development, the definition of done is a working agreement between teams that defines a set of standards to identify something as “done.” Because Agile is an iterative approach to software development, sometimes it is unclear when a task is complete. Product managers often report that something is ready for delivery when it’s “Dev done” and not “QA done.” In other words, the code has been written, but no documentation or tests have been created or executed. For other parts of the software organization, code without QA or documentation is not “done.” Intuitively, the definition of done exists to eliminate confusion about what “done” means across an organization.

How to Create a Definition of Done for Your Software Organization

For some software organizations, the definition of done is a static set of guidelines and benchmarks established at the start of a project. For other organizations, including here at Allstacks, engineers have the latitude to amend the definition of done as they go. That approach empowers an engineering team to make decisions that will produce the best possible results.

When establishing a definition of done within an engineering organization, consider these tips:

● Communicate criteria in a way that the whole software organization can universally understand.
● Ensure that the definition of done addresses both tactical and technical requirements.
● Create a living document for alignment that can be amended because of unforeseen circumstances, like losing a team member or a significant market shift.
● Consider that the robustness of the document will vary by team maturity, and specific teams might need an addendum for special requirements. For example, bigger enterprise IT teams frequently need more detailed documentation compared to two-person shops.
● Establish an understanding among all teams that, until all criteria are met, the user story, task or project is not “done.”

A definition of done acknowledges the shared responsibility between the frontend, backend and QA. Beyond those roles, other stakeholders might contribute to defining “done,” like Scrum masters, board members or business analysts, depending on the organization and its structure. Further, the product owner may also define “acceptance criteria” with goals targeted toward the end-user experience that the engineering team must also address as part of their work.

How a Definition of Done Can Help Teams be Successful

The definition of done helps a software organization maintain alignment on projects and keeps everyone on the same page. Misalignment can have heavy consequences. For example, it can initiate premature conversations with customers before a product is really ready, create friction between internal stakeholders and/or risk other go-to-market initiatives. When the definition of done encompasses defined steps for a project down to the task level, Agile teams can create more cohesion across an organization and eliminate unforeseen delays.

A definition of done outlines each task in a software engineering project. Noting the completion of each task can be an indicator of whether a project is on track to meet established deadlines. By noting the time it takes to complete each task, an engineering team can measure the velocity of a project, and that can help protect due dates or raise a warning flag about the need to shift timelines.

The Definition of Done Helps Support Software Team OKRs

Objectives and key results (OKRs) are a goal-setting framework to help senior leadership focus on what matters to the organization. By defining software engineering OKRs, senior developer leaders can keep their teams focused on aligning engineering goals and activities to business outcomes while also giving them room to experiment.

Creating engineering OKRs involves taking the organization’s objectives and crafting actionable, generalized engineering objectives while identifying the metrics and milestones that drive progress. Translating the organization’s objectives for the engineering team helps them to understand exactly how their work impacts the company.

One of the key benefits of the OKR framework is giving the engineering team a shared language and vision for success. However, without an established definition of done, OKRs will have a lot of unintentional gaps (and teams will have a lot of misunderstandings). Start with a definition of done to ensure that each team knows what their part is and when it’s officially ready for handoff to another team. Teams will pass the baton smoothly across the organization and there will be fewer obstacles in the race toward release in the market.

A Definition of Done Can Help Organizations Grow Smarter

In a small startup with a handful of software employees, establishing a shared understanding of timelines and goals is easy. As software organizations grow, casual chats at the water cooler aren’t enough to keep an organization in alignment. All team members must understand and have input on goals and objectives and those goals must be in writing and accessible to everyone. The benefits of establishing a definition of done include creating a shared understanding and unified language for software delivery, ensuring that new employees have access to tribal knowledge and process expectations, and detecting and removing false agreements between teams to create real consensus. By creating a definition of done early in an organization’s history, company leaders can ensure alignment even as more people join.

September 14, 2022