Author: Bill Doerrfeld

  • How the SRE Role Is Evolving

    How the SRE Role Is Evolving

    In recent years, site reliability engineering (SRE) has garnered much interest. In 2019, LinkedIn listed site reliability engineer as the second most promising job in the United States. Now, in 2021, the role continues to grow and evolve within many organizations.

    Initially spearheaded at Google and credited to engineer Ben Treynor, the strategy seeks smarter accountability for application reliability. An SRE team sets service level indicators, makes error budgets for new features and uses tools like application performance monitoring (APM) to visualize performance insights, among other tasks. SREs are quickly becoming a key check to increase business output in response to new digital innovations.

    I recently met with AppDynamics Regional CTO Gregg Ostrowski to discuss the emergence of SRE and how the approach is evolving. In short, it appears SREs will only continue to grow in importance across more companies, adopting new tactics like chaos engineering, and diversifying their teams with new domain knowledge in response to increasing technological complexity.

    What is an SRE?

    Traditionally, organizations employed system administrators, or sysadmins, to maintain operations for large computing systems and services. However, this typically produced a dichotomy: product developers want to push new features, while operations teams (sysadmins) want to make sure the existing service doesn’t break. The split between developers and operations can be unhealthy and cause friction.

    Whereas sysadmin positions are detached from development and involve a lot of manual work, an SRE approach, on the other hand, takes a more engineering-focused approach to automate operations. Google’s SRE book, which has become a bible of sorts, defines the role as “What happens when you ask a software engineer to design an operations team.”

    So, what does an SRE do? Well, at Google, they are in charge of availability, latency, performance, efficiency, change management, monitoring, emergency response and capacity planning for the services they operate. At Google, they spend no more than 50% of their time on traditional “ops,” like tickets, on-call and manual tasks, and are encouraged to use the remaining 50% of their time to develop automation that will replace human operations labor.

    5 Ways the SRE Approach is Evolving

    It’s been a few years since Google introduced the SRE concept, and naturally, unique iterations have popped up within other enterprises. But, on the whole, where is the concept heading? Ostrowski sees SREs evolving in a few key areas: growth and maturity, increased diversification with domain-specific experts and new monitoring tactics.

    1. Increased Adoption

    First, not all companies have embraced an SRE model. A recent study by Blameless found “… 50% of respondents employ an SRE model with dedicated engineers focused on infrastructure and tooling, or an embedded model where full-time SREs are assigned to a service.”

    The SRE model is gaining momentum, but there is still room for greater adoption. There is also room for internal growth. Ostrowski sees a single SRE team as a single point of failure. “It needs to be a whole department,” he said.

    In addition, SREs are gaining a more prominent voice at the table, influencing feature rollout. “With proper and mature SRE involvement, teams can’t willy-nilly deploy,” he said. Ostrowski views these teams as maintaining a critical balance between business risk and introducing new technology.

    2. Larger, Diversified SRE Departments

    Many companies are experiencing rising user demands, and thus must rapidly scale their application networks. Simultaneously, there has been a Cambrian explosion of deployment types — systems could be using any assortment of legacy infrastructure, mainframe, microservices, cloud environments and multiple cloud vendors. “The complexity and topology of the IT space has grown substantially, with many interdependencies,” Ostrowski said.

    Due to increasingly complex technical stacks, SRE must now cover many domains. Thus, SRE departments will likely require a more diversified team of domain experts. Ostrowski likens SREs to “Navy SEALs of the IT team.” For example, assigning them with experience navigating the nuances behind Google Cloud Platform (GCP) or Microsoft Azure could maintain service reliability in respective clouds.

    Individuals’ personality matters for suitability to SRE, too, said Ostrowski. The role requires a specific type of innovator who considers the repercussions of code, and loves improving processes and performances.

    3. New Testing Tactics Will Emerge

    Ostrowski also foresees SRE departments introducing new monitoring and testing approaches to maintain reliability. One of these is chaos engineering, which champions the idea of intentionally breaking application systems. Different strategies will undoubtedly emerge to help drive user experience (UX) and ensure that performance is always top of mind.

    4. Businesses Rely on SREs to Mitigate Risk

    All things will inevitably fail, at some point. SREs accept failure and learn to manage it, designing repeatable operational mitigation structures. “As companies become reliant on the consumerization of IT, business is dependent on SREs to drive business,” said Ostrowski. This reliance will likely increase as businesses tap SREs to maintain stability.

    Whether it’s reducing mean time to repair (MTTR), programming service level indicators to monitor website load time, or forecasting error budgets for new feature introductions, SREs will be increasingly relied upon to maintain business stability and high performance.

    “The SRE mindset is about coming to terms with a blameless environment,” said Ostrowski. They can help “balance a risk between the business and application team,” described Ostrowski.

    5. SREs Steer UX

    In this digital-only environment, “The application has become the business,” described Ostrowski. If the application is core to the business, monitoring the user journey is necessary to improve it. Ostrowski believes SREs oversee a unique territory that could produce valuable business insights too.

    In addition to monitoring uptime to ensure URL response times meet SLAs, SREs could track UX-related insights, such as conversion rates or cart abandonment percentages. Tracking such analytics and setting standards baselines could help pinpoint problems affecting UX. It could also assist product development in designing better-performing software.

    Final Thoughts

    “Anything that you do more than twice has to be automated.”
    -Adam Stone, CEO, D-Tools

    Traditional operations teams are typically detached from product development and are culturally very different. However, within an SRE approach, operations instead aim to run systems that automate work typically performed manually by sysadmins.

    In this brief introduction, we barely scratched the surface of the topic. And of course, each company is unique in its approach to hybridizing product development and operations. DevOps tenants are closely aligned with SRE — often they are intermixed or one and the same.

    Ultimately, truly reaping business value from this approach will depend on breaking down silos and opening conversations — or at least automated notifications — between disparate units. Only then can operations and development coexist in the most productive way.

  • Amid COVID-19, Developers Struggle to Keep Pace

    Amid COVID-19, Developers Struggle to Keep Pace

    It’s been a wild ride for digital innovation. It’s also been a stressful year for developers. At most organizations, the pandemic has increased reliance on digital experiences (and its programmers) exponentially. IT spending is rising, and competition is skyrocketing. More than ever before, the pressure is placed on developers to meet rigorous demands.

    A recent study by Couchbase revealed how developers are revolutionizing digital transformation efforts throughout the COVID–19 crisis. Yet, this output may not be sustainable. The 2020 Tech Lead Survey, which surveyed 450 senior IT decision-makers worldwide, exposed stressors for development teams, which, if not mitigated, could easily result in fatigue and burnout.

    So Much To Do, So Little Time

    Some digital transformation efforts that were slated to take five years have had timelines reduced to a matter of months. In this unprecedented era, flexibility is an indispensable characteristic to adapt to change. In fact, 63% of respondents said that flexibility to change direction on a whim is critical to meeting digital transformation goals.

    But, extreme flexibility comes at a price. Forty-nine percent of respondents indicated their development teams have been asked to do too much in too little time. As a result, 40% of respondents said their development teams were behind schedule with their current projects.

    “In a year when organizations have had to struggle to adapt to new decentralized ways of working and rapidly evolving markets, developers have come under severe pressure to help their organizations adapt,” said Steve Yen, co-founder of Couchbase.

    It’s apparent that developers are under escalating pressure. If the weight continues to pile on, exhaustion could bring overall production to a grinding halt. At least leaders realize who is driving digital innovation — 92% acknowledge developers are the unsung heroes of current digital transformation initiatives.

    Lacking Clear Direction From On High

    Eighty-six percent of respondents experienced challenges with their development teams. However, this does not wholly reflect poorly on developer inabilities — managers have a responsibility to set more explicit guidelines.

    Another obstacle: goal setting. Forty percent of respondents reported that setting clear, measurable goals for their team is a serious challenge. In addition, ensuring developers use the right technology, and ensuring teams understand overarching strategic goals were oft-cited pain points, as well.

    Resolving these issues will be critical to reducing developer complaints and remediating the stressors outlined above. “There needs to be investment in not just people, but also in the technology, management and communication that will help individuals succeed,” said Yen.

    Solution: DevOps

    So, where should architects direct new investments? Well, of the 450 IT leaders surveyed, 92% believe DevOps could have a revolutionary impact on their digital transformation efforts.

    The report refrains from offering specific technical advice, but makes the following general recommendations:

    • Don’t rely on legacy technology ill-suited for the tasks at hand. Current investments must be future-proofed for tomorrow’s scalability and integration requirements.
    • Don’t assume onboarding new tech will be easy. As digital innovation matures, new technology becomes more refined and specialized. As I’ve covered, whether it’s Kubernetes, service mesh or multi-cloud strategies, all require time to learn, along with some pretty specialized skillsets.
    • Take time to communicate with developers and understand obstacles. Be realistic with timelines, and do not ask for too much. Plan out project goals strategically, because haste does not equate to overall speed.

    Under pressure to perform and maintain flexibility, it makes sense that engineers are feeling the pressure of an amplified development pace. As that pace is likely not to decrease in 2021, finding methods to better support developers is key for long-term sustainability.

    At many organizations, a greater DevOps investment equates to operationalizing and automating processes for areas like microservices CI/CD, and networking and zero-trust security. It could also mean supporting developer workflows with some-code tooling. Other institutions are even introducing low-code tools to democratize the development process, intending to relieve some burden from professional engineers.

    Invest in Developers, Invest in Smooth Operations

    On average, development teams grew by 20% in 2020. As expectations for developers rise, so will investment in developers and the processes that support them. As teams move forward, it will be critical to gather feedback and understand what’s causing gridlock so that developers maintain agility and avoid burnout.

    In this competitive climate, choosing the correct people and systems to support new digital advances will be a competitive advantage. “Investing in [developers] in the right way will help them play to win,” said Yen.

    Since 2017, Couchbase has surveyed IT decision-makers on the challenges they face throughout their digital transformation efforts. Here, I highlighted the major takeaways from the 2020 report. You can view the full findings here.

  • The State of Digital Innovation, One Year Into the Pandemic

    The State of Digital Innovation, One Year Into the Pandemic

    It’s obvious COVID–19 accelerated digital reliance for many companies. And it’s no surprise that, according to a recent study, 89% of tech leaders believe digital experiences are critical to addressing new COVID–19 challenges. In this race toward 100% digital business, agile innovation is the new elixir of life — without it, a company may face an early retirement.

    Kong just released their 2021 Digital Innovation Benchmark, which summarizes the tech priorities of 400 IT leaders. The findings stress the importance of rapid digital innovation, open source, microservices architectures, and high quality practices for scaling today’s application ecosystems. Here are the key points technology leaders need to know to remain competitive.

    Innovate or Dissipate

    The urgency to innovate is increasing. According to the survey, 84% of respondents predict a business will go under within six years if they lag behind in digital innovation. Fifty one percent of tech leaders believe their business can only survive three years.

    If digital transformation happens too slowly, it can pose an existential threat. But there are many factors that can hinder faster digital innovation; the complexity of using multiple tech stacks, reliance on legacy IT and lack of automation. Not only are there internal woes, but external danger — 62% of respondents worry about failing to keep up with competitors who innovate faster.

    It seems the urgency has increased over the last year. In last year’s report, just 37% reported they’d be out of business in three years if they didn’t keep pace with digital innovation. That figure rose to a whopping 51% in 2021.

    COVID Amplifies Digital Drive

    Virtually no one is anticipating a decreased IT budget in light of COVID–19. On the contrary, long-term digital transformation efforts are well on target at the majority of institutions, and 75% expect to see an increase in IT/developer budgets over the next 12 months.

    Interestingly, when it comes to securing the digital-only economy, European tech may have a head start. Only 40% of European tech companies have increased their security focus, compared to 51% in the U.S. This could be due to previous investment in security and stricter regulation in European countries.

    In terms of cloud adoption, 40% of tech leaders feel COVID–19 has accelerated their move to the cloud. Nearly 40% say the pandemic has encouraged the use of microservices and cloud native architecture.

    Quality Outweighs Speed

    When gauging what matters most to tech leaders, “improving operational efficiency” is still a top priority. However, this year’s report saw a spike in respondents saying application performance and reliability, as well as application security, were moving up the priority list. Though 3 in 5 (62%) respondents said they are “extremely concerned” about competition from speedier rivals, application stability outweighs the collective drive to innovate.

    These trends are likely indicative of a notable rise in cybercrime. As I’ve covered previously, to respond to emerging threats, DevSecOps practices and new security tooling are shifting left within the development cycle, automating things like vulnerability detection. Simultaneously, roles to increase application integrity, like site reliability engineers, are in high demand.

    New Technologies and Open-Source Thrills

    As mentioned earlier, IT budgets are increasing across the board. When adopting new technologies, companies place the highest emphasis on the tech that increases efficiency. A surge in automation follows a low-code movement gaining steam. Companies consider the following key factors when introducing new tech: reducing deployment risk, increasing collaboration, and increasing development. Ninety five percent of US respondents also listed the “need to integrate with existing tools” as necessary — no surprise in today’s API-driven development landscape.

    Ninety one percent of organizations use open-source software in 2021 — this is up from 82.5% in 2020. Across the board, open-source is intrinsically embedded into most software projects. We see open-source especially used in open databases, infrastructure automation, API design and testing, CI/CD tools, Docker containers, Kubernetes and service mesh. Arguably, modern DevOps would not be possible without such open source initiatives.

    Bring it on, Microservices

    If you haven’t heard of microservices yet, we would be worried. An impressive 87% of survey participants believe that microservices-based applications are the future.

    While that sounds substantial, a path toward 100% distributed microservices architecture has not been mapped within most organizations. In 2021, only 33% of companies have transitioned entirely to a distributed architecture, whereas most (53.5%) adopt a hybrid mix of microservice and monolithic styles.

    “This is strong evidence that microservices have quickly become the de-facto approach for modern application architectures that enable digital innovation,” the report said. Though more development is required, a microservices-centric future seems obvious.

    Within organizations that have already adopted a microservices approach, the average number of microservices in production is 102. And those are primarily to increase security. Ironically, the report also found maintaining security to be the top challenge of deploying a microservices system. Other drivers for microservices adoption include integrating new tech, increasing development speed and boosting flexibility.

    Kubernetes Becomes Ubiquitous

    Kubernetes has clearly emerged as the industry standard orchestration platform. Eighty six percentof all respondents report they now use Kubernetes in production, or plan to within 12 months. This is a massive jump from VMWare’s 2020 report, which put Kubernetes use at 48%.

    However, there are challenges to implementing Kubernetes in practice. Companies cite security, complexity, performance and monitoring as top setbacks when introducing Kubernetes. To say Kubernetes has a steep learning curve may be an understatement, as abstraction layers are introduced to rectify a widening knowledge gap.

    Service Mesh Prickly for Early Installs

    Interestingly, API gateway adoption and container usage, according to the survey, are both 38%. Yet, service mesh adoption is only at 20%. This could underscore the fact that service mesh is still an emerging concept — service mesh experts say the tech isn’t ready for widespread adoption. Or, it could reflect that production use cases are rare (primarily for supporting vast microservices deployments).

    Early service mesh adopters see increased service connectivity and discovery as a top benefit of the technology. This seems logical, as service mesh aims to be a common networking framework. Other common benefits include cost reduction, meeting compliances, zero-trust security in networking and traffic observability.

    Naturally, service mesh isn’t without its thorns. Complexity in deployment and ensuring performance at scale top the list of challenges faced when deploying service mesh, slowing its adoption.

    Hybrid Multicloud

    The current status of multicloud deployment sees a mix of on-premises and multiple cloud providers, with 41% of organizations using multicloud deployment strategies. While cloud adoption rates are increasing, 31% of organizations are still running on-premises servers. The saturation of cloud tooling, and the market validity of Azure and Google Cloud Platform to take on AWS, have influenced a multicloud environment. Cloud expertise is now a hot commodity to navigate multicloud modalities — so much so that some companies have introduced a new role, The Other CFO, or Cloud Economist, to balance the cloud checkbook.

    Riding out the Virtual Wave

    The pandemic has accelerated the move toward digital experiences, and in turn, quickened the pace of enterprise technological evolution across the board. As COVID–19 effects continue, there’s no indication the virtual wave will recede anytime soon.

    To summarize the current state of digital innovation:

    • Rising digital tide: The pace to innovate is rising, and those left below the new-digital-normal water line will drown.
    • IT spend is up: Most companies have increased IT budgets moderately to support new digital norms.
    • Needs for adoption: Desires for automation, speed, security and integration rank high for adopting new tech.
    • Microservices on fire: The report describes “accelerated mainstreaming of microservices.” Microservices, and the tech that supports them, are clearly the future.
    • Kubernetes everywhere: As Kubernetes rises to ubiquity, last-mile efforts center on security and developer usability.
    • Bullish on service mesh: Service mesh sees early production uses, which will likely signal more growth in this area.

    There’s nothing unexpected in these findings; instead, they affirm what most of us already know — digital innovation is going full throttle. Check out the entire 2021 Digital Innovation Benchmark here.

  • Low-Code Opens API Integration Potential for Citizen Developers

    Low-Code Opens API Integration Potential for Citizen Developers

    APIs have an accessibility problem. Could a low-code API approach make them more inclusive and accessible for citizen developers?

    Most software today integrates with third-party SaaS. TomTom for geolocation. Stripe for payment. Twilio for SMS. Kairos for facial recognition. AWS for NLP … the list goes on. Whatever function you’re looking for, there’s probably an API for that.

    These specialized utilities help software teams avoid reinventing the wheel for things outside of their core competency. They’re standardized, too — most services are programmable following REST or GraphQL guidelines. Serving integrations over HTTP aids in decoupling the business logic from the client interface. Furthermore, OpenAPI Specification, now part of the Linux Foundation, has risen as a standard description format, making APIs more defined and interoperable.

    However, for all this progress, there’s still friction in the integration process, even for seasoned developers. Each API is a bit unique. It takes time to read through the documentation to authenticate and discover methods, parameters and accepted fields. It requires effort to test requests within Postman, decipher error messages and set up calls within your unique environment. On top of all this, downtime and breaking changes are legitimate concerns for over-the-wire dependencies.

    APIs require effort and maintenance, and just integrating third-party APIs alone won’t build you a fully functional application. Yet, organizations find themselves scrambling to find and hire engineers, and increasingly the onus is on non-tech folks to create new digital workflows. Naturally, companies are currently searching for even more low-code solutions.

    So, what’s the glue holding together API products? How can we better connect APIs to the software applications we’re already building and allow citizen developers to reap the benefits?

    My theory: If low-code platforms decrease the API integration hurdle, more problem-solvers can stitch together business utilities to create more advanced internal applications. Making third-party APIs more inclusive for citizen developers could boost the API economy at large and enable creative solutions to flourish throughout enterprises.

    The Need for Low-Code API Integration

    Perhaps the best way to demonstrate the need for low-code is to scare you with some code. Let’s say you want your application to initiate a direct deposit into a bank account.

    You don’t want to build a financial service from scratch, so you turn to Stripe API. You set up an account, grab a key, and dig in to the ‘Getting Started’ materials. You peruse their comprehensive documentation, and find out how to create a payout.

    The docs demonstrate a sample HTTP curl request for creating a payout. Other parameters to pass include destination, methodor source_type.

    curl https://api.stripe.com/v1/payouts \
     -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \
     -d amount=1100 \
     -d currency=usd

    If there are no errors with the request, the response will be a payout object, formatted in structured JSON. It details the balance, data created, currency type and other fields. It looks something like this:

    {
     "id": "po_1HmNKQ2eZvKYlo2CuRNBRWUW",
     "object": "payout",
     "amount": 1100,
     "arrival_date": 1605116514,
     "automatic": true,
     "balance_transaction": "txn_1032HU2eZvKYlo2CEPtcnUvl",
     "created": 1605116514,
     "currency": "usd",
     "description": "STRIPE PAYOUT",
     "destination": "ba_1HmNKQ2eZvKYlo2CzlT6m5gk",
     "failure_balance_transaction": null,
     "failure_code": null,
     "failure_message": null,
     "livemode": false,
     "metadata": {},
     "method": "standard",
     "original_payout": null,
     "reversed_by": null,
     "source_type": "card",
     "statement_descriptor": null,
     "status": "in_transit",
     "type": "bank_account"
    }

    Now, you can parse the response into data your application can ingest. You can also send other requests to update, return and cancel this payout object.

    For specialists familiar with HTTP calls and JSON, this is pretty straightforward. Stripe docs typically generate sample requests for many languages, like PHP, Ruby, Python and others to help within your own environment.

    But what if your users are more comfortable with UIs? Integrations are pretty opaque to the non-technical. Even for seasoned developers, APIs can quickly become too dense, and not all have a great developer experience.

    Hurdles In API Integration

    There are many issues preventing non-tech-savvy users and novice programmers from inserting API-driven business logic into their applications:

    • Onboarding time: Setting up an account, acquiring authentication, reading docs and testing calls to familiarize yourself all takes time.
    • Data Types: Data passed in headers must conform to explicit data types, or else requests may fail.
    • Parsing: Responses are often too verbose; developers must parse the JSON object responses to retrieve the data they need.
    • Programming language: Each language handles network communication differently, meaning requests look different in every language.
    • Breaking Change: Third-party services change often, so if you’re not actively maintaining an integration, you may face breaking change.
    • Differing formats: There are various API standards. REST, SOAP, GraphQL all are designed a bit differently. Plus, webhooks, event-driven and publish-subscribe patterns have nuanced communication models.
    • Security and legal: If APIs aren’t appropriately secured, hackers could easily steal user data or cause outages.

    As you can see, a simple programming task like “make a bank deposit” becomes extremely complex the more closely we look at the logistics of actual implementation.

    Furthermore, just because you’ve made an integration work, that doesn’t mean you have a fully-functional application, securely deployed in the cloud, with a friendly UI  that’s universally accessible. A single application may have numerous integrations, meaning the integration process must be duplicated for every external service you connect.

    API Categories

    Overcoming these challenges is a lot to ask of non-professional developers, who may not be comfortable consuming business applications from the command line. This issue is further compounded as integration extends beyond public APIs, too.

    In a recent interview with Divyesh Kharade, co-founder/CEO at DronaHQ, he identified three main API categories:

    • Ready-made SaaS APIs: Microsoft Teams, Slack, Sendgrid, Twilio, Google Sheets, etc. These are the connections to third-party operational business utilities and applications.
    • Connecting to databases: Many companies require secure connections with custom internal databases or legacy internal MySQL databases.
    • Internal custom software: These are private or partner APIs. They may be new API-first microservices or custom-built legacy internal apps. Such middleware may expose many APIs.

    If a low-code application wants to become a holistic software development add-on, it must anticipate all potential environments. This includes integration realities for various scenarios.

    Citizen Developer vs. “Non-Tech”

    So who, exactly, benefits from more accessible integrations? The term citizen developer is often used to refer to any non-professional software developer with minimal programming training. “When most people speak of citizen developers, they mean functional guys that can build tech,” says Kharade.

    As I’ve said previously, fostering a culture of citizen developers is becoming a bigger priority amid dwindling developer talent and a combination of other forces impacting software development requirements.

    Yet, citizen developer tends to means something a bit different to everyone involved in the low-code space. Kharade seems to prefer the term “non-tech.” Non-tech users may include designers who may not have a software background but the relevant UX/UI knowledge to assemble workflows. “Design is a big part of any application,” said Kharade.

    Masking the Complexity of APIs

    If low-code tools mask complexity and make APIs more accessible for these non-engineers, it could inspire many creative solutions. With low-code, “creativity is limitless, and integrations really help with that,” said Kharade. “Once they see that power, they aren’t going back again.”

    For example, if an internal application requires chat functionality, Slack could be a great addition. Low-code layers could hide complexity in the backend, to the point that it “doesn’t feel like a request in background,” noted Kharade.

    The solution works both ways. Utilizing low-code platforms to manage data and expose APIs could enhance visibility and extensibility. Kharade described an industrial manufacturing use case, in which shift workers manually entered machine operations data on paper. That was before COVID; once lockdowns were mandated, the company started digitizing this record-keeping process.

    Access to real-time data, and the ability to filter and organize historical data, is already beneficial.  “Six months down the line, they may consider how to make this data accessible using APIs,” said Kharade. “The journey has just started — it will only get smarter and better.”

    Trigger-Based Workarounds Were A Start

    Of course, tools like IFTTT and Zapier have already been making APIs accessible to laymen for years. These platforms use friendly, UI-based front ends over the foundation of complex integrations, allowing users to set up trigger-based automation. If this particular service does this, this other service should do that.

    These are cool tools for stitching together service-service communications. However, these workflows are a bit isolated from the larger application development picture. For example, an application for hiring employees may require more than connecting a form API’s input to a Google Sheets field. This could require a custom database, a way to upload documents and the ability to deploy an application to the cloud.

    Since low-code already has standard components for drag-and-drop, adding a common method to insert third-party API integrations feels like a natural feature enhancement for these platforms.

    Making Integration More Accessible

    ProgrammableWeb records 23,000 APIs as of this moment. Clearly, many innovative, programmable SaaS offerings are ripe for the picking. Some paid, some open, all with slightly different integrations.

    “It’s difficult for a non-tech user to register and interact with an API,” said Kharade. Figuring out authentication, OAuth, headers, payloads and responses requires expertise. Though a professional developer may be required to “register them once,” says Kharade, once third party integrations are set up, they “should be drag-and-drop and click-to-use.”

    Solving integration problems is just one of the many arenas where low-code platforms could shine. If platforms can successfully establish (and maintain) connectivity, consumers could more easily access the core logic they are searching for. Democratizing robust solutions for traditionally non-tech companies is especially important amid the COVID-19 crisis.

    To be clear, I don’t believe it’s the API provider’s job to cater to end-users. Instead, they should focus on creating excellent developer experiences. Naturally, the type of business application you are trying to construct and your access to software engineering talent will, to an extent, determine a project’s technical stack. The JAMstack style, for example, may be a better fit for your team’s makeup and project objectives.

    That said, there seems to be room in the market for additional low-code layers to open up integration. In the same way APIs avoid reinventing the wheel for specialized utilities, low-code layers could standardize the pains of integration itself.

  • AI Automation Won’t Replace Jobs – It Will Replace Processes

    AI Automation Won’t Replace Jobs – It Will Replace Processes

    As Software 2.0 takes shape, how are AI processes—and the jobs that go with them—changing?

    When it comes to new AI automation, the Software 2.0 movement can be seen as akin to the auto industry’s recent evolution. Beginning in the 1990s and rapidly progressing into the 2000s, the auto industry saw a huge rise in AI automation such as digital vehicle diagnostics.

    Throughout this, people worried that digital transformation might take jobs away, when instead it led to “greater profits, productivity, and competitiveness,” according to a 2008 study by the journal of Technological Forecasting and Social Change. In short, these innovations didn’t replace jobs, per se; they simply changed the processes of jobs from the manufacturer’s assembly line down to the mechanic’s garage.

    An auto diagnostic specialist used to perform tedious tasks such as counting flashes and converting them to error codes on printed-out tables. Computerized diagnostic tools made this process faster, more reliable and, you might imagine, far less tedious. According to diagnostic specialists Helmut Frank and Uwe Schmidts, digital transformation went from being “a necessary evil to being a key to new, interesting and innovative functions.”

    You might liken the above example to the innovations that AI automation and machine learning (ML) have offered to software developers, who until recently, were tasked with a much more hands-on approach to deploying and maintaining applications. In many ways, DevOps practices have heroically come in to break them free of this minutiae, automating many of the repetitive tasks involved in application management. By design, it’s increasing efficiency and security while decreasing tedium and the chance for human error.

    Speaking to tedium, Mike Loukides and Ben Lorica, in their article “The road to Software 2.0” noted, “Up until now, we’ve built systems by carefully and painstakingly telling systems exactly what to do, instruction by instruction. The process is slow, tedious, and error-prone; most of us have spent days staring at a program that should work, but doesn’t.”

    In short: Much like digital transformation brought efficiency to the life cycle of automobiles, AI automation is offering efficiency to every stage of the software development cycle.

    Don’t Change the Players, Change the Game

    Indeed, a new generation of AI automation and ML tools is now emerging, shaping what some call a Software 2.0 movement. But before you begin to fear that these tools will lead to a “they took our jobs!” revolt, consider that they may be celebrated instead as a means of not only augmenting the work of software developers but also helping offset the ongoing developer drought. According to a 2019 Stack Overflow survey, over a third of developers named “not enough people for the workload” as their primary challenge to productivity. Rather than replacing jobs, automation should be viewed as offering efficiency in an industry that desperately needs it.

    As an example, the process of writing code, though more streamlined today than it was 25 years ago, has remained a mostly manual process. This has presented an excellent opportunity for the automated processes we are beginning to see, which bring in AI to automate aspects of code development such as program synthesis, smart code completion and static code analysis.

    Assaf Araki spoke to this in the article, “Software 2.0 takes shape.” Araki described how automation is filling gaps in many other areas, such as regression testing, code analysis, AI for vulnerability detection and observability tools to monitor software health.

    From Monolith to Microservices

    Just like the recent, mysterious disappearing monolith in the Utah desert, the relevancy of monolithic development may be on its way out.

    In the past, software cycles were longer. Teams updated monolithic-style deployments only monthly or yearly. Nowadays, companies are under pressure to push much more rapidly, often turning to microservices and progressive release strategies. This new paradigm requires much automation to increase release efficiencies and enable a seamless CI/CD.

    As Araki pointed out, “Automation in software is best viewed through the lens of Continuous Integration (CI) and Continuous Deployment (CD).”

    How to Know When AI Automation is Worth Its Salt

    It’s smart to consider the investment of both money and time when deciding whether an AI solution is worth adopting. “AI systems need to add value that exceed any new costs and requirements (such as the need for labeled data, hiring new staff, etc.),” Araki said.

    You should also be asking if the solutions are wide-ranging enough to meet the needs of your developers and anyone else on the team who may be employing them. Key considerations for AI automation, Araki noted, are focusing on tasks and systems that are frequently used, adjusted or tuned.

    4 Examples of Versatile AI Automation and ML:

    1. Data management systems from simple systems that employ ML to more complex systems that enable self-driving data management.
    2. Dashboards that are automatically created, ideally with intuitive and visually appealing UX, that use AI automation and ML to bring up the right elements—more progress is expected in these areas in the coming years.
    3. Websites and mobile apps that use AI automation to become highly personalized interfaces, such as those that employ real-time data for online and reinforcement learning—these are also expected to progress in the coming years.
    4. UX and design tools that use simple automation to design and revise digital media, with ML being a promising way to kick the automation up a notch, such as deep learning for game developers.

    AI Automation and ML to Grant Opportunities

    In short, Software 2.0 doesn’t currently appear poised to threaten jobs or decrease developer need. Instead, it seems to be changing the processes engineers are using and creating efficiencies in their work, at a time when their expertise is in increasing demand. Demand for novel roles, like the Cloud Economist, demonstrates the emergence of new opportunities for humans with cloud knowledge who work alongside AI.

    The new evolution of AI and ML also could create a unique subset of developers with skills keeping par with these new innovations—much like digital transformation in the auto industry led to evolving specialties in computerized vehicle systems.

    Not to mention that just as low-code has paved the way for citizen developers and AutoML led to citizen data scientists to fill large gaps in demand, AI and ML solutions have the potential to further broaden and democratize software and code development.

  • Low-Code To Become Commonplace in 2021

    Low-Code To Become Commonplace in 2021

    A new study finds 85% of engineer leads predict low-code and no-code will soon become common

    New findings from an Internal-endorsed study suggest 2020’s software development challenges will undoubtedly carry into 2021. Many businesses are making immediate transitions to a digital-only economy, forcing a ton of new app development requirements. Yet, hiring top engineering talent is especially difficult, amid a developer drought and pandemic uncertainties.

    One in three organizations has implemented a hiring freeze. Respondents also cited rising development challenges and insecure internal software, putting a strain on normal operations. To help address these challenges going forward, a whopping 85.1% believe implementing low-code and no-code will become more and more omnipresent within their company.

    Below, we dive into the report’s major findings, which surveyed 500 IT and engineering workers at medium-to-large enterprises. Using this data, we’ll forecast hurdles software development teams may face in 2021, and gauge if low-code tooling adoption is the ultimate panacea or no more than a Band-Aid for a much larger ongoing problem.

    Development Challenges Will Increase

    In the thick of the pandemic, software is often the only touchpoint many traditional companies have with a user. Naturally, the number of digital experiences required to meet this new reality is skyrocketing. Experts and data show this trend has no signs of stopping.

    In fact, 60.1% of surveyed saw app development increasing due to work-from-home policies. Only 9.7% reported a decrease in app development. One could easily attribute rising app development with growing software requirements brought on by the current digital-first climate.

    In this new economy, finding bold ways to increase productivity is critical. Improving the efficiency of engineering teams was a priority for most respondents and a high priority for 40% of respondents. For small groups with lean margins, improving automation for software delivery could help compete with larger players.

    Talent Will Become More Competitive

    The biggest challenge employers face is finding talent—30.6% of respondents cited increased competition for talent as a top challenge in 2020 and 22.2% also recognized developer drought as a significant obstacle. If you had trouble finding skilled software engineers in 2020, good luck next year …

    Looking to 2021, 45% see competition for talent remains a top challenge. Trailing problems include developer drought, immigration issues and a poor reaction to new work-from-home policies.

    Hiring Practices Will Continue to Shift

    With COVID–19 in full force and a developer scarcity, you’d think hiring would come to a freeze. Yet, the results of this study were mixed—hiring practices are pretty varied across teams.

    According to the report, 34.5% of organizations surveyed had paused hiring new engineers. Conversely, 37.3% have hired more engineers, and 18.5% of organizations reported no change to hiring practices at all. About 10% had layoffs.

    Whereas some tech companies are thriving in the new normal, others may pause hiring due to uncertainness around COVID-19. These companies could be turning to low-code technologies as a replacement. “If you pause hiring, you still need to build, and low-code is a great option for that,” said Arisa Amano, CEO of Internal.

    Low-Code Will Become Commonplace

    To address the above issues, companies are turning to low-code solutions. These platforms help relieve the development burden for internal business tools and open up citizen developer use. And, it’s more than a fad—the report was able to squarely demonstrate an uptick in low-code use.

    Most companies (75.2%) have already implemented a combination of low-code and traditional engineering methods. It appears that low-code is complementing existing software engineering practices at a number of institutions.

    In addition, 28.8% of respondents predict low-code and no-code will become a high priority in the coming months, and for 21.6%, it will become an essential focus. Impressively, 85.1% of developers foresee low-code and no-code as becoming more commonly implemented within their company.

    Low-Code Will Open Doors for Non-Tech

    The citizen developer concept has been gathering marketing steam, but it might not be just hype. Now, data suggests low-code tools are actually opening doors for such non-developers. Seventy percent of companies said non-developers in their company already build tools for internal business use, and nearly 80% predict to see more of this trend in 2021.

    It should be noted that low-code and no-code do not seek to replace all engineering talent; instead, to free them up to engage in more complex tasks. “With low-code, you free up your engineers to work on harder problems, instead of having them work on basic things,” said Arisa Amano, CEO of Internal. She believes this could translate into more innovation companywide.

    Surprisingly, bringing non-traditional engineers into the development fold is not being met with ambivalence—69.2% of respondents foresee that citizen developers positively affect engineering teams, with the rest primarily exhibiting a neutral reaction.

    Improved Access Control Will Boost Internal Security

    The costs of internal security threats are high. Breaches could decrease customer trust, harm brand reputation and lead to escalating legal fees. With cyberattacks a prevalent concern, cybersecurity must come back in style.

    Not only are threats external, but internal too—46.8% of respondents were extremely concerned about employees having access to customer data. The cause for concern is duly noted, as the majority, 58.1%, have experienced internal data security breaches in the past.

    The reasons for internal breaches vary across the board. Respondents attributed the top cause simply to a rise in the number of employees and contractors with access to sensitive information—a surge in employees means a greater chance of human error or leaked credentials. It also means an increase in the number of devices with access to sensitive data—this is particularly tricky, as CISOs face a dilemma on how to govern privileged access for housebound laptops and mobile devices.

    Respondents also noted the increasing complexity of technology and the overabundance of new cloud applications. It’s hard to expect DevOps to secure modern infrastructure when they’re still trying to figure out how it works! As a knowledge gap could quickly introduce vulnerabilities, 62.7% of respondents are actively investing in increased employee security training.

    Other security problems, such as a lack of granular access control or lack of logging and audit trails, seem like they could be mitigated with the right configurations and DevSecOps approach. In fact, 55.4% of respondents are in the process of baking in fine-grained access controls. “Having that at the forefront of internal tools is key,” said Bob Remeika, co-founder, Internal.

    Shifting security left could correct an age-old trend of adding security after the fact, which only prolongs the problem. On the whole, companies require a combination of better access enforcement, multi-factor auth and increased monitoring to mitigate mounting threats.

    Low-Code Becomes Mainstream in 2021

    Software development teams face many challenges in the coming year. Demands for new software will increase and processes to increase efficiency will fall under the microscope. Rising engineer competitiveness and developer drought will also continue to shift hiring practices.

    Companies that weren’t 100% digital will be forced to embrace the new norm and develop internal software and automation. Yet, building internal tools isn’t straightforward and will be especially difficult for companies less technical than their software-as-a-service counterparts. All this, paired with ongoing security threats, means internal app development in 2021 has its challenges.

    Although conditions look dicey, the underlying issues are not unusual, Remeika said. They have just been amplified. “There’s always been a developer drought, companies have always been trying to find ways to do more with less, and competition for talent is never getting less fierce,” he said.

    Though we’ve seen a spike since COVID-19, the desire to offload work has been everpresent. Remeika believes sophisticated low-code platforms will empower technical non-engineers and assist with things people typically tend to disregard when building internal applications, such as security, deployment, testing and review capabilities.

    Nevertheless, development team output appears to be resilient throughout work-from-home policies. And as I recently covered, a low-code movement could also enable a democratization effect, helping SMEs who can’t afford the top talent to still compete at the same table as large tech companies. We expect low-code and no-code adoption to spill well into 2021, seemingly, as a panacea to these mounting concerns.

  • Demystifying the Low-Code Category

    Demystifying the Low-Code Category

    The low-code category is estimated to grow substantially in the coming years. By 2024, Gartner projects 75% of large enterprises will have adopted four low-code or no-code tools for both IT application and citizen development needs. But what exactly is low-code, and what are the scenarios in which it is needed?

    Low-code, like so much other software jargon, is somewhat ambiguous. Under this umbrella term exists a well of various tool types—website generators, form builders, API connectors, database builders, workflow automators and more, each requiring varying levels of technical proficiency. Some are 100% codeless and UI-based with drag-and-drop templates. Others need programming acumen to set up to complement a developer’s pre-existing workflow. When we talk about low-code, we’re typically referring to platforms that help create internal business utilities.

    I recently connected with Divyesh Kharade, co-founder and CEO at DronaHQ, to demystify the low-code category. Below, we define citizen developers, break apart low-code versus no-code and dig into specific types of low-code scenarios. Given this knowledge, technology leaders can better identify what tools they need to increase rapid development for their business needs.

    Hard Code Is to No-Code as Manual Is to Driverless

    So what’s the difference between traditional coding, low-code and no-code? To demonstrate the nuances between emerging code paradigms, Kharade compares them to automobile types.

    Manual Coding

    Traditional coding is like a manual car with a gearbox

    With a manual car, you must know how to switch between gears. Not to mention, you must understand driving itself and how to navigate busy streets, all while following traffic laws. We can compare manual driving to traditional coding. Manual coding is for trained developers.

    For example, consider we need to develop an application to interact with a database. To create this from scratch, this could mean coding a PHP application, connecting it to a back-end database management system such as PostgreSQL and hard-coding commands for ingesting data and manipulating entries.

    Manual coding requires an adept understanding of programming languages and development frameworks and libraries and collaborative development processes such as git. Naturally, traditional coding is in the professional developers’ realm and often necessary for software teams developing proprietary software-as-a-service offerings. It affords ultimate customizability.

    Low-Code

    Low-code is like an automatic car

    With an automatic car, you must still know how to drive, but gear-switching is automated. Cruise control is a nice perk, too. Similarly, low-code seeks to make development easier. It even enables some non-tech folks to construct applications. Low-code is used by professional developers and citizen developers.

    If we consider our database example, a low-code solution would likely involve a one-time registration of a database. This would require a little technical expertise to initiate, Kharade said. However, after which, it would be smooth riding—a user interface would be utilized to develop applications that interact with the database.

    Low-code platforms often incorporate various drag-and-drop building blocks to help develop applications. These pre-generated components may include tables, textboxes, dropdowns, charts, maps, forms, user directories and other elements. Low-code solutions typically make code under-the-hood accessible and portable, too. Degrees here vary. Some, such as some-code, are more geared toward alleviating operational tasks for skilled engineers.

    Low-code solutions also may assist with the surrounding operations necessary to create a fully functional app, such as hosting, deployment and monitoring. Example providers in the low-code spectrum include Appian, Claris Filemaker, Mendix, Outsystems, Quick Base and Salesforce Lightning.

    No-Code

    No-code is like a driverless car

    In an automated car, AI handles all. Driving through traffic and re-routing is automated. You don’t need your hands on the wheel—but you do need a destination. Similarly, no-code is the most usable, non-tech option for application development. No-code is geared toward citizen developers and non-tech users.

    No-code takes low-code usability to a new level—100% UI-driven, with pre-built templates and the ability to string together utilities. No-code is popular in connectivity and triggers, taking the burden out of integration. Of course, the more you automate, the more granular control you sacrifice. Many no-code options simply require logins and permissions for business applications you already use. Examples include Zapier, IFTTT, Kissflow, Airtable and Retool.

    The trend toward low-code and no-code is industry-agnostic, too, having seen adoption by banking, manufacturing, consumer goods, eCommerce, and other areas, Kharade said.

    5 Common Low-Code Scenarios

    Now that we understand the variants, what are enterprises building using low-code? Kharade sees four common scenarios:

    1. Building internal forms: Collecting data in a form, such as for an inspection or audit. May require PDF generation. Non-tech folks with an understanding of their industry could create forms.
    2. Extending with publicly available forms: Making the form public for registration or lead generation. Requires little added functionality and a no-code user could likely implement.
    3. Working with databases: Junior engineers will readily understand database mechanics, but it’s not entirely non-tech, said Kharade. For example, skills may be required to synchronize databases with other cloud operations.
    4. Business process management (BPM): Digitizing the traditional corporate project approval processes. What was once paper-based or email-based is streamlined. It could involve picking up an order, pushing to a manager and a manager approves. Digitizing this process requires a deep understanding of the data structure.
    5. Purely open applications: Perhaps used for onboarding or internal job postings. Requires a fuller amount of user interaction design, workflows, file hosting and API connectivity. May require skilled developers.

    “Building your data structure at the start is extremely critical,” Kharade said. To him, more complex projects will still likely require skilled engineers at some point. But, as you can see above, there are many areas where low-code and no-code could streamline the creation of simple apps and tools.

    Low-Code, Demystified

    In other pieces, I’ve attributed the rise of low-code to an ongoing developer drought and the emergence of a digital-first economy during COVID. Throughout the pandemic, “there has been a frenzy to digitize any manual process,” noted Kharade. “People are scared even to use a pen someone has used.”

    Low-code has in part become a critical option to stand up replacement business applications rather effortlessly, providing citizen developers or non-tech users new digital capabilities. With its surge in popularity, the marketing language around this area is a bit befuddling, and many applications claim to be low-code or codeless.

    Of course, you could call any tool with a UI “no-code.” Technically, your iPhone apps are codeless from a user’s point of view. However, what sets these low-code development styles apart is their use in forging internal business applications and automating repetitive software development tasks. By utilizing low-code, traditionally non-tech companies could wrap operational capabilities around basic applications and scale them like tech companies.

    Another key to low-code, which I will expand on in a follow-up piece, is solving the integration hurdle with third-party APIs. “It’s difficult for a non-tech user to register an API,” Kharade said. One must understand authentication, OAuth, payloads, HTTP methods, responses and headers just to make a single call.

    API-driven services have needed better usability for quite some time. Low-code layers could make consuming a vast array of SaaS more inclusive—a fascinating prospect to problem-solvers of all skillsets.

  • The Future of Application Development

    The Future of Application Development

    What lies ahead for application development? Can we look forward to a less hierarchical system?

    The current state of application development has an often overlooked problem: It’s inaccessible.

    Top tier companies such as Google, Microsoft, Apple and Facebook are at a major advantage. With their large capital, they can invest in innovative development tools and round up extensive workforces of leading talent to use them—which matters, especially amid an ongoing developer drought. Meanwhile, companies lacking these privileges must make do with a leaner toolset and smaller, less cherry-picked workforce—which is not to say that they don’t make smart, creative use of what’s available to them, but that they deserve to play ball on a more equal playing field.

    Unfortunately, the pandemic compounded this issue, at times bringing even greater advantages to the tech giants. To overcome what some consider a trend of software elitism, efforts have been made to accelerate the pace of application architecture for all. Employing no-code and low-code tools, for example, could help the smaller guys compete. However, as Robson Grieve, CMO of enterprise software company OutSystems sees it, “these latest approaches are mostly point solutions solving a limited set of problems.”

    I recently invited Grieve to share more of his take on this issue and how he envisions a future of application development that works inclusively for a wider set of companies.

    Application Development: The Other 99%

    “It’s frustrating that the best, most advanced development tools have traditionally only been accessible to the largest, richest companies,” said Grieve. “Everyone else suffers with ballooning backlogs and struggles to take advantage of the latest, greatest technologies.”

    While it may sound like a platform out of a Bernie Sanders speech, recognizing the advantages of the top 1% in the application development world brings up a valid line of questioning as to whether maintaining this status quo is really how we want to go forward. After all, wouldn’t it be better if more companies had access to the best technologies? Wouldn’t we all benefit from that?

    Creating a More Equal Playing Field in Application Development

    Luckily, we are already feeling the winds change in this department, as top tier companies are beginning to lose their stronghold on the crème de la crème of development tools. What’s beginning to change, Grieve believes, is “access to the most sophisticated tools and advanced technologies are being democratized … sophisticated, feature-rich, beautiful, and mission-critical applications are no longer the sole domain of the elites.”

    How is this being realized? In many ways, it’s coming from the makers of the development tools themselves, who are bringing careful thought and creativity to tool design. Some of the strongest approaches in Grieve’s eyes include the following:

    • Holistic approaches that streamline the most critical innovations into single, integrated platforms.
    • Low-code approaches that help accelerate development while retaining the power and expressiveness of traditional code.
    • Platform approaches that offer design, development, deployment, monitoring and management capabilities in one place.
    • Multi-functional approaches that serve a diverse team, support the full DevOps life cycle and, in effect, improve productivity and collaboration. For example, platforms crafted to serve the unique needs of UI/UX experts, business analysts, professional developers and architects.
    • Automated approaches that allow applications to be continuously updated and seamlessly modified once they’ve been deployed, using an AI-powered automation layer that underpins the tool, managing application dependencies and providing self-healing capabilities to ensure that regardless of what changes are made, the tool never breaks.

    One thing that stands out here, according to Grieve, is the idea that broad, multi-functional development platforms reign supreme. As Grieve noted, “the most valuable tools are the ones that solve the broadest set of problems.” In general, approaches that keep in mind security, scalability, reliability and availability will be robust solutions for diverse teams, both large and small. A bonus is if the tool is visually appealing and model-driven for the best ease of usability and user experience.

    All for One, and One for All

    The issue of software elitism is one problem that can be seen as a clear opportunity, and that opportunity is not being missed among the smartest makers of development tools. As Grieve said, “Modern application platforms have the potential to help the vast majority of the economy succeed by offering a better approach—one that takes state-of-the-art technology and makes it more approachable, accessible and adaptable.”

    Again, we see a version of AI democracy at play. This is an exciting shift for many companies and teams that exist outside the elite tier. As an example, Grieve said that instead of “procedural code or minimalistic point solutions,” modern applications platforms “harness the power of advanced technologies and AI to create efficiencies for companies outside of the tech bubble.”

    This could be one way to level the playing field and allow companies of all sizes to complete the work needed, shrink backlogs, overcome developer drought with tools that are friendlier to citizen developers, support team collaboration, accelerate efficiency, and—for the underdogs—compete with the big dogs.

    Looking to the Horizon

    So, what does Grieve see for the future of application development? To him, it is bright if we continue on the current trajectory. “More and more,” he said, “successful development projects will require business and IT to work together and collaborate closely.”

    This means investing in sophisticated technologies, with powerful built-in AI, that can help companies ensure that even though applications are built quickly and efficiently, they are built securely and reliably—and in ways that can be easily scaled and evolved.

    If this is all to be realized, Grieve forecasts that we can look forward to “a bright future that is welcoming of a much more diverse and representative community of people who build applications together to solve all the most pressing problems.” It’s an optimistic view, but one that many will welcome as a fresh breath of air in what have been some particularly trying times.

  • KubeCon Coverage: Incentivizing the DevSecOps Culture

    KubeCon Coverage: Incentivizing the DevSecOps Culture

    Just as DevOps set to de-silo development and operations teams, the DevSecOps movement is bringing security to the same table. A shift-left security mindset is permeating much discussion of late.

    Cyberattacks are on the rise in the era of COVID-19 and cybersecurity has become paramount to arm business-critical applications. Furthermore, new regulations have emerged to protect personally identifiable information (PII). A new cybersecurity awareness feels quite apparent throughout the tech spectrum. As a result, security policies are poised to integrate more directly within the process of engineering and deploying applications.

    Nonetheless, while DevSecOps feels necessary to implement at this critical hour, it does present challenges. Primarily of which is a significant cultural shift. Uniting security and development isn’t as easy as it sounds, especially when faced with rigid enterprise bureaucracy. Plus, how can an organization incentivize DevSecOps if developers fear security tools will slow down their workflows?

    The DevSecOps panel at the recent KubeCon + CloudNativeCon North America 2020 brought together leaders in the DevSecOps arena to advise introducing a security-first mindset to DevOps. Dan Papandrea, field chief technologist at Sysdig, led this super-informative panel. It featured:

    • Emily Fox, DevOps Security Lead, NSA
    • Peter Bosch, Cisco
    • Sunil James, HPE
    • Christian Weichel, Gitpod
    • Nicolas Chaillan, U.S. Air Force

    Below, I’ll cover some of the key insights garnered from the event. To summarize, the panel identified how policy-first tools and open source initiatives are placing security controls within IDEs and CI/CD pipelines. And what will it take to get everyone onboard? It appears that transparent tooling automation, making security adherence natural and a new wave of security-aware employees will be critical components in the future era of DevSecOps.

    What is DevSecOps?

    First off, to get a better picture, what exactly is DevSecOps? And why has it come about?

    In a previous article, we defined DevSecOps with six traits: security automation, security culture, de-siloing IT, shifting security left and security as an enabler (not a stalling force).

    The KubeCon panelists seemed to echo all of these points. Fox described DevSecOps as “a term used by organizations to ensure their DevOps approaches have a security-first mentality.” To her, this means placing security automation within workload environments.

    DevSecOps is something new. For most organizations, security isn’t baked into the standard development processes—it historically hasn’t been the first priority, Chaillan said. DevSecOps is changing that, adding continuous monitoring, zero trust enforcement and reducing the overall attack surface.

    The move toward continuous release is also an impetus for DevSecOps. “Our cadence of delivery has been quickened,” said James. All organizations now feel pressure to deliver value to customers rapidly. An interesting byproduct of building security-first, he said, is “the ability to create the consistency and speed with which you can actually offer these capabilities.”

    Another cause for DevSecOps is the sheer imbalance in the workforce. On average, the number of developers far outweigh the number of security folks, noted Bosch. So, DevSecOps could scale security across an organization more effectively than human power could.

    Bosch also is hopeful DevSecOps could put an end to time-consuming bureaucracies stalling an agile security process. “Developers used to write code and wait for free months to open a firewall.” This may have been standard operating procedure decades ago, but in the age of continuous everything, enterprises must evolve how they handle IT security policies. “By uniting security with developers, we bring speed, robustness, avoid OWASP issues, and build applications that better contribute to the bottom line,” he said.

    Finally, embracing DevSecOps is a significant commitment. The combination of culture and technology is essential, noted Weichel. As we infuse security testing into CI/CD pipelines and make everything security-first, he said, we indeed undergo a cultural shift—it’s something that permeates everything.

    Challenges in Implementation

    Companies attempting to introduce DevSecOps may run into a few challenges, including instilling the culture, implementing regulatory compliance and finding the right blend of open source and propriety tools to get the job done.

    Bosh is no stranger to sluggish enterprises and the systemic change required to introduce DevSecOps. In these environments, changing companywide antiquated processes may require executive sponsorship. “By and large, it’s mostly culture,” he said, yet acknowledged a vast divide between the two groups: “How do you convince security do get out of their comfort zone? And how do you convince developers to get out of theirs?”

    There is also developer experience to consider: Developers often view security tools as limiting their ability to deliver code quickly. Perhaps the biggest challenge to overcome is improving the experience of working alongside new security policies. If new practices are causing more pain, instead of automating pain points away, DevSecOps could suffer.

    Outside of team dynamics, there are also technical hurdles involved when introducing and enforcing legal regulatory requirements. “Getting any sort of compliance baked into CI/CD pipelines is a challenge,” said Fox. Part of the problem is that the tooling to enforce such requirements isn’t there yet. “Cloud-native is continually evolving and security is constantly playing catch up,” she noted. “It’s an ongoing challenge.”

    Incentivizing the DevSecOps Culture

    Employees need incentives to change established habits. With the above challenges in mind, how can organizations create incentives to adopt DevSecOps? Well, outside of monetary incentives or straight up demanding it, incentivizing that security culture could happen in a few different ways, the panelists noted.

    To Fox, it’s about “making security transparent to the developers.” You want developers to quickly build and deploy secure applications without the need for intensive cybersecurity training. Thus, the tools and processes must be as transparent and usable as possible. DevSecOps is possible “if you make it second nature,” she said.

    Another way to consider DevSecOps culture is the end performance. If customers are happy, this is a positive gauge. James said he has witnessed CIOs at a few organizations using customer satisfaction scores as a guiding light for not only the business but for security effectiveness well.

    Bosch similarly stressed an indirect incentive about overall company performance. “Companies that really embrace DevSecOps leads to better code,” said Bosch. This leads to more robustness, higher speed, and better-operating enterprises. Co-building a thriving success could be ample incentive alone for those who have the whole team in mind.

  • How Service Mesh Enables a Zero-Trust Network

    How Service Mesh Enables a Zero-Trust Network

    How can we achieve high-grade security within a microservices ecosystem? Service mesh may be the answer

    Enterprise applications often assign an identity to humans to determine security clearances. Using standards such as OpenID Connect and SAML, we can delegate authorization scopes across a suite of applications to determine precisely what a user is permitted to do. But what about the applications themselves? Some believe that microservices should follow this identity-first approach, too.

    For large-scale application ecosystems, microservices is arguably more secure than a monolith. As Marco Palladino, CTO of Kong, described, “A monolithic application is a security incident waiting to happen.” Yet, microservices shouldn’t necessarily trust one another. So, how can we achieve high-grade security within a microservices ecosystem?

    The answer may lie in service mesh. I recently met with Palladino to discover how a service mesh can achieve bank-grade security for established and emerging microservices patterns. Similar to how enterprises assign an identity to end users, a zero-trust security network gives an identity to microservices themselves.

    A Zero Trust Network Trusts Nothing

    With more personal data used in cloud operations, more and more personally identifiable information (PII) is vulnerable to attack, and the costs are high. In Q1 2019, the cost of breaches in the financial sector was $6.2 billion. In 2018, 97% of breaches exposed PII, a top target for cybercriminals.

    Organizations are thus feeling pressure to secure their assets. The most precious data is held within financial services and health care, where GDPR and HIPPA enforce strict accordance with data privacy. This pressure is leading U.S. banks to clamp down third-party application access. Simultaneously, European banks face similar security dilemmas while appeasing PSD2 regulation for open banking.

    To safeguard user data, organizations are adopting a zero-trust security model. “Zero trust security means that we’re not trusting anybody,” said Palladino. “We don’t trust our own services. We don’t trust our own team members.”

    Placing too much trust in users, services and teams could cause a catastrophic failure. And, “there is no bigger risk than thinking you are secure, while in reality, you are not,” he said.

    Monolith ⮕ Microservices

    In practice, architecting a zero-trust application network runs parallel with the trend toward microservices designs. Though it may seem counterintuitive, a microservices system is arguably more secure than a monolithic architecture.

    In a monolithic application, every resource—whether invoices, billing orders or transaction history—is always just one line of code away. Since everything is connected, you could expose a resource with a single access.

    A microservices architecture decouples applications into separate services, each responsible for specific resources that make up the smaller applications, which can be secured and provisioned over a network.

    Segmenting the service landscape avoids one cascading defect affecting all applications. It also enables more control. “By decoupling from monolith to microservices, we can apply fine-grained security permissions and rules to determine what’s possible in the context of the application,” Palladino said.

    Microservices ⮕ Service Mesh

    Implementing these permissions is where a service mesh comes in. The application teams often build security, yet it’s generally bad practice to build your own cybersecurity. Security for microservices requires high expertise, and standardizing connectivity between various microservices can easily result in fragmented security implementations.

    Instead of building your own security infrastructure, Palladino recommended utilizing service mesh. By using service mesh as a control plane for microservices, platform architects can specify specific rules and attributes to generate an identity on a per-service basis.

    Service mesh also removes the burden of networking from developers, enabling them to focus more on their core logic. “Application teams become consumers of connectivity, as opposed to the makers to this connectivity,” Palladino noted.

    Identity for Services

    In the same way enterprises create identity profiles for humans, Palladino said we must extend the same model to services, pairing each service with authorization scopes. “Services must provide their own identity,” he noted. “Based on that, they can do certain operations in their infrastructure.”

    When we think of identity for services, attributes such as geographic location and certificates come to mind. These and other traits could make up a microservice’s authorization scopes. Utilizing a service mesh, architects could assign such attributes to each workload and service to determine what operations are allowed or what requests can be made to what other services.

    For example, services from one region could be designed not to consume services from another region. Such attribute-based control for microservices is a novel, well-needed strategy, Palladino said. “It was crazy this was not in place before; not having zero trust is like having all your money in an open vault.”

    Best Practices for Service Identity

    Knowing that we want to implement identity-based security for microservices, where are some best practices? Palladino recommended:

    • Automate some attributes: A service mesh could automatically assign identity attributes upon creation, such as cloud zone or geographic boundaries.
    • Provide a way to manage attributes: A dashboard would be necessary for DevOps engineers to create, edit or delete service-level attributes.
    • Governance: Architects must manage the control plane and create identities for the right parties to access this capability.
    • Secure encryption and Mutual TLS: Teams will require a system for rotating and expiring certificates for hundreds of thousands of workloads running across an organization, noted Palladino.
    • Standardize: Secure Production Identity Framework For Everyone (SPIFFE), a CNCF incubating project, is a good framework for service identity, making it operable with other cloud-native projects.
    • Play nice with CI/CD: Attribute creation could be hooked into an underlying CI/CD process to occur when deploying products automatically.

    Once microservices have identities, the services they interact with can determine what actions they are authorized to perform. This limits machine-to-machine communication to allowed operations.

    Zero-Trust Security With One Click

    Many teams have transitioned to a microservices architecture to deploy more often, be more distributed, decoupled or improve overall reliability. While these innovations are largely positive, Palladino cautioned against innovation without the proper security forethought: “It’s silly to transition to a microservices strategy without thinking about how these services are secured,” he said.

    Teams will also be running microservices on various environments—VMs, cloud or physical data centers. Thus, considering microservice-to-microservice communication, how do we route traffic between such environments? How do we observe metrics and logs? Palladino believes an agnostic service mesh that supports multi-region, multi-zone, multi-cloud and hybrid modes is the answer to these dilemmas.

    If used correctly, service mesh can plug security gaps for internal service-to-service communication, thus obfuscating high-value resources, such as financial services data, from nefarious players. The key to solving the service identity challenge is a usable dashboard. As Palladino described, “Zero trust security with one click.”