devopsy.kinsta.cloud

Category: DevOps Toolbox

  • Driving Software Delivery Automation With AI/ML

    Driving Software Delivery Automation With AI/ML

    What if we could use an artificial intelligence (AI) engine to predict the next lines of software code to build? What if machine learning (ML) could trigger the creation of automated test cases to validate our solutions?

    Automation has been a focus for the information and communications technology (ICT) industry for years, and there are some lessons that could apply to the software developer community, too, as well as other verticals. Today, more streamlined execution of operational and software delivery processes, such as continuous integration, continuous delivery and continuous deployment (CI/CD) in app development, continue to drive new levels of cost reductions, allowing products to get to market and update faster. 

    However, with the increased amount of data comes more complex ecosystems and the need for the next level of automation. Hyperautomation, or the use of AI/ML to further streamline processes, could build on these CI/CD processes, revolutionize the way multiple industries deliver software and optimize operational processes.

    Today’s Network Complexities and Challenges

    In the telecommunications industry, disruptions from hyperscalers and other innovative players, as well as increasing levels of investment in 5G technology, are forcing the hand of communications service providers (CSPs). This is causing them to turn to automation to improve their levels of operational efficiency and innovation to keep up with the need to push frequent updates. 

    However, challenges remain in the form of:

    • Total cost of ownership and operating costs: With more frequent needs for upgrading existing networks, additional levels of automation including AI/ML and hyperautomation are required to manage and maintain today’s complex networks. 
    • Time to market: In an increasingly competitive market, the CSP’s ability to react to market needs and/or new marketing ideas and to rapidly launch those needs and ideas to their end users and enterprise customers is a key competitive differentiator. 
    • Customer experience: Digitalizing business processes, providing an increased level of self-service and seeking ways to optimize customers’ overall journey remains an important need.
    • Employee productivity: In an effort to increase revenue, more efficiently drive their networks and improve end user customer experience, CSPs also need to increase employee productivity. They can do this by engaging their employees to solve bigger problems. As a result, the simplest and most repetitive low-level tasks can be automated.

    The Power of AI and ML

    The automation journey started with breaking down complex processes into smaller parts and automating discrete parts using scripts, runbook automation (RBA) and/or robotic process automation (RPA). 

    With the introduction of AI emulating human intelligence through mechanical and computational processes and ML using algorithms that learn from data to make predictions and decisions, CSPs can now bring automation and hyperautomation to the next level. 

    The potential use cases for the delivery and maintenance of telco software solutions are unlimited, and many of these lessons can be applied to other sectors. Some examples include:

    Autonomous Fault and Anomaly Detection for Cloud-Native Solutions

    Today’s network solutions are deployed on containers and microservices. The identification of issues and assurance of service quality becomes increasingly challenging with thousands of workloads running in parallel. Container failure can cause severe service disruptions. In fact, a single service failure or abnormal behavior could have a butterfly effect on the overall system and negatively affect customer satisfaction. 

    A proactive, AI-based monitoring solution can continuously search for anomalous behavior in the environment. Different container metrics and logs can automatically analyze and detect problems proactively and help find the right corrective actions for the application workflows. This is based on identifying anomalies in the container orchestration metrics related to pods, workloads and namespace. These techniques enable a more reliable and predictable application runtime environment with higher service quality and faster identification and resolution of system faults and, ultimately, a better overall customer experience

    Machine Learning-Triggered Closed-Loop Automation

    Large clusters of compute, storage and networking resources are managed throughout a data center to enable the deployment of custom workloads. Due to the inherent nature of applications running on the virtualized and cloud infrastructure, there are usage highs and lows which need constant infrastructure component deployment or reconfiguration of the cluster to ensure cost-efficient service availability. 

    A machine learning-based solution can detect anomalies in the infrastructure. Specifically, with closed-loop automation, enterprises can bring in a layer of intelligence over the current setup to monitor application performance and infrastructure metrics and identify anomalies that require immediate system attention. If those are found, they can trigger the automated addition of capacity to the infrastructure. When anomalies have subsided and once the utilization is capable of being handled by the original capacity, the solution can consistently monitor metrics to roll back the virtual or cloud-native functions. This allows for an efficient and fully automated use of network capacity, minimizing human oversight.

    Smart Code Recommendations

    One of the most clear-cut ways tools can reduce the effort of software development for programmers is through code completion. AI-powered code completion is a convenient way to access descriptions of functions and their parameter lists. These solutions also speed up software development by reducing keyboard input and the necessity of name memorization. They also allow users to refer less frequently to external documentation, as interactive documentation appears dynamically in the form of tooltips. AI-powered code completion can also use an automatically generated in-memory database of classes, variable names and other constructs or references based on machine learning to open code repositories.

    AI-Driven Test Recommendation

    Significant time is spent understanding solution requirements and identifying critical scenarios for testing. This is usually followed by a manual test plan creation and execution. To solve this, AI/ML techniques can be applied to identify what tests will need to be done during each cycle of the software development stage, including automating the analysis of required documentation. This can be used to increase overall test-cycle time and improve productivity. 

    AI-Enabled Test Automation

    API testing is critical for validating application logic and needs to go beyond the GUI layer to validate the core of an application. This implies complexity because the protocols are usually difficult to understand, the API inventory is usually large and sequencing of the API calls is critical to a successful test specification and execution. With AI/ML and natural language processing techniques, the analysis of the solution test plan can be automated to obtain recommendations on what test frameworks to utilize, identify and suggest the parameters for each test case, identify the API call sequence and automate the development of test cases based on reusable functions. This can increase overall quality of service, simplify change management, improve the productivity of the testing team and offer more reliable test cases.

    Test Failure Analytics

    During execution of automated test cases, multiple failures can be observed in various logs. Manually analyzing the logs and correlating them with the application issues can be time-consuming. Also, the identification of unreliable tests impacting system stability is complex. 

    AI/ML techniques such as vectorization, K-means clustering or topic modeling can be applied to automate the collection of test logs across applications and label failures including the type of cluster failed tests. The test subject matter expert (SME) is provided with a failure analysis report summarizing issues and problems automatically from many logs across applications, simplifying the identification of issues in test automation areas or scripts. These techniques can significantly increase test productivity and overall troubleshooting time.

    Accelerated Root Cause Identification and Analysis (RCI&A)

    When a problem occurs in a telco network, it tends to cause a ripple effect, affecting other integrated services and creating an alert storm. In such scenarios, root cause identification can become complex and cumbersome. Event aggregation and synchronization are needed to reduce environment complexity and variance. 

    Machine learning weighted analysis techniques can be used to determine the influence individual IT metrics have in affecting the overall health and performance of hosted applications and services so that pinpointing the root cause of any service impact in real-time can become easier for the team operating the network. Efficient AI/ML techniques for accelerated RCI&A can lead to significant timeline reductions. 

    AI/ML Driving Operational Innovation and Market Agility

    In telco, there is great potential for AI/ML-driven use cases to further improve the cloud-native journey and enable an IT-like software life cycle management process for the solutions of the future.

    More broadly, highly automated and self-healing networks will drive higher levels of efficiency and customer satisfaction. This includes faster testing with less resources; more efficient utilization of network resources and cloud infrastructure; faster time-to-market with more software releases per year with similar capacity as well as an increased quality for faster and more agile monetization of the CSPs capabilities.

  • The Power and Potential of Stored Procedures

    The Power and Potential of Stored Procedures

    One way of saving time when writing code is with stored procedures; prepared SQL code that you can save so the code can be reused over and over again. Stored procedures are a powerful vehicle for speeding up your code and giving you gains in performance. 

    Stored procedures have many benefits—they can be easily modified, are reusable and they possess the ability to automate a task that requires multiple SQL statements. Another advantage of stored procedures is the ability to access or modify data in a database.

    Before incorporating stored procedures, it’s important to know exactly what a stored procedure is, how it differentiates itself, how to create a stored procedure and how to use them in embedded analytics.

    Performing Tasks With Stored Procedures

    A stored procedure is defined as a set of SQL statements and other PL/SQL constructs that are stored in a relational database management system as a group that you can save and reuse. A stored procedure can be comprised of multiple SQL statements like SELECT, INSERT, UPDATE or DELETE. These statements run as a unit and are used to solve a specific problem or perform a set of related tasks.

    Stored procedures are among the most frequently used and most popular database subjects. Using stored procedures can help simplify and speed up the execution of SQL queries.

    Stored procedures can reduce network traffic between servers and clients, for example. This is a result of the commands being executed as a single batch of code rather than multiple. Rather than every single line of code being sent for execution individually, only the call to execute the stored procedure is sent to the server.

    The other benefits of using stored procedures include maintainability, performance and efficiency and complex business logic. And maintaining one stored procedure is easier on the server than maintaining copies on multiple machines. Procedure scripts are stored in one location and they can be freely used in other applications, too. In the event of a definition change, only the stored procedure will be affected—not the application that calls it.

    Advantages of Stored Procedures

    Once you compile a stored procedure, it can then be used many times. This can save a lot of time writing, especially if you are doing ad-hoc queries. This also gives you a slight performance edge on the first execution of a stored procedure over the first execution of an ad-hoc query.

    A stored procedure is ultimately a function call on a database. You may include IF statements, LOOPS, security checks and more to ensure the correct data is being sent back to the call of the stored procedure.

    Another advantage of stored procedures is that they are easily modified. Simply change one or more statements in a stored procedure with the ALTER PROCEDURE command and it quickly updates the results sent back to the calling app. Unlike a middle-tier component which includes business logic that would need a potentially costly re-deployment, a quick change to a stored procedure takes seconds.

    Another benefit of using stored procedures is the security it provides to its data source. The security of an application can be enhanced when using stored procedures by restricting unauthorized access. You may grant someone access to a procedure that updates the database table without allowing access to the table itself. You may also use stored procedures to “hide” the underlying table names from the end user of the stored procedure. Since the caller of the stored procedure has the name of the stored procedure and not the name of the tables used in the stored procedure, they cannot directly access the tables with your data.

    Best Practices For Creating a Stored Procedure

    Creating a new stored procedure is fairly straightforward. Here are the three basic steps:

    1. Within the SQL Server Object Explorer, right-click on the Stored Procedures node under Programmability.

    2. Within the New Stored Procedure Template, modify the CREATE PROCEDURE statement to include the tables, parameters and query options to return the desired data from the stored procedure.

    3. Within the Toolbar, click the Execute button to compile and save your stored procedure.

    There are multiple tools you can use to create stored procedures. Using the interactive SQL Server Management Studio is the easiest, as it gives you an easy way to start connecting to many different types of database servers, including Azure and SQL Server Analysis Services.

    Behind the Scenes in Embedded Analytics

    More organizations are using data to predict future outcomes and stay competitive, offering their customers what they want when and where they want.

    Embedded analytics business intelligence software holds the key to unlocking the power of data. Stored procedures can be incorporated into embedded predictive models and continuous computational language code for data streaming scenarios.

    To create a new visualization on a database that supports stored procedures, first review the list of stored procedures that are available. You will treat the stored procedure you select just like any other data source. The only difference is a parameterized stored procedure, which you can use in a Filter by passing variables to the stored procedure.

    Some embedded analytics platforms, for example, offer complete programmatic control over stored procedures and parameters via code. To pass a stored procedure name, as well as any required parameters, use the ChangeVisualizationDataSourceItemAsync function. This is an example of passing a stored procedure named CustOrderHist and passing the CustomerID parameter in the Reveal platform.

    The Power of Stored Procedures

    Stored procedures are a powerful tool to query a database and return results to a calling app. They are secure, offer performance advantages and are easy to maintain and modify. Stored procedures can be a good avenue for simplifying the way you present data and deal with parameters in your application code versus using error-prone and lengthy ad-hoc queries. As you develop applications, you may want to consider incorporating useful stored procedures in your next project or visualization database.

  • Oracle Adds AI Services to Cloud Portfolio

    Oracle Adds AI Services to Cloud Portfolio

    Oracle today unfurled Oracle Cloud Infrastructure (OCI) AI services, a collection of services that make it easier for developers to use application programming interfaces (APIs) to invoke a wide range of services based on algorithms trained by Oracle on their behalf.

    Elad Ziklik, vice president of product management for AI services and data science at Oracle, said the primary goal is to offload the data engineering tasks associated with machine learning operations (MLOps) from organizations that simply want their developers to include AI capabilities within their applications without having to necessarily hire and then retain a dedicated data science team.

    However, organizations also have the option of uploading their own data into OCI AI services to retrain the models provided by Oracle, he noted.

    Ultimate, the goal is to make OCI AI services a simple extension of DevOps workflows by making it simpler to employ AI models; these models become just another software artifact that can be stored and updated in a Git repository alongside all the other artifacts managed by a DevOps team, said Ziklik. In effect, Oracle wants to make AI ‘boring’ in the sense that these capabilities can now be readily invoked by any development team, he added.

    The OCA AI services include OCI Language, a text analysis capability that enables developers to add sentiment analysis, key-phrase extraction, text classification, named entity recognition and other similar capabilities to their applications, and OCI Speech, which converts file-based audio data containing human speech into text transcriptions.

    Oracle has also included OCI Vision for image recognition and document analysis tasks, OCI Anomaly Detection to create models that flag irregularities in processes, OCI Forecasting for tracking business metrics over time and OCI Data Labeling for creating datasets to train AI models.

    These services are designed to address the most critical AI requirements of the average enterprise rather than attempting to outperform on a set of AI benchmarks that don’t reflect the requirements of business development teams, said Ziklik.

    In addition, Ziklik noted Oracle has also made it easier to address AI compliance requirements that will eventually require organizations to document precisely how their AI models work.

    At some point soon, just about every application will be invoking AI capabilities to one degree or another. That shift creates a major workflow challenge for DevOps teams. AI models are typically created by data science teams that typically don’t successfully deploy an AI model in a production environment more than a handful of times a year. Those AI models, however, need to be integrated with applications that, in many cases, are being continuously updated via DevOps workflows.

    Oracle said that goal becomes easier to achieve using models it creates versus allowing data science teams to build out a set of custom MLOps workflows on platforms that then need to be maintained by IT professionals. Instead of having to set up, for example, a feature store to track and manage AI models, the models created using OCI AI can be stored in an existing Git repository.

    Naturally, each organization will need to determine the degree to which they will converge MLOps and DevOps best practices. Oracle, however, is betting that the simplest way to achieve that goal is via AI models surfaced via a cloud service that developers can readily invoke whenever required.

  • FinOps 101: Demystifying Cloud Financial Operations for DevOps

    FinOps 101: Demystifying Cloud Financial Operations for DevOps

    In an ideal world, all organizations would have a dedicated FinOps team—or at least a single specialist who could work to ensure that the cloud was operating in a financially savvy way. Of course, in reality, most organizations are strapped for skills and are operating lean teams where responsibilities are spread thickly across employees. In that case, DevOps professionals are likely to get left holding the FinOps bag and tasked with implementing FinOps best practices into their everyday cloud operations. If that sounds familiar—here’s your guide to getting started.

    The Cultural Shift of Considering FinOps

    In many ways, it’s not that great a leap for DevOps to start thinking about FinOps. After all, the underlying goals of both strategies are to optimize, improve and streamline operations. The difference is that DevOps is focused on software development and technical skills, while FinOps looks at the financial costs of operating on the cloud.

    The biggest challenge is that sometimes these goals may be at odds with each other. If DevOps teams are tasked with getting new software releases and fixes out of the gate as quickly as possible, that can often come at the expense of the most cost-effective solution. Think, for example, about AWS on-demand instances in the cloud. It’s arguably more FinOps-friendly to look for reserved instances, savings plans and committed use discounts and to put into place right-sizing opportunities where possible. However, with the pressure on DevOps teams to keep up an agile release cycle and offer quick fixes, anything that stands in the way of their flexibility and freedom can feel worth the price tag.

    As a result, it can be a true paradigm shift for engineers and DevOps teams to start thinking about FinOps first, especially if they already feel that their plate is full. Here are some best practices that can help to smooth over any bumps in the road.

    Make Engineers Accountable

    Don’t simply add new stakeholders who are responsible for cost management and FinOps, as they will quickly become adversaries of your engineers. Instead, ensure that your technical teams and developers realize that they are responsible for their cloud costs. They need to know what they are using and they need to codify and document how they are looking to use their budget more effectively.

    Of course, the business has a role in this too, showing engineers best practices for FinOps, such as shutting down idle resources, right-sizing instances or making smart architecture decisions. 

    Explain That FinOps Isn’t a Cost-Cutting Initiative

    If DevOps teams have been given the equivalent of a blank check thus far, it can be a tough sell to start tightening the purse strings. Explain to DevOps teams that you aren’t looking to cut their budget. In fact, you might even be increasing it!

    FinOps is all about smarter financial operations, so if engineers can show that they can save money in one direction, they can channel that to new features, technologies and tools that make them happy. Giving DevOps teams control over FinOps decisions offers more autonomy and control—and that’s a great thing.

    Ease the Mental Load With Low-Hanging Fruit

    Once DevOps teams get much-needed visibility over their cloud usage and realize that they are gaining responsibility and autonomy over their behavior, you want to guide them to recognize where they can make changes. Challenge all your DevOps stakeholders with the same KPIs, and see who can come up with innovative ways to incorporate cost savings into business-as-usual. Some KPIs that could be a great start include:

    • Coverage: Use of reserved instances, savings plans, spot instances or committed use discounts.
    • Right-sizing: The opportunity value of downsizing or terminating instances that are idle or underutilized.
    • Orphan/idle status: Tracking the percentage of orphaned EBS volumes or snapshots; you can drill down into a specific time period, such as over one month.
    • Forecast metrics: Consider using the variance between a forecasted bill and the actual costs and aim to reduce this as much as possible.
    • Incorrect assignation: This would highlight oversized or incorrect instance types, S3 or EBS storage on the wrong tier, unattached elastic IPs or poorly tagged/ untagged resources.
    • Peak hours: What percentage of development resources are running during peak hours or running when they could be shut down; for example, on weekends?

    Automation Incorporates FinOps Best Practices in a Hands-Off Way

    If this all sounds like a full-time job in and of itself—you’re not wrong. You’d be forgiven for wondering when DevOps teams are meant to actually develop and build! To keep engineers and technical teams productive, make sure that you’re offering the right tools and technology to lighten the load where possible, especially where automation and AI can do a better job than a room full of DevOps or FinOps experts.

    One example is automating the purchasing and selling of reserved instances in real-time, so that you’re never left buying on demand. Another is automating the removal or rightsizing of idle or underutilized resources, respectively. When your DevOps teams see that you’re offering the technology to make FinOps simple, they are much more likely to do their part elsewhere.

    DevOps and FinOps

    In recent years, security teams have championed the idea of DevOps ‘shifting left’ on security best practices to get security included in the process of agile software development during the build stages.

    Well, you heard it here first: It’s time for a similar shift for finance and cost optimization.

    Just as engineers can no longer get away with ignoring security in the way that they work, financial optimization needs to be front and center of DevOps teams’ day jobs. For this to work and for DevOps to incorporate FinOps goals seamlessly, businesses need to champion visibility for cloud usage, a culture of cost-optimization and a strong foundation of automation that bakes best practices into everyday work.

  • Couchbase Revamps DBaaS Based on Document Database

    Couchbase Revamps DBaaS Based on Document Database

    Couchbase Inc. has unfurled a revamped database-as-a-service (DBaaS) platform dubbed Couchbase Capella hosted on the Amazon Web Services (AWS) cloud.

    Scott Anderson, senior vice president for product management for Couchbase, said that as application development continues to evolve, it’s becoming apparent more organizations prefer to have providers of databases manage those platforms on their behalf. That approach frees up database administrators (DBAs) to spend more of their time building and deploying applications, he noted.

    Couchbase Capella is based on Couchbase Server 7, which added multi-statement SQL transactions capabilities to the document database, based on a JavaScript Object Notation (JSON) format, to make it possible to also run in microseconds transaction processing applications that previously would have required a relational database to process SQL ACID transactions.

    That release of Couchbase Server 7 also added schema and table-like organizing structures, called “scopes and collections,” within the schemaless database to make it possible to add a table while transactions are running.

    As a document database Couchbase has gained traction mainly thanks to developers who have opted to employ a database they could download and deploy themselves to build applications instead of requiring a DBA to set up a relational database. However, as the number of instances of a document database proliferates, a need to manage them across fleets of servers and cloud services has emerged.

    The challenge then becomes determining whether to hire a dedicated administer or employ a DBaaS managed by a vendor. Many organizations are opting for the DBaaS approach because it eliminates the need to hire a full-time administrator. Once an organization opts for the DBaaS approach, however, it next needs to integrate that service within its DevOps workflow to build those applications.

    There is, of course, no shortage of options when it comes to DBaaS platforms. The global cloud database and DBaaS market size is expected to reach $26 billion by 2025, according to a report from Adroit Market Research. Overall, the as-a-service market reached a record $13.4 billion in the third quarter, up 55% from prior year, according to a report published by Information Services Group (ISG).

    It’s not clear to what degree organizations will ultimately wind up relying on as-a-service platforms, but since the start of the COVID-19 pandemic, the shift to this approach to consuming IT resources has increased significantly. As a result, the way internal IT teams are organized is shifting as lower-level tasks are either automated or handled by an external IT team that works for a vendor.

    Regardless of approach, the amount of data being created and analyzed within organizations is only going to grow. Organizations will need to determine to what degree it makes sense for them to manage that data versus relying on an external services provider. The challenge, of course, when relying on an external service provider isn’t just technical; it introduces a new cultural dynamic that takes some time for many IT teams to appreciate fully.

  • Salt Security Adds Support for GraphQL APIs

    Salt Security Adds Support for GraphQL APIs

    Salt Security has extended its platform for securing application programming interfaces (APIs) to include support for APIs built using GraphQL.

    GraphQL is an open source data query and manipulation language for APIs that was originally developed by Facebook. It provides a more efficient approach to querying data than the REST APIs that are widely used today.

    Elad Koren, chief product officer for Salt Security, said adding support for GraphQL alongside existing support for REST APIs to the Salt Security API Protection platform will make it possible for security teams to discover those APIs, mitigate data exposure, stop attacks and eliminate vulnerabilities. The Salt platform parses the complex structure of each GraphQL query to identify unique object entities that are then used to create a complete inventory of the GraphQL APIs being employed.

    APIs built using GraphQL are more challenging to secure because of the unique call and response formats, noted Koren. Cybercriminals can also, for example, leverage GraphQL capabilities such as nested queries and query batching, to launch distributed denial of service (DDoS) attacks, he said.

    Many developers will tend to assume that, because GraphQL is relatively new, most cybercriminals are primarily focused on exploiting REST APIs. However, as more developers employ GraphQL, it’s only a matter of time before cybercriminals begin to find ways to exploit APIs that may not be as secure as the REST APIs that many organizations already know how to secure.

    The Salt Security platform itself is based on a big data engine that employs machine learning algorithms and other forms of artificial intelligence (AI) to identify attacks in real-time. It creates a baseline to identify legitimate system behavior while preventing cybercriminals from using penetration testing tools to perform reconnaissance.

    From a DevOps perspective, the Salt Security API Protection Platform integrates with tools such as Jira and Slack to ensure that remediation details are routed to the right development team in addition to helping track tickets to ensure remediation fixes are implemented. It also can be integrated with security information event management (SIEM) platforms from vendors such as Splunk and Sumo Logic to enable incident response for both DevSecOps and security operations teams.

    Most IT teams will not be replacing REST APIs with GraphQL-based APIs overnight. However, there will soon be enough APIs based on GraphQL to warrant additional security measures. Less clear is to what degree those security measures will be put in place by DevSecOps teams versus a security operations team.

    One way or another, however, the number of APIs that need to be secured is accelerating rapidly as organizations roll out a new generation of microservices-based applications that drive mission-critical digital business transformation initiatives. Each microservice has its own API that needs to be built and then secured. Unfortunately, most APIs are built by developers that don’t always have the greatest appreciation for security. As such, while more responsibility for API security is shifting left toward developers, many security teams would be well advised to make certain every API being deployed is fully secure.

  • Postman Survey Shows Focus on Building and Deploying APIs

    Postman Survey Shows Focus on Building and Deploying APIs

    A survey of more than 28,000 developers and application programming interface (API) professionals published today by Postman found nearly half of respondents (49%) are now spending the majority of their time developing APIs. More than a quarter (28%) now spend 20 or more hours per week working with APIs.

    Kin Lane, chief evangelist for Postman, a provider of a platform for building APIs, said that shift reflects an increase in software development and integration activity that began at the start of the COVID-19 pandemic. Almost two-thirds of respondents (63%) said the pandemic somewhat or significantly affected their work, with almost half of those respondents noting that changes made with their organization were enabled and supported by APIs.

    Additionally, a full 94% of respondents said the investment of time and resources into APIs will increase or stay the same for the next 12 months. The biggest obstacle to creating APIs is lack of time (45%) while the biggest obstacle to consuming an API is lack of documentation (55%). In terms of API goals, the survey found that quality (80%), agility (66%) and reliability (65%) are the top three goals among survey respondents.

    However, the survey also noted only a third of respondents said they could build and deploy an API in anywhere from a single day to a week, with most requiring one week to a month. About one-third also said they deploy APIs to production between once per week and once per month. More than 80% of respondents noted that less than 25% of their changes fail while a similar percentage said they can recover in less than a day with a third reporting they fully recover in less than an hour.

    The Postman survey also found that more than three-quarters of respondents (76%) have less than five years of experience with API development. Only half of respondents indicated that they agree or strongly agree that the leadership of their API team has a clear sense of where their API practice should be in five years. However, microservices (58%) and Kubernetes (50%), containers (46%), serverless architecture (44%) and GraphQL (35%) were ranked among the technologies that respondents were most excited about heading into 2022.

    On the plus side, more than 60% of respondents felt their API leadership and governance teams demonstrated a “willingness to change” as well as duly considering “the interests and opinions of employees.” When it comes to regularly providing feedback and process, more than half of leadership and governance teams (53%) scored well.

    The top two ways API teams collaborate is via a dedicated collaboration platform (50%) or by publishing API artifacts to repositories such as GitLab, GitHub and Bitbucket (45%), according to the Postman survey. The most widely used DevOps tools are Jenkins (37%), Azure DevOps (26%), GitHub Actions (25%), AWS DevOps (23%) and GitLab Pipelines (22%).

    In terms of deploying APIs, continuous integration/continuous delivery (CI/CD) platforms are the most widely employed (57%), followed by deploying APIs in the cloud (38%), frameworks (34%) and custom-built deployment methods (27%).

    Just over 70% of respondents ranked security and performance as the top factors considered when integrating with an API. When producing an API, however, the top factors are integration with internal systems, integration with external systems, adding functionality to an internal system and enhancing functionality for an end customer. Similarly, when consuming an API the top factors are integration with external systems, adding functionality for end customers, adding functionality to internal systems and integration between internal systems.

    In general, Lane noted that, as more APIs are developed and deployed to drive various digital business transformation initiatives, organizations will need to pay more attention to API discovery and management. There’s a tendency for different teams within organizations to not only create redundant APIs but also forget about so-called zombie APIs that are no longer supported but can still be exploited by cybercriminals.

    Overall, Lane said the survey makes it clear more organizations are pursuing an API-first approach to digital transformation. The challenge now is managing all the APIs that can quickly become too much of a good thing.

  • What 30 Years of Linux Taught the Software Industry

    What 30 Years of Linux Taught the Software Industry

    Linux has become the largest collaborative development project in the history of computing over the last 30 years. Reflecting on what made this possible and how its open source philosophy finally imposed itself in the industry can offer software vendors valuable lessons from this amazing success story.

    The web may not have reached full adulthood yet, but it has already crafted its own mythology.

    August 25, 1991: Linus Torvalds, a 21-year-old university student from Finland, writes a post to a Usenet group: “Hello everybody out there using minix — I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386 (486) AT clones […]”. A few weeks later, the project, which will eventually be known as Linux, is published for the first time.

    This is the starting point of an epic that few could have foreseen.

    Fast-forward 30 years and the Linux kernel isn’t only running on most of the web servers and smartphones around the globe, but it also supports virtually all of the much more recent cloud infrastructure. Without open source programs like Linux, cloud computing wouldn’t have happened.

    Among the major factors that propelled Linux to success is security. Today, the largest software companies in the world are taking open source security to new levels, but the Linux project was one of the first to emphasize this.

    How Linux Became the Backbone of the Modern IT World

    Brief History

    Open source predates the Linux project by many years and is arguably as old as software itself. Yet, it is the success of the latter that propelled this movement in the 1990s. When it was first submitted for contribution in 1991 by Torvalds, the Linux kernel was the GNU project’s ‘missing link’ to a completely free software operating system, which could be distributed and even sold without restrictions. In the following years, and as the project started to incorporate proprietary licensed components and grow in popularity, a clarification on the meaning of “free software” became necessary.

    This led to the coining of the term “open source” as we use it today, thanks in part to Eric Raymond’s seminal paper The Cathedral and the Bazaar, a “reflective analysis of the hacker community and free software principles.” Open source was chosen to qualify software in which the source code is available to the general public for use or modification from its original design, depending on the terms of the license. People may then download, modify and publish their version of source code (fork) back to the community.

    Open source projects started gaining traction in the late nineties thanks to the popularity of software like Apache HTTP Server, MySQL and PHP to run the first dynamic websites on the internet.

    Facts and Figures

    Today, not only is Linux powering most of the digital era, but open source has become the leading model for how we build and ship software. Though most people don’t realize it, much of the technology we rely on every day runs on free and open source software (FOSS). Phones, cars, planes and even many cutting-edge artificial intelligence programs use open source software. According to the Linux Foundation, 96.3% of the world’s top one million servers run on Linux and 95% of all cloud infrastructure operates on it. Other infrastructure also relies on open source: 70% of global mobile subscribers use devices running on networks built using ONAP (Open Network Automation Platform).

    Linux adoption is very high in professional IT, where it’s become a de facto standard, especially with the advent of the cloud era. In fact, 83.1% of developers said Linux is the platform they prefer to work on. This success is due, in large part, to the community that contributed to its source code since its creation: More than 15,000 developers from more than 1,500 companies. Linux went on to become, arguably, the biggest success story of the free software movement, proving that open source could lead to the creation of software as powerful as any sold by a corporation.

    The Linux Foundation, a non-profit technology consortium founded in 2000 to support the collaborative development of Linux and OS software projects, is itself a big success. It now has more than 100 projects under its umbrella, spread across technology sectors like artificial intelligence, autonomous vehicles, networking and security. Several subset foundations have also emerged over the years, including the Cloud Foundry Foundation, the influential Cloud Native Computing Foundation, and the recently announced Open Source Security Foundation. The Foundation estimates the total shared value created from the collective contributions of its community to a whopping $54.1 billion.

    All these achievements may not have been possible without the embrace of open source by the enterprise world, which may represent its biggest win.

    Enterprise Adoption

    Companies began to realize that many open source projects were easier and cheaper to implement than asking their developers to build the basic pieces of an internet business over and over again from scratch.

    Twenty years ago, most businesses ran atop proprietary software from Microsoft, Oracle and IBM, and the idea of collaborating on big software projects might have sounded laughable to them. Today, these companies, along with relative newcomers such as Google, Facebook and Amazon, are not only employing thousands of full-time contributors to work on open source projects like Linux, they also regularly choose to open source some of their state-of-the-art projects; from Google Brain’s machine learning platform TensorFlow and container orchestration platform Kubernetes to Facebook’s React.

    There’s no question that open source software created a new wave of business opportunities. As more companies took an interest in open source projects, they realized they didn’t necessarily have the in-house expertise to manage those projects themselves and turned to startups and larger companies for help.

    Even Microsoft, which famously warred against the very concept of Linux for nearly a decade, made a strategic shift to embrace open source in the 2010s, led by CEO Satya Nadella. The IT giant finally joined the Linux Foundation in 2016 and acquired GitHub, the largest host for open source projects, two years later. It has since become one of the biggest sponsors of open source projects.

    As a consequence, the stakes have been raised for open source software, which is the engine powering the shift toward the cloud for virtually every company. In this context, security is becoming a topic of the utmost importance, and the commitment to secure the open source ecosystem is growing fast.

    Setting a Standard for Security and Trust

    Open Source Security Issues

    Following the OSS adoption boom, the sustainability, stability and security of these software packages is now a major concern for every company that uses them.

    The Census II report on structural and security complexities in the modern-day supply chain “where open source is pervasive but not always understood” revealed two concerning trends that could make FOSS more vulnerable to security breaches. First, the report said it is common to see popular packages published under individual developers’ accounts, raising the issue of security and reliability. Second, it is very common to see outdated versions of open source programs in use, meaning they contain fewer security patches.

    The 2021 OSSRA report agrees: “98% of the codebases audited over the past year contain at least one open source component, with open source comprising 75% of the code overall.” The report also noted that 85% of the audited codebases contained components “more than four years out of date”.

    This highlights the mounting security risk posed by “unmanaged” open source: “84% of audited codebases containing open source components with known security vulnerabilities, up from 75% the previous year. Similarly, 60% of the codebases contained high-risk vulnerabilities, compared to 49% just 12 months prior.” Not only is the security posture affected, but there are also compliance issues that can arise from unsupervised integration of open source content because licenses can be conflicting or even absent.

    Because large corporations are now a big part of the open source ecosystem, their sponsorship is a welcome source of financing for many people whose work had been done for free until now, yet it may not be enough. The open source community is well-known for its commitment to independence, its sense of belonging and its self-sufficiency, and expecting contributors to voluntarily address security issues is unlikely to succeed.

    This is where the experience of building Linux over 30 years and coordinating the work of thousands of individual contributors may be an example to follow.

    Linux Foundations

    In Linux kernel development, security is taken very seriously. Because it is an underlying layer for so many public and private software ‘bricks’ in the digital world means that any mistake can cost millions to businesses, if not lives. Since the beginning, it has adopted a decentralized development approach with a large number of contributors collaborating continuously. Therefore, it has consolidated a strong peer-reviewing process as the community development effort grew and expanded.

    The last stable release at the time of writing is 5.14, released on August 29th, 2021, only a few days before the 30th birthday of the project. The most important features in the release are security-related: One is intended to help mitigate processor-level vulnerabilities like Spectre and Meltdown and the other concerns system memory protection, which is a primary attack surface to exploit. Each Linux kernel release sees close to 100 new fixes per week committed by individuals and professionals from the likes of Intel, AMD, IBM, Oracle and Samsung.

    With such broad adoption and long history, the Linux project has reached a level of maturity that few, if any, other FOSS projects have seen. The review process and release model have built confidence for numerous downstream vendors. Although the world is not perfect and it is arguably difficult for them to keep up with such a high rate of change, they can at least benefit from strong security enforcement mechanisms and they can adapt their security posture in concordance to their “risk appetite”: Vendors are able to do the calculus of determining how old a kernel they can tolerate exposing users to.

    Pushing the Boundaries of Open Source Security

    Heartbleed and the Fragility of OS Security

    In April 2014, a major security incident affecting the OpenSSL cryptography library; disclosed as “Heartbleed.” The developer who introduced the bug acknowledged that, though he was working on the project with a handful of other engineers:

    “I am responsible for the error because I wrote the code and missed the necessary validation by an oversight. Unfortunately, this mistake also slipped through the review process and therefore made its way into the released version.” OpenSSL, an open source project, is widely used to implement the Transport Layer Security (TLS) protocol. In other words, it’s a fundamental piece used to secure a large part of the web.

    Open source was seen as fundamentally secure for a long time because the more people examine a line of code, the better the chances of spotting any weakness. Additionally, this model prevents “security by obscurity,” whereby the bulk of the protection comes from people not knowing how the security software works—which can result in the whole edifice tumbling down if that confidential information is released or discovered externally.

    This incident was a major turning point for a large share of the biggest web corporations: They realized that many open source technologies underpinning their core operations could not be “assumed to be secure” anymore. Any human error could have huge implications; therefore, a specific effort had to be made to improve the security in this specific space.

    A New Era for Open Source

    As we advance in an era where open source is omnipresent in codebases, tooling, networks and infrastructure and is even in fields other than software, security awareness is starting to take hold. But it needs a lot more work.

    A big part of the challenge, to begin with, is for the industry to understand the scope of the problem.

    Google just announced that it will be committing “$100 million to support third-party foundations that manage open source security priorities and help fix vulnerabilities.”

    The Secure Open Source (SOS) pilot program, run by the Linux Foundation, will reward developers for enhancing the security of critical open source projects that we all depend on.

    In doing so, Google leads the way in enlarging the financial sponsorship of big players like companies and governments — which are increasingly sponsoring open source both directly and indirectly. However, they also recommend that organizations “understand the impact they have on the future of the FOSS ecosystem and follow a few guiding principles.”

    What could these principles look like?

    A Roadmap to Safely Use and Contribute to Open Source

    The Linux Foundation proposed a specific Trust and Security Initiative which describes a collection of eight best practices (with three degrees of maturity) open source teams should use to secure the software they produce as well as by a larger audience to “raise the collective security bar.” Here they are:

    • Clarifying the roles and responsibilities, and making sure everyone is aware of their security responsibilities across the organization.
    • Setting up a security policy for everyone; in other words, a clear north star for all members of the organization.
    • ‘Know your contributors’ is defined as a set of practices to make risk-based decisions on who to trust and fight offensive cyberwarfare techniques, such as the poisoning of upstream code.
    • Locking down the software supply chain: This has become a preferred target as adversaries clearly understood that they can have a bigger and more effective impact with less effort than targeting individual systems.
    • Provide technical security guidance to narrow potential solutions down to the more appropriate ones in terms of security.
    • Deploy security playbooks to define how to do specific security processes, specifically incident response and vulnerability management processes, like creating roles and responsibilities or publishing security policies. This may feel formal, antiquated and old-school but having pre-defined playbooks means that teams can focus on shipping software and not learning how to do security, especially at the least convenient and most stressful time.
    • Develop security testing techniques with automated testing strongly recommended since it scales better, has less friction and less cost to the teams and aligns well to modern continuous delivery pipelines.

    However, the authors of the guide are aware that some major challenges are still facing the industry and, as such, need to be addressed. They mention:

    • The lack of open source security testing tools
    • The fact that open source package distribution is broken
    • The fact that the CVE format for vulnerability disclosure is also broken
    • The lack of a standard for a security build certificate which would allow any consumer to transparently verify that a product or component complies with the announced specifications

    “The types of verification can and should include the use of automated security tools like SAST, DAST and SCA, as well as verification of security processes like the presence of security readmes in repos and that security response emails are valid.”

    A scheme like this could have a significant and lasting effect on the security quality of open source software and the internet at large.

    The Linux project, born 30 years ago, is present in all layers of the modern software stack today. It is used by all the largest server clusters powering the modern web and any business going digital will use it at some point. This unparalleled longevity and success have demonstrated that the open source model was compatible with the requirements of enterprise-grade services and economically viable. Now that open source is all the rage in the software industry, a consensus and action plan on how to ensure the sustainability of this ecosystem becomes urgent. The top priority for businesses that depend on it is to adopt strong application security guidelines, like the ones promoted by the Linux Foundation, which have proven their value.

    One last note on the nature of open source: As businesses are now much more bound by the common use of open source components to build upon, they should not fall into the “tragedy of the commons” trap. This would mean waiting until others take action; for instance, to improve the global software security landscape. This might be one of the biggest challenges confronting our highly collaborative industry.

  • Intel Woos DevOps Teams with API and Processor Advances

    Intel Woos DevOps Teams with API and Processor Advances

    During the online Intel Innovation 2021 event, Intel announced today a unified Developer Zone for accessing application programming interfaces (APIs) that it makes available via a oneAPI toolkit to IT teams that want to more easily invoke a wide range of processor types and classes.

    At the same time, the company also revealed it is creating a set of oneAPI centers of excellence that will make it easier for DevOps teams to consume those APIs. Partners participating in that initiative include Oak Ridge National Laboratory, University of California Berkeley, University of Durham and University of Tennessee. Intel will also make the Intel Graphics Visualization Institutes of Xellence part of this program.

    Next year, Intel is also promising to deliver oneAPI 2022 toolkits that will add 900 additional capabilities including an ability to develop applications for CPUs and graphical processor units using what Intel described as the first unified C++/SYCL/Fortran compiler.

    The company is also planning on making available Data Parallel Python tools in addition to expanding its Advisor accelerator performance modeling tools to include support for VTune Flame Graph to visualize performance hot spots.

    There will also be additional oneAPI integrations to add support for Microsoft Visual Studio Code and the Microsoft Windows Subsystem for Linux 2 platform. Intel also created an Intel oneAPI AI Analytics toolkit to enable out-of-the-box AI development that is supported by workstations running Linux from Dell Technologies, HP and Lenovo. Additionally, Microsoft and Intel have agreed to create a complete data science toolchain to Windows 11 that will become part of the Surface Laptop Studio framework for building applications.

    Greg Lavender, chief technology officer, senior vice president and general manager of the software and advanced technology group at Intel, said it’s critical to make sure that DevOps teams add support for the acceleration engine and extensions that Intel makes available within its processors to the runtime environments they deploy. Otherwise, developers will never be able to routinely invoke those capabilities to build next-generation applications.

    The company also revealed that next-generation Intel Xeon Scalable Processors, code-named Sapphire Rapids, will have additional acceleration engines to both simplify deployments at scale and optimize processors’ core utilization in a way that reduces power consumption. Overall, Intel is promising up to a 30x total performance gain for artificial intelligence (AI) applications running on these processors.

    Other hardware advances announced today include a 12th generation Intel Core family based on 7-nanometer processors and a Mount Evans project through which Intel is collaborating with Google Cloud to build processors based on application-specific integrated circuit (ASIC) chips.

    The company also unveiled an Intel ArcTM Alchemist family of graphics systems on chips (SoCs) that take advantage of machine learning algorithms to optimize performance and touted an Intel Tofino fabric processor (IFP) that adds intelligence to network switching via support for P4 programming tools.

    Finally, the company noted that an Aurora supercomputer, due out in 2022, will exceed two exaflops of peak double-precision compute performance.

    As Intel seeks to regain the processor high ground from rivals such as NVIDIA, ARM and AMD, it’s clear that the future of IT will be defined by a broad range of processor classes that all have unique capabilities. The challenge now is deciding which of those capabilities to actually make available to developers that are largely dependent on the runtime environment that DevOps teams decide to make available.

  • GitHub Expands Developer Productivity Tools Portfolio

    GitHub Expands Developer Productivity Tools Portfolio

    During the online GitHub Universe 2021 conference, GitHub today unveiled a bevy of updates intended to boost developer productivity by making it easier for developers to write code faster and collaborate more easily.

    The GitHub Copilot tool that GitHub developed in collaboration with OpenAI, a research and development lab, to help developers write better code faster is being extended via a technical preview to add support for Java and the integrated development environment (IDE) JetBrains.

    In addition, the Codespaces tools that GitHib makes available to pre-configure development environments is being extended to include support for devcontainer feature composition, the GitHub command line interface (CLI) and REST application programming interfaces (APIs) along with additional access controls.

    GitHub Actions has also been updated to add support for deployments using the OpenID Connect standard in addition to providing access to deployment environments to simplify approvals, improvements to reusable workflows and auto-scaling functionality for self-hosted runners.

    There is also now a Command palette tool in beta to help developers navigate GitHub along with a public beta of a GitHub Issues tool that makes it simpler for developers to filter, sort and group issues and pull requests. Additional features that have been added since GitHub Issues was first previewed include iteration support, reporting and data visualization tools and the ability to employ it across public projects.

    Finally, GitHub is adding, in beta, support for Ruby as a programming language that CodeQL analysis engine can now scan for vulnerabilities along with the ability to customize permissions for accessing GitHub Enterprise Cloud.

    Ryan Salva, vice president of product management for GitHub, said collectively GitHub is committed to making developers more productive by, for example, autocompleting code as developers write. Support for additional programming languages will also be forthcoming in the months ahead. The goal is not to leverage AI to replace developers but, rather, eliminate the drudgery that gets in the way of developers becoming more creative, noted Salva.

    There won’t come a day any time soon when developers won’t have to type to write code, but it is possible to leverage AI to provide a type-ahead capability that reduces coding errors, added Salva.

    The goal should be to enable developers to create more application experiences at a time when more organizations than ever are consuming applications that drive a wide range of digital business process transformation initiatives, said Salva.

    As part of that effort, Salva noted it’s also critical to reduce the amount of friction developers encounter writing code starting with setting up the development environment itself.

    It’s not clear yet what impact all the focus on developer productivity and collaboration will have on the rate at which applications are being built and deployed. However, as developers become more productive, the number of application development projects that can be launched simultaneously should increase. In fact, there may come a day when applications are being built faster than organizations can deploy them without adopting continuous delivery best practices that rely on higher levels of automation to install software.

    Regardless of that application delivery challenge, however, it’s a problem most organizations right now would consider themselves fortunate to have.