Category: Application Performance Management/Monitoring

  • Low-Code Providers Continue Enhancing Their Development Platforms

    Low-Code Providers Continue Enhancing Their Development Platforms

    Enterprise low-code development continues to grow in popularity as software vendors continue to unveil tool enhancements designed to help business users and non-developers build functional applications and business processes.

    The help from low-code platforms (and other platforms that are integrating low-code capabilities) couldn’t come at a better time, as enterprises continue to struggle to digitize operations and dig through their application backlogs.

    Consider open source digital process automation platform Bonitasoft and its recent increased support for low-code development and what the company promises to be accelerated user interface development for more customized user experiences. The platform (including a new Bonita Cloud edition) helps to enhance collaboration between professional developers and so-called citizen developers, according to the company.

    Recently, low-code development platform provider OutSystems announced it is adding new artificial intelligence (AI) and machine learning capabilities into its platform. The company said the enhancements provide customers the ability to help automate the creation of self-service portals, respond to text and voice queries and improve customer service.

    OutSystems said citizen developers now can more easily build communication channels into their apps and better utilize virtual assistants such as Alexa, Google and Siri, as well as common chatbot platforms. The company claimed developers don’t need any specialized development skills or knowledge in data science to use these features. “These new, high-level components package everything organizations need to implement rich use cases from end to end. As a result, many more companies can easily adopt AI and machine learning to positively impact their business,” said António Alegria, OutSystems’ head of artificial intelligence.

    According to the research firm Gartner, by next year, consumers will manage 85% of their relationship with enterprises without having to interact with a human. However, despite the use of AI having tripled over the past four years, two-thirds of organizations have yet to deploy AI in some form, it noted.

    According to Research and Markets, the global low-code development platform market size will reach just over $27 billion by 2022, up from $4 billion in 2017 and a 44% annual growth rate.

    According to the research firm, it is the increased demand in software automation and innovative applications and the ease of development of web and mobile apps that are driving enterprise demand for low-code development. “Moreover, the cloud-based deployment mode is gaining high traction in the market, as it requires less capital investment, helps decrease the operational and maintenance costs, and reduces the management’s efforts,” the research firm said.

    While many analyst market predictions do fall short, as long as low-code platforms reduce costs and help organizations move forward rapidly with their digitization efforts, low-code development is here to stay and will continue to grow.

    — George V. Hulme

  • The Purpose of Microservice Architecture

    The Purpose of Microservice Architecture

    Microservice is an architectural style that can be implemented during complex application development within an organization. The implementation of this architecture allows the application being developed to be highly maintainable and testable, loose, independent during deployment and also strongly organized around business capabilities. In another way, the microservice architecture can be explained quite easily. It is a software development technique that allows for the structuring of an application. The structure of the application being considered will be that of a set of coupled services.

    Any application developed using the microservice architecture is easy to understand, develop and test. More so, the application will have a sort of rigid attitude to degradation in the long run. This is due to the concept adopted during design.

    The design of the microservice architecture ensures that applications are separated into smaller services, states, etc., until they become fine-grained and light in weight. Therefore, the modularity achieved encourages the adoption of methodologies that will allow for the division of labor. These methodologies will aid in the concurrent design, development and testing of the sub-services independently.

    While this architecture is suited to certain scenarios, it (like every other architectural style) has its demerits. Microservice experts advise on the study of the architectural platform before adoption. The implementation of the microservice architecture can be affected by a number of programming languages.

    Within the context of this article, I discuss the purpose of the microservice architecture, and also examine the upside and the reasons why they should be learnt and implemented. I also take a look at particular niches that allow the use of microservice architectures and why they use it.

    Architecture, Examples and Types of Microservices

    The microservice architecture is a major decomposition of the monolithic architectural method. The monolithic architecture had applications developed from the top to bottom and also had the entire development process run as a single unit. The microservice architecture works in contrast to this by enabling modularization. The core of the microservice architecture has requirements that are intensified and made possible by the separation of the primary task into smaller services, in which one service represents just a requirement, function or need.

    Examples of microservice architectures can be found in airline service scenario. If we pick out the flight booking component, we could find out that this primary task could be separated into inventory adjustment, seat allocation, fare calculation, timetable lookup, reward management and customer update.

    Applying Microservices

    Research has it that the majority of the companies that make use of microservices are quite on the high side. This majority includes organizations such as Netflix, eBay, Amazon, PayPal, Twitter, The Guardian, etc. This list also includes the United States Government Digital Service.

    The application of microservices can work with a number of technologies but a number of researches attribute effective usage of this architecture with the cloud environment and the DevOps environment as well. These tools and environments help to manage the operational complexities that affect the swift functionality of the architecture.

    Why Do They Use It?

    Most of the organizations that make use of the microservice architecture did transcend from the monolithic method. This is mostly because the method offers organizations the ability to decompose primary applications into smaller composed pieces. These pieces can then be built independently (which is easier) and also maintained independently as well. Developed pieces have the ability to communicate with each other and work together as well.

    The adoption of microservice architecture provides the management of an organization and everyone involved in the development of the application with a decentralized approach towards building the software. This approach breaks the development process into a series of deploy, rebuild and redeploy.

    The choice of a microservice architecture also grants organizations the ability to implement variant strategies, tools, techniques, etc., for services that are involved. Studies show that organizations that implement this architecture have a high proficiency rating. This is due to the usability of the architecture in independency, deployment, replacement, etc.

    The architecture encourages easy debugging of errors and detection of faults in general. When an error is spotted, the segment from which it originated can be easily traced and the fault can be found shortly after. This, as explained earlier, is because primary tasks are separated into smaller services. Each service can then be fine-grained to simple state. Before the simple state is attained, it is also possible that the service can be separated into a smaller group of services. This option helps the development team actualize a major concept within this architecture, which makes sure that a service handles just one task. Also, the architecture allows for the adoption of management techniques that assist the organization. These management techniques include options such as change management process, continuous software development processes, problem and incident management techniques, etc.

    Benefits to Using Microservices

    The benefits to using the microservice architecture include:

    1. The actualization of scalability when considering customer-focused services.
    2. Independent deployment of services.
    3. The ability to work with more than one development tool (such as  programming languages).
    4. Easing the modularization of tasks within the project as a whole due to the modularization of the application being developed.
    5. Support for a wide range of platforms and devices such as web, mobile, internet of things, etc.

    — Subramonian

  • How to Automate an API Security Program Without Adding Staff

    How to Automate an API Security Program Without Adding Staff

    In today’s information economy, data is a primary raw material and a source of value to both providers and consumers. For many companies, entire business models are built on the exchange of information.

    Consider a ride-sharing business that owns no vehicles of its own. What the company does possess, however, is a database of private vehicle owners who are willing drivers, a roster of people who will gladly pay for a ride and an application that can instantly connect a rider with a driver via an application programming interface (API). The ride-sharing company makes its money by taking a cut of the fare.

    Another example is a company that collects and refines financial data and makes it available to investors who might want to buy stock. Companies such as E*TRADE, Merrill Lynch and Fidelity buy that investment data to make it available to their clients. APIs are at the heart of these important data exchanges.

    How Prominent are APIs?

    The information economy is synonymous with the API economy. APIs are increasingly the way business partners—internal and external—exchange and monetize data and services. APIs allow for machine-to-machine data retrieval, essentially removing barriers and accelerating access to data. There is hardly a modern application today that doesn’t provide an API to integrate with other applications and data sources. The typical mobile app today uses an average of 10 to 15 APIs to move meaningful data into and out of the app. Modern web apps called SPAs (single-page apps) do the same. APIs have proven to be so efficient and useful that many companies have begun creating new business ventures by mobilizing their data to third-party developers and the public.

    DevOps teams heartily embrace APIs to advance their ability to produce applications much more quickly. APIs, however, are largely a mystery to security teams that must ensure that this means of rapid data exchange does not create unnecessary risk and negative exposure for the company or its brand. Enterprise security teams are right to question the vulnerabilities and threats posed by APIs. Insecure APIs have led to some very serious data breaches, some of which are outlined in the Dark Reading article, “Expect API Breaches to Accelerate.”

    For example, the mobile payment app Venmo exposed sensitive details about hundreds of millions of transactions through a poorly secured API. In another case, an insecure Panera Bread API used by a mobile app exposed users’ private details through a searchable API that required no authentication to access. As a result, as many as 37 million records, which included customer personal information identifiers (PII) such as names, email addresses, physical addresses, birthdays and the last four digits of credit cards—all in plain text—were exposed. T-Mobile, McDonald’s, Symantec, Instagram and numerous other companies have suffered through significant data breaches traced back to insecure APIs. The damage is high because most APIs are designed to serve data without security in mind.

    DevOps and security teams can agree that a data breach coming from their applications is completely untenable. Thus, designing and applying a security program that spans across all APIs across all applications is an essential function in protecting data and brand. Just exactly how can this be achieved? Let’s look at the steps of automating an API security program without needing to add additional security staff.

    The Challenge of Securing APIs

    The very nature of identifying all APIs makes it a challenge to secure them. A developer can create an API in a matter of minutes and publish it on the internet with almost no resistance when using public cloud services such as Amazon, Microsoft and Google. What’s more, she can make changes to that API as often as she wants, and every change cycle presents an opportunity to introduce new risk. It’s not unusual for a developer to change her API weekly or daily.

    Let’s go back to that example of the financial services company that sells data to stock brokers. Suppose the developer makes a change to her API which unintentionally enables the brokers to see the client lists of every other broker that consumes data via the API. That would be a catastrophic breach, yet this is precisely the type of API breach suffered by the U.S. Postal Service a year ago.

    APIs today increasingly are built on serverless infrastructures such as Amazon Lambda, Azure Functions or Google Cloud Functions. Traditional firewalls, gateways and agents can’t protect this type of API built on ephemeral infrastructure. Now consider that a large enterprise might develop and/or use hundreds or even thousands of APIs. The magnitude of the challenge to secure every API becomes apparent. Manual security oversight and enforcement are simply out of the question. Automation of security assessments done on a continuous basis is the only scalable approach.

    Start With an API Security Framework

    Every organization that creates or uses APIs needs an API security framework that consists of three steps:

    1. Continuous API discovery and specification creation.
    2. Continuous API specification analysis and inspection.
    3. API policy enablement and enforcement.

    Let’s have a look at each step and its impact.

    Know What APIs Are in Use and What They Are Intended to Do

    Gathering API specifications is an important first step in defining what APIs are intended to do. However, many APIs exist without any specification. A service that creates API specs helps to deal with those APIs void of documentation. However, this spec creation service also needs to continuously monitor and discover new APIs to remedy the problem of not knowing what APIs exist and their intended function. It’s common for a developer to document what her API is designed to do initially, but updating that documentation every time she makes a change to her API is rare. To compensate, the organization needs to have an automated tool that gathers this information and notes each time an API changes, then accordingly updates the API specification.

    For example, consider an API that exists to transfer customer information between a company’s back-end server and a mobile application. The developer adds a feature or two and the API changes to support these new features, resulting in the API doing things it wasn’t originally expected to do. How would the organization know about these changes? There is a need to continuously analyze the API for such changes to understand the current state of specifications for what the API is doing.

    The organization needs to create specifications of all the APIs, especially publicly facing APIs hosted in the cloud. This service needs to operate continuously to look for changes in the APIs constantly. There are tools that can automate these various activities, including discovering new APIs and changes to existing APIs. These tools also can uncover other cloud resources and data services related to these APIs; enumerate API domains, functions and associated methods; and generate a standards-based (Swagger, OpenAPI v3) spec if one does not exist.

    Analyze the APIs for Security Threats

    The next step is to perform a security inspection on each API operation every time it changes. The security team needs to know: Does it have the right data encryption? Is proper authentication in place? What type of authorization policy is being applied? What is the API’s level of availability? What kind of data sources does it access? Knowing each API’s current security posture is critical if the organization wants to avoid a data breach.

    This API analysis needs to take place continuously and at scale since manual processes are ineffective for organizations using numerous APIs. Fortunately, there are tools and technologies that automate this process and perform tasks such as finding potential vulnerabilities within the authentication and encryption layers of the API. The tools also can generate security tasks with recommended changes for developers to remedy their respective API problems and alert security personnel when there are discrepancies between the API specification and its functional operations.

    Create and Enforce Security Policies

    No two companies have the same appetite for risk. If there’s one aspect of creating an API security program that requires a manual investment of time, it’s policy creation. Automation then can be used to enforce the policy. A security policy for APIs should start with two primary questions: Who should be able to utilize the API? And what level of sensitivity, regulatory oversight and/or privacy concerns does the API have? Based on the answers to those first two questions, you can form the basis for the security policy.

    For example, a bank has an API that provides transaction statements for a specified account. That API has access to extremely sensitive and private information for each of its customers. One customer should never be allowed to access another customer’s account statements. Therefore, the authentication, authorization, encryption and availability of that API would be set to the most restrictive security standards available to ensure data exposure is tightly controlled. Another API from this same bank may provide the current interest rates of savings and money market accounts. This is data the bank likely would share with anyone potentially interested in opening an account with them. In this case, the level of security restriction would be lowered to support a wide variety of systems, browsers, SDKs and apps to ensure this information is highly available.

    Historically, API policy enforcement for authentication, encryption and availability was often done at the network gateway layer. As applications have migrated to modern architectures such as mobile and cloud, the gateway approach has increasingly become less practical, less scalable and more restrictive. Application developers are often leveraging authentication, encryption and availability zones provided to them through SDKs and/or cloud services. In these cases, an automated service that integrates with these security services is a better approach for API policy enforcement versus traditional network gateways.

    Automation is Key

    Security teams are notoriously overworked and short-staffed. Worldwide, there’s a dire shortage of skilled security professionals available for hire, so adding additional security staff to build and execute the API security program isn’t a practical option. Bringing in consultants is expensive and can be budget-busting. Thus, automation through tools and technology is a sensible approach.

    Automation provides the benefits of saving time and money when it comes to executing the three steps outlined above. What’s more, an automated program provides consistency in the repetitive tasks necessary to discover, document and analyze the APIs in use, and enforce corporate policies to control risk.

    APIs are critically important to modern applications. DevOps people heartily embrace them, and security teams can learn their value as well once they understand how a robust and continuous API security program will benefit their organization.

    — Doug Dooley

  • Continuous Reliability: How QA and SREs Can Improve Their CI/CD Workflow

    Continuous Reliability: How QA and SREs Can Improve Their CI/CD Workflow

    A key challenge in maintaining and accelerating the speed of software delivery is balancing the rate of change with the reliability of your software. With the move toward an increasingly automated workflow, as well as teams of dozens to hundreds of engineers introducing changes at a high frequency, the ability to identify the issues that will have a severe impact early on becomes critical.

    This challenge poses many questions: With so many errors, will spending my precious time to resolve a particular issue be really worth it? How do I detect severe issues in code or infrastructure that could be impactful ahead of time? Once a release has been moved forward, how can I know how well it’s doing–should it be fully deployed, hotfixed or even rolled back?

    That’s why we need our CI/CD workflow to work both ways, helping us upgrade and improve our application as well as allowing us to learn and adapt according to how it performs with our users. Enter continuous reliability.

    A Feedback Loop Between Everyone

    Code quality gates and contextual feedback loops are the CI/CD building blocks that define the emerging practice of continuous reliability. The current and most common feedback loop is between dev and QA teams, and it’s usually the backbone of every good development cycle. However, feedback loops are relevant to the whole development cycle, and they don’t end with QA/dev deploying and testing elements in staging environments.

    As we all know, production tends to display unexpected and surprise-oriented behavior, some might even say that issues and bugs appear out of thin air. That’s why QA and SRE teams need accurate answers as to the actual quality of releases. They need to identify when critical anomalies move from staging and production, and ideally have the ability to gate them before they harm the application, workflow or customers.

    Defining the Anomalies That Matter

    Anomaly detection is critical to continuous reliability. In most applications, error and performance anomalies can be mapped into three core types:

    1. New defects: the introduction of new errors into a system, usually a result of code changes.
    2. Increasing Errors: an increase in the relative volume (i.e. as compared to throughput) of certain errors, usually as a result of a code or infrastructure change.
    3. Slowdowns: a decrease in the response time of critical parts of the application due to inefficient code or improper provisioning or configuration of infrastructure.

    The introduction of any of these can lead to CPU, GC and IO increases. As such, it is not only imperative that QA and SRE teams gate them from moving into production, but also to prioritize and assign severities that help tackle the most important issues.

    However, detecting anomalies in large-scale systems can be nearly impossible due to a high noise-to-signal ratio, and we’re often left to rely on our users and customers to alert us when something goes wrong. That’s why there’s a need for an objective measure to quantify the quality of a release—one that lies in being able to converge machine learning and critical code-level data to automatically gate bad releases from moving up the chain.

    Continuous Reliability Through Quality Gates

    Once we have an understanding as to what should count as an anomaly within our application, the next step is setting a quality gate to help us identify, understand and hopefully stop bad code from being promoted to production.

    Taking the three core type of anomalies—new issues, regressions and slowdowns—we can evaluate four core quality gates that our code should pass before being promoted:

    1. Error Volume: The normalized error rate of an application, based on calls into the code, should never increase between releases.
    2. Unique Error Count: The number of unique error counts, especially in key applications, should not increase between releases.
    3. New Errors: New errors of a critical type or occurring at a high rate should block a release.
    4. Increasing Errors and Slowdowns: Severe regressions and slowdowns should block a release.

    These four gates can help you define a benchmark that is powerful and broadly applicable to complex environments, but also isn’t complex to the point that it remains an academic exercise. They will help you understand whether your code is ready to be deployed to your pre-production or production environment, and help reduce or even eliminate errors, issues, slowdowns and anomalies within your application.

    — Tal Weiss

  • Survey Finds Reliance on APM Mainly Reactive

    Survey Finds Reliance on APM Mainly Reactive

    A survey of 358 strategic IT decision-makers in enterprise organizations across North America and Europe conducted by the market research firm GigaOm on behalf of SolarWinds, a provider of IT management tools, finds that roughly half of the respondents consider themselves proactive in their use of application performance management (APM) tools.

    That definition is based on those respondents either strongly agreeing or agreeing that their organization was employing APM tools within the context of all the following criteria:

    1. Engaged in a digital business transformation initiative.
    2. Makes use of a hybrid or multicloud computing strategy.
    3. Has a public-cloud first IT strategy.
    4. Makes use of best practices for IT operations.
    5. Line of business teams are driving technology change.

    Denny LeCompte, general manager for application management at SolarWinds, said that while significant progress is clearly being made it terms of APM adoption, it’s clear that half or more of the potential market is still wrestling with a variety of challenges that often stem from the cost and complexity associated with acquiring and employing an APM platform at scale.

    LeCompte said most of the use of APM is still reactive in that most organizations don’t actively engage an APM tool until there is an actual issue. The survey finds the primary driver for adopting APM is to fix problems (61%), followed by keeping on top of complex application environments (57%), delivering the best possible customer experience (52%) and managing costs and delivering efficient operations (34%). However, it’s worth noting that among the half of the respondents claiming to be proactive about employing APM, customer experience was the primary driver for adopting APM (61%).

    Even so, LeCompte noted reliance on APM platforms is often uneven at best. When it comes to adoption, he noted, most organizations are on a journey. Initially, the focus is on IT troubleshooting. Over time, however, customer experience with digital business processes becomes a much bigger factor. Different teams with the same organization are at different stages of that journey, he said.

    SolarWinds views the rising tide of APM awareness as the justification for jumping into the category, LeCompte said, claiming the company is committed to providing APM capabilities an IT organization is likely to require, at a fraction of the cost of rival platforms. That’s critical because many organizations have limited the number of applications they track using APM tools because of cost, he said.

    As IT environments become more complex in the age of microservices built using highly ephemeral containers, LeCompte said it’s only a matter of time before more IT organizations will need to embrace APM platforms proactively to optimize overall application performance. Individual IT administrators won’t be able to keep track of all the dependencies that might be impacting a “cloud native” application without relying on an APM platform, he noted.

    The issue now is finding a way to provide access to an APM platform that everyone on the IT team can regularly employ without breaking the IT budget.

    — Mike Vizard

  • How to Create Bug-Free Blockchain Apps

    How to Create Bug-Free Blockchain Apps

    While all developers strive for bug-free code, it’s particularly crucial in a blockchain deployment where sensitive data or other confidential info is being exchanged, such as in health care or finance. However, some businesses have learned that lesson the hard way. Cryptocurrency exchange Binance recently revealed a devastating security breach that resulted in a loss of more than $40 million. While this is a large sum, a security breach could be even more costly for businesses that lose private messages, confidential contracts and more.

    Rebuilding (and regaining customer trust) is not where organizations should be spending their time. A better way forward is for company leadership to mandate a blockchain test automation strategy from the very beginning.

    While most software testers are familiar with testing web applications, they may not be sure how to approach blockchain testing. The good news is, some of the same focus areas of web testing apply to blockchain testing, too (functional, performance and security). However, blockchain involves a greater number of tests, effort and focus areas, such as infrastructure orchestration, staging and simulating scenarios in a distributed environment, which requires an even greater level of expertise.

    Because of these additions, fixing a bug in a blockchain app requires several extra steps and, as a result, it is more time-consuming and costly than fixing bugs in other apps. For example, to reproduce bugs or verify code, developers should perform these steps multiple times:

    1. Form a new endpoint.
    2. Deploy new code to the new endpoint.
    3. Migrate the current data to the new back end.
    4. Suspend the old back end.
    5. Update all front ends to the current version.

    After that, verifying the bug-fix performed by testers or continuous integration (CI) tools will require these steps again. This complexity makes it difficult to release bug-free blockchain applications, yet doing so puts organizations and their customers at risk of financial harm. Due to the high-risk nature of deploying blockchain applications, it’s critical that testing is included in the managerial vision and strategy.

    To release blockchain apps with bug-free code, businesses should follow these guidelines:

    Integrate Test Automation Into Your Blockchain Testing Strategy

    Test automation can relieve testers from performing thousands of orchestrating tests manually, which is both difficult and inefficient. A good system architecture should be scalable, maintainable and testable, so teams can test components independently along with mocked components. If a test requires a different environment or infrastructure, infrastructure as code (IaC, such as Puppet, Chef, Ansible and AWS is invaluable in setting up appropriate test environments. Another crucial part of the strategy is to provide the infrastructure with declarative definitions and allow developers to implement codes that support test automation.

    Run Unit and Integration Tests to Cover Both Back End and Front End Code

    Blockchain applications contain at least a back end (a set of smart contracts or chain codes running on a blockchain) and a front end (where users can interact with data stored in the blockchain). Performing unit tests will cover both back end and front end code, and, because of its critical role to blockchain apps, its primary focus should be on the chain codes. Integration testing, however, should cover the various types of oracles and front end applications. Performing additional tests will eliminate any holes that can threaten the back end by verifying the integration between the UI and chain codes.

    Perform Tests in a Local Infrastructure

    Though public blockchains (such as Bitcoin or Ethereum) have several test-nets, it’s optimal to perform the tests in a local infrastructure. Local infrastructures are lightweight, faster and more stable for unit and integration tests. For example, in a Ganache local infrastructure, it’s possible to run more than 1,000 unit tests on five Ethereum smart contracts with approximately 3,000 deployments in an hour or less.

    Continuously Update Security System Test Suites

    Blockchain is a multi-party infrastructure and in some cases, it is a public network. As a result, these applications will be visible to many stakeholders and may even be open-sourced to the world. This level of transparency means that hackers are able to study the programs and plan attacks based on a particular system. Continuously updating security system test suites will ensure applications are protected against all common security vulnerabilities.

    Minimize Continuous Testing Pipelines for Each Phase of the Development Process

    Layering integration and unit tests in a local infrastructure with continuous updates to the security system test suites creates a continuous testing pipeline that is essential to high-quality blockchain code. Continuously verifying code changes in multiple environments, such as local blockchain and test-net, ensures high-quality code is delivered to the production blockchain. These tests, however, can last hours or days for each code change, so it’s important for teams to minimize and create lean pipelines that focus on test stages, test types and test environments, and develop multiple pipelines for code at each phase of the development process.

    Build Customized Tests Suites to Eliminate Prospective Cheating Holes

    Test teams also need to customize test suites in the pipelines to cover both functional and security testing. Covering security cases is important, but unit test modules should also exercise smart-contract-unit potential security threats, such as overflow and re-entrancy, to eliminate as many prospective cheating holes as possible.

    Due to the nature of decentralization and orchestration of underlying infrastructure, developing bug-free code for blockchain applications presents many more challenges than other applications. By taking best practices from traditional software testing, especially test automation, and customizing them for blockchain development, businesses can achieve bug-free blockchain coding and protect their most vulnerable applications.

    — Thong Nguyen

  • New Relic Doubles Down on DevOps

    New Relic Doubles Down on DevOps

    New Relic today unfurled an extension of its monitoring platform specifically designed for DevOps teams tasked with managing complex IT environments.

    Nadya Duke Boone, vice president of product management for New Relic, said New Relic One, available to existing customers at no additional cost, makes it possible for DevOps teams to now visualize all the data that exists across multiple New Relic accounts their organization has created in a single set of dashboards.

    New Relic One also makes it possible to create an “entity” to track their relationships and dependencies to provide more context to the metrics surfaced by New Relic. That context is critical anytime a major incident occurs because these days so much of the revenue any organization generates is determined by the availability of their applications, said Duke Boone.

    To achieve that goal, New Relic One provides Service Maps spanning entities; search and discovery tools based on universal tag filtering; dashboards through which queries can be launched using visual tools; an ability to more easily extend visualizations; and a new home page designed to make it easy to drill down into various dashboards.

    Other capabilities included in New Relic One include Monitoring for the AWS Lambda serverless computing framework, a Kubernetes cluster explorer and distributed tracing global search.

    New Relic also revealed today it plans to deliver both a log monitoring solution and AIOps offering within New Relic One by the end of the fiscal year. New Relic earlier this year acquired SignifAI to gain access to AIOps platform.

    Duke Boone said all this is now possible because of the investments New Relic has made in developing a persistent store for monitoring platform, which she said represents a strategic initiative on the part of New Relic to expand the scope of the services and capabilities it provides.

    Downtime in the age of digital business directly equates to lost revenues. Organizations that face that challenge typically are at the leading edge of adopting best DevOps practices. That challenge these organizations face today is they are drowning in complexity, a situation that likely will only worsen as organizations deploy cloud-native applications based on containers, she said, adding New Relic One not only serves to make it easier to manage all that complexity, it also makes it simpler for teams within an organization to share best DevOps practices.

    Of course, one of the issues that restrains organizations from adopting DevOps practices in the first place is a reluctance to make IT environments any more complex than they already are. Organizations are demanding that IT organizations become agile enough to support a wide range of digital processes. The challenge they all typically face is the current reliance on highly structured processes that are intended to minimize potential errors. As well-intentioned as that might be, however, they often result in IT organizations being unable to adjust rapidly changing business requirements. The good news is as DevOps tools become more sophisticated, the skepticism of many IT professionals have toward DevOps will continue to quickly diminish.

    — Mike Vizard

  • Logz.io Unveils Azure Open Source ELK Monitoring Solution

    Logz.io Unveils Azure Open Source ELK Monitoring Solution

    Needless to say, DevOps as a practice is largely fueled by open source. The growth and popularity of open source tools such as Elasticsearch, Kubernetes and others are driven largely by the increased flexibility, innovation and familiarity of options. While this isn’t news for developers (65% of which are active open source contributors), leading cloud providers are taking notice, as they try to increase their appeal to DevOps engineers to finally win the cloud wars.

    Currently, AWS is the leading cloud provider. However, with Azure gaining more and more momentum, all three major cloud providers are scrambling to offer their customers the best open source options and empower them to use the tools they know and love in a manner that is convenient and native to their cloud environments.

    Going Big on Open Source

    Today, Logz.io and Microsoft Azure announced that our ELK-as-a-service will now be offered on Microsoft Azure. Logz.io offers an intelligent machine data analytics platform built on ELK and Grafana, two of the most popular open source monitoring and troubleshooting tools. Our platform enables engineers to easily deploy, run and scale ELK without the hassle and pain of maintaining and managing the stack themselves.

    Now, with Logz.io for Azure, users can use open source ELK as a highly integrated experience in Azure. The new offering includes seamless integrations to various Azure services, out-of-the-box Azure dashboards, built-in alerting and crowdsourced machine learning capabilities.

    The integration between Logz.io and Azure is not only significant in the context of the cloud wars, but in the culture of Microsoft itself. While the company was previously associated with a more proprietary structure, they have made tremendous strides in recent years toward empowering open source by opening its .NET framework and Azure SDK, as well as their acquisition of GitHub.

    According to Julia Liuson, corporate vice president of Microsoft’s developer division: “At Microsoft, we see open source as a major driver of innovation within the engineering community. Since we strive to empower our customers and facilitate their success, it is important to us to meet them where they are by making it easy to access and use open source tools. Our partnership with Logz.io will strengthen Microsoft Azure ties with the developer community, enabling teams to easily build and monitor their products using ELK.”

    AWS vs. Google Cloud

    Microsoft’s investment in ELK  comes just in time to provide an answer to its competitors, AWS and Google Cloud. Just two months ago, AWS announced Open Distro for Elasticsearch, which in their own words “is a value-added distribution of Elasticsearch that is 100% open source (Apache 2.0 license) and supported by AWS.” The new offering includes managed Elasticsearch and Kibana along with features such as built-in alerting, security and more. This was AWS’ most recent appeal to the engineering community, offering one of the most popular monitoring tools as a managed service on AWS.

    Following this release, Google Cloud announced its own open source offering featuring integrations with leading companies such as Elastic, MongoDB, Confluent and more. According to Google, the purpose of  these integrations was to “provide users with a seamless user experience and the ability to easily leverage these open-source technologies in Google’s cloud.”

    Is Open Source the Future?

    If what’s happening between cloud vendors is any indication of future trends in the engineering space, it seems imperative that all organizations targeting DevOps teams must consider how they’ll fuel open source and empower its vast user community. At Logz.io, we are proud to be part of this initiative.

    Overall, it is great to see the strides large organizations are taking in furthering the interests of the developer community and enabling them to work using the tools, methodologies and strategies that best suit their workflows and ideals.

    — Daniel Berman

  • Application Monitoring Using Stagemonitor

    Application Monitoring Using Stagemonitor

    This article covers important steps to monitor a typical Java application using Stagemonitor tool, Kibana as a dashboard viewing tool and Elasticsearch as a data processing engine.

    Stagemonitor is an open source Java application performance monitoring (APM) tool that provides everything you need in development, quality assurance and production to monitor the performance of your Java web application.

    Using this tool, we can have different dashboards such as JVM dashboards, application server, request metrics, host, request analysis, web analytics, etc.

    Architecture of the Stagemonitor

    Figure 1

    The Stagemonitor agent sits inside the application and sends metrics and details about requests (request traces) to Elasticsearch. Metrics can optionally be sent to Graphite, Influx DB and Prometheus.

    Prerequisites

    • Minimum of 2GB RAM and 200GB hard disk
    • Java version of 1.8 or more
    • Install latest versions of Kibana and Elasticsearch

    Steps to Monitor Java App with Stagemonitor

    Dev Time Monitoring With Browser Widget

    This step illustrates how to enable a widget in dev time monitoring, adding the Stagemonitor plugin inside the Java app. We can do this in two ways: We can directly edit the pom.xml/build.gradle file by adding the required stage monitor plugins, or we can add dependencies in the application’s LIB directory.

    Some of the important plugins we must have in pom.xml/build.grade to get monitoring dashboards are Stagemonitor configuration, Stagemonitor tracing Elasticsearch, Stagemonitor tracing, Stagemonitor-jdbc, Stagemonitor-jvm, Stagemonitor-os, Stagemonitor-web-servlet and Stagemonitor alerting.

    We’ll get the Stagemonitor plugins from the following location: https://mvnrepository.com/search?q=stagemonitor

    After installing the agent, will have to create one Stagemonitor configuration file as stagemonitor.properties file under src/main/resources in case of maven application. Otherwise, the file must be located at the root of class path. At the end, it has to be under the WEB-INF/classes section.

    The following is what the stagemonitor.properties file looks like:

    stagemonitor.applicationName=My Application

    stagemonitor.instanceName=dev

    stagemonitor.instrument.include=org.example.app

    Request Dashboard

    In this section, we focus on setting up the time series database to collect the data and visualization tool to view the metrics in detail. Here we’ll use Elasticsearch as the time series database engine and Kibana as the visualization tool.

    To get the data from Elasticsearch database, we will include properties in the stagemonitor.properties file:

    stagemonitor.reporting.elasticsearch.url=value,

    stagemonitor.reporting.elasticsearch.username=value

    stagemonitor.reporting.elasticsearch.password=value

    stagemonitor.reporting.interval.elasticsearch=60

    After adding all the required values and restarting the server, we will be able to open the (automatically added) request analysis dashboard in Kibana. Along with that, we will be able to see different dashboards automatically.

    To use Influx DB as a time series database, we’ll have to configure the values in stagemonitor.properties file:

    #following the details contains influxDB

    stagemonitor.reporting.influxdb.url=http://localhost:8086

    stagemonitor.reporting.interval.influxdb=60

    stagemonitor.reporting.influxdb.username= value

    stagemonitor.reporting.influxdb.password=value

    Logs Monitoring using Kibana/Stagemonitor

    We can track logs using Kibana/Stagemonitor. In the metrics dashboard, we can see matrices such as what URL address we’ve hit and the time to complete the request (Figure 2). We can track the host details such as how many servers our applications are running, the status of the server, memory stats, CPU stats, network stats, disk usage stats, networks processed, network packets errors, network packets dropped and how many cores of the CPU the host is using.

    Stagemonitor provides all the required dashboards under the request analysis dashboard. To open the request analysis dashboard, click on: Dashboard – Load Saved Dashboard and select “Request Analysis.”

    Figure 2

    Figure 3 shows the details of the CPU, such as CPU usage and CPU load.

    Figure 3

    Below, Figure 4 shows total page views, unique number of visitors to the site and call tree, and how much time it takes to complete.

    Figure 4

    Figure 5 shows the transaction history and individual transaction details.

    Figure 5

    Figure 6 shows the JVM details and all the other memory matrices.

    Figure 6

    As mentioned above, we can monitor various application metrics and infrastructure metrics.

    — Sai Babu Tanguturi

  • Wavefront Provides Map for Microservices

    Wavefront Provides Map for Microservices

    VMware announced it is adding a service mapping capability to the Wavefront 3D Observability platform it provides with the Wavefront by VMware monitoring tools.

    Stela Udovicic, senior director of product marketing for Wavefront, said Wavefront Service Maps will make it easier for IT organizations to visualize what will soon be thousands of microservices running highly distributed computing environments, including those based on containers running on Docker Engine, connected via service meshes such as Istio or spanning proxy servers such as Envoy.

    Capabilities provided by Wavefront Service Maps include packaged visualization of request flows with a map of service dependencies, built-in tracking of health metrics and view into individual traces to pinpoint performance bottlenecks.

    Accessed as a set of software-as-a-service (Saas) applications, Wavefront by VMware relies on an agentless approach to monitoring that supports all the major public clouds along with providing integrations with platforms such as Kubernetes, GitHub, Slack, PagerDuty, Kafka and Pivotal Application Service.

    Udovicic said by making it easier to visually identify and troubleshoot microservices, IT organizations should be able to accelerate their transition to modern cloud-native applications. Building and deploying a single application based on microservices is complicated enough. Trying to support hundreds of those applications is simply too daunting without the ability to visualize all the interdependencies between those services. Troubleshooting any issue then becomes easier because Wavefront Service Maps is designed to enable DevOps teams to drill down to the traces being employed to monitor the application, she said.

    Earlier this year, Wavefront added support for distributed tracing and machine learning algorithms that also make it easier to identify anomalies within a microservices-based application.

    In terms of adoption of cloud-native applications, Udovicic said there are basically two classes. The first are early adopters who are already highly committed to developing cloud-native applications. The bulk of the enterprise IT community, however, is still in the early stages of building and deploying these types of applications, said Udovicic.

    There’s naturally a high correlation between organizations that have adopted cloud-native application development using microservices aggressively and those that have implemented best DevOps practices. In fact, one of the benefits of a visualization tool is that it makes it easier to bring new members on to those teams. Rather than having to piece together what is occurring in the environment, a new member of a DevOps team can visually inspect each microservice they might be responsible for managing.

    It may be a while before DevOps processes become the dominant default method for managing IT. But as DevOps tools become easier to use, the number of DevOps practitioners should increase steadily. Arguably, one of the reasons DevOps adoption is not as far along as it might be is that the learning curve is still too steep. But as visualization capabilities gets added to the analytics frameworks baked into those tools, it’s now only a matter of time before the number of IT professionals that might be intimidated by DevOps processes starts to decline steadily.

    — Mike Vizard