devopsy.kinsta.cloud

Category: Enterprise DevOps

  • Getting RPA Right the First Time

    Getting RPA Right the First Time

    Remember when every article you read about IT focused on digital transformation (DX)? Now the broad concept of DX seemingly has taken a back seat in the headlines to a number of hot technologies. One of these, Robotic Process Automation (RPA), is frequently positioned as the new next big thing of the tech world. But if you look closely, you’ll notice that DX never really went away–RPA is really just the latest piece being added to that larger digital transformation puzzle.

    Digital transformation is often explained as an ongoing journey that promises to help organizations break the traditional boundaries of enterprise IT, empowering business line workers to enable the delivery of faster outcomes and better experiences to customers. An important part of DX is eliminating the rote task work that plagues day-to-day operations. That can only be done with transformative automation technology. It’s no wonder that many organizations worldwide are focused on exploring, implementing or scaling RPA to help check that box.

    But as the RPA hype cycle starts to crest, don’t be fooled into thinking that RPA is the single turnkey solution to your enterprise problems. There are ways to avoid common pitfalls that are starting to emerge in RPA projects and ensure its successful implementation and achievement of business results.

    RPA projects need proper planning, a strategy and an implementation team, all tied into broader enterprise systems and process management. Without these, organizations are bound to face costly delays that can affect deployment plans and return on investment (ROI). In fact, a recent study from EY found 30% to 55% of initial RPA projects fail. In another industry survey, it was reported that 87% of respondents had experienced RPA bot failure while reporting that maintenance was the second biggest concerns users face.

    Here are the most common mistakes organizations should avoid to achieve real results.

    Thinking RPA is Easy Since It’s Everywhere

    This is perhaps the most common myth around RPA–enterprises expect to purchase a solution, train business users to build automation and see the ROI start to accumulate. It is reported that only 39% of bots are deployed on schedule, and it typically takes 18 months to push bots into live production. This is hardly an out-of-the-box solution.

    But we all know how difficult technology implementations can be. Very few things in our industry are truly plug-and-play–no matter what sales professionals might say. While RPA is pitched as a way to solve menial tasks, there is nothing menial about the process of automation and developing an application tailored to how work actually gets done. It is possible to train up employees to build simple automation, but the true results will not be realized without an end-to-end approach that takes into account the broader outcome the automated task is supporting.

    Remember, your business has rules on security, compliance and governance–all critical to the success of an RPA deployment. Are you considering these factors and all the affected stakeholders across the entire organization when trying to fix a problem within one department?

    There is a place for business users building automation, but to get to real scale and potential for transformation, IT needs to work together with stakeholders across different lines of business to design an end-to-end intelligent automation solution that ties into pre-existing business systems and accounts for their sophistication/complexity, while also accounting for where humans need to be involved.

    Attended or Unattended?

    RPA is the hottest trend since cloud computing, and it boils down to two basic forms: attended and unattended. Simply put, attended requires the help of human interaction on the desktop, while unattended takes over tasks without the need to interruption. Is one better than the other? Not necessarily. In the right situation, both can generate quick wins and task automation to assist the workers.

    Consider the desktop of call center agents. These workstations are known for having deep integrations with legacy technology that are geared toward repetitive, scripted and manual tasks. This provides the perfect opportunity for attended RPA deployments to address these types of functions that take place across the entire call center, delivering better customer outcomes and generating business value.

    There is a time and place for both attended and unattended solutions–it is important to understand which capabilities the RPA vendor is offering and how they will address the needs of your current situation. In many situations, attended can be an unexpected source of ROI opportunity, as the total volume of work automated in environments such as contact centers, can far exceed the amount of non-human assisted work that can be automated.

    RPA Will Solve All My Problems

    RPA is great—I wouldn’t be writing about it if it weren’t. But it’s not the end-all digital investment automation platform; it’s part of an organization’s digital transformation strategy and can create serious business value when used the correct way. Automating and streamlining the complexity of multiple, long-running processes that interlink internal and external systems, machine and human work, bespoke software and third-party apps and systems is not what RPA was created to do. That’s the work of a BPM solution  supported by  Case Management. Businesses need to realize how RPA fits into their overall digital transformation strategy and where specifically it can be applied across their organization to realize its true benefits.

    You’ve likely heard of the old saying, patience is a virtue. Let’s apply that to RPA deployments. There is so much promise and opportunity for RPA within the enterprise, but the major ROI numbers that some expect may not be there–at least not yet.

    Finding the right opportunities for RPA technology within the organization, getting company buy-in and developing and deploying the solutions all take time to create an impact. Businesses need to approach their projects with a strong understanding of where RPA can be applied and what it does well, like automating high-volume, low-complexity, rule-based tasks within an end-to-end enterprise process automation strategy, to realize the business benefits it brings.

    — Nolan Greene

  • When Should You Go for Microservice Architecture?

    When Should You Go for Microservice Architecture?

    There is steady increase in the adoption of microservice architecture style since 2014 (Figure 1). Microservice architectural style structures an application as a collection of loosely coupled services that implement business capabilities.

    An architecture style is a family of architectures that share certain characteristics. For example, N-tier is a common architecture style. Architecture styles generally don’t dictate the use of particular technologies, but some technologies are well-suited for certain architectures. For example, containers are well suited for microservice since they go hand-in-hand for the deployment.

    In this article, you will learn about the typical considerations, challenges and benefits of the microservice architecture style.

    Figure 1: Microservice Google Trends.

    While microservice applications are built as a set of modular components, they are easier to understand, simpler to test and effortless to maintain over the life of the application. It enables organizations to achieve agility and be able to improve the time it takes to get working enhancements to production.

    It also enables the continuous delivery/deployment of large, complex applications and the evolution of its technology stack. Having detailed out on the benefits side there are quite a bit of challenges as well, which we will discuss in the next section.

    Constraints Imposed by Microservice Style

    By adhering to the following constraints imposed by microservice architecture style, what we get is a system where services can be deployed independently, faults are isolated, frequent updates are possible and it’s easy to introduce new technologies into the application.

    • Each service represents a single responsibility.
    • Every service is independent of the others.
    • Each service needs its own JVM (or equivalent) so instances are isolated.
    • Data is private to the service that owns it.
    • Services do not share data and need to implement queries that need to retrieve data owned by multiple services.
    • Well defined mechanism to implement service-to-service communication and scheme to deal with partial failure.
    • Requests that span multiple services and require careful coordination between the teams.
    • Testing to cover the interactions between services.
    • Deploying and managing a system comprised of many different services.

    Consider Challenges Versus Benefits

    Each architectural constraint has its own associated challenges/benefits, so it’s important to understand and balance both. Before adopting any style of architecture, you have to ask yourself if the benefits outweigh the challenges for the application you are considering.

    In the table below, we take a look at the benefits versus challenges of microservice style applications.

    Benefits Challenges
    Enables the continuous delivery and deployment of large, complex applications. There is an additional complexity of creating a distributed system.
    Improved maintainability: Each service is relatively small so it’s easier to understand and change. Implementing requests that span multiple services is more difficult.

    Maintaining data consistency between service(s) is a challenge.
    Better Testability: services are smaller and faster to test. Testing the interactions between services is more difficult.
    Better deployability: services can be deployed independently. Increased operational and deployment complexity of deploying and managing a system comprised of many different services.
    Each team can develop, test, deploy and scale their services independently of all of the other teams. Implementing requests that span multiple services requires careful coordination between the teams.
    Improved fault isolation. Inter-service communication and dealing with partial failure implementation is challenging.
    Eliminates long-term commitment to a technology stack. Overhead of multiple JVM runtimes (or equivalent) and increase in memory consumption needs to be taken care of.

    Scenarios Where You Can Consider Going for Microservice

    Consider an scenario where you are developing an enterprise application that must support the following characteristics:

    • Support for different variety of clients that includes desktop browsers, mobile browsers and native mobile applications.
    • Expose an API for third parties to consume.
    • Integration with other applications via web services or a message broker.
    • Run multiple instances of the application on multiple machines to fulfill scalability and availability NFR requirements.
    • Adopt emerging technology stack.
    • Planning to go for continuous deployment pipeline setup for the application.

    From the requirements above, you can very well say the application is 100% fit for a microservices architecture.

    You have to define an architecture that structures the application as a set of loosely coupled, collaborating services. Services can communicate using either synchronous or asynchronous protocols. Services can be developed and deployed independently of one another. Each service has its own database in order to be decoupled from other services.

    Let’s look at some of the typical scenarios where you can consider going for microservice style of architecture:

    1. Monolithic application migration due to improvements needed in scalability, manageability, agility or speed of delivery.
    2. Re-platform a legacy application by transforming functions/modules to microservices.
    3. Rewriting legacy application to modern languages, technology stack to meet the demands of modern business.
    4. Individual cross-cutting services that are independent in nature. For example: encryption services, authentication services, etc.
    5. Independent business applications or services reused across multiple channels. For example, payment services, login services, flight search services, customer profile services, notification services, etc.
    6. Commonly used enterprise applications. For example, time tracking application.
    7. Scenarios where a service provider makes computing resources and infrastructure management available to the customer as needed. For example, forecasting services, price calculation services, prediction services, etc.
    8. Back-end services for responsive client-side web application where data could be coming from multiple channels or different data sources.
    9. Highly agile applications or applications demanding speed of delivery or time to market or innovation pilots, etc.
    10. Applications that got polyglot, multi-language, cloud application development.

    Make the Choice

    Before going for microservice architecture, make sure you know the challenges and benefits. Otherwise, your application design could end up with a design that fits the style but does not achieve its full potential. Adopt microservices only if you see enough value for your applications. Also note sometimes relaxing a constraint is better than insisting on architectural purity.

    — Karthikeyan Shanmugam

  • Harnessing the Power of Developers

    Harnessing the Power of Developers

    In our fast-moving digital economy, organizations across just about every industry are faced with a stark reality: to remain competitive and deliver the best possible customer experience, they must evolve continuously. The fintech industry is a great example of this–the sector’s growth and focus on excellent customer experience has been driven by external developers rather than incumbent players.

    For many companies, particularly in the service industry, improving customer experience now means developing new apps and services that will maximize the customer journey, engender deeper trust and secure customer relationships for the future.

    The aim is to continuously produce innovations that create delightful customer experiences, which in turn will drive ROI. In its 2019 Technology Industry Outlook, Deloitte highlighted the fact that the “democratization of innovation” is set to be a key strategic factor in the coming years. As cloud infrastructure makes it easier to innovate at speed, “medium and small enterprises can harness powerful capabilities once limited to a select few.”

    That might be a new SaaS application, for example, or the implementation of emerging tech such as AI or VR–anything that helps the company provide a more efficient or accurate customer service. A simple example could be integrating complementary, adjacent solutions to add additional value to end customers.

    Companies need to innovate in order to create unexpected functions and capabilities that will delight. But how do companies go about making that a reality? In a nutshell, there are two groups that can help you innovate: your own developers and the wider developer community. Having the right people involved and the right programs in place for both of these channels will allow you to scale and deepen your innovation more effectively–and ultimately create experiences that your customers cannot live without.

    The more you engage with developers inside and outside your organization, the faster you can accelerate your time-to-market for new innovations and, crucially, ensure that those innovations meet real customer needs.

    In-House Developers

    Building an effective in-house development team to meet customer needs is about more than just hiring the right computer skills. Companies need to aim for a fearless, fail-fast culture that lets teams go above and beyond in answering customer needs. This need not apply to all developers–there will be some focused on refining the core product set, for whom fail-fast may not be the ideal working model. However, in the pursuit of unexpected offerings that will delight customers, a healthy attitude to both fail-fast and conventional development is essential.

    Often, organizations are held back from being the best they can be by ingrained, deep-rooted cultures. The power of “the way we do things ‘round here” can be hard to crack, but good leaders must help staff buy into change as an ongoing process that will help them and the business to succeed in the long run.

    That’s where the concept of experimentation and fail fast comes into play. In these environments, developer teams are given permission to try things out without having to go through multiple rounds of approvals and senior buy-in. That means they can go from an initial idea to a fully testable concept in, say, weeks rather than months or years. It also means they can run their blue-sky ideas through a crash course of reality, bumping up against unforeseen problems–what some people might consider failures, but I think of more as learnings–without being penalized for it.

    When developers are given this permission to get ideas off the ground quickly, the benefits to the organization are obvious. Improved access to innovation; a greater chance of discovering more effective ways of working; and a supportive, forward-looking culture that helps to enthuse non-technical staff as much as developers themselves. The same is true for customers–they will benefit from innovations that improve their experience, surprising and delighting them with the idea they didn’t know they needed, but which exactly addresses their needs.

    Practical Considerations

    There are other key issues to address, particularly senior buy-in and customer-centricity. First, it’s essential that developers know their work is valued by senior leadership. Senior leaders need to trust their experts and not just allow but also champion the fail-fast model if the company is to benefit from the successful innovations that come out of it.

    Second, developers need to work in close communication with customer-facing teams to ensure their efforts are directed toward innovations that will translate into improved customer experiences. In other words, it’s important developer teams innovate with the company’s core goals in mind–not just for the sake of it, however tempting that might be.

    There are also pragmatic issues to be addressed, such as ensuring your APIs or SDKs are well developed, and there are good documentation/guides to help use these APIs. This is required for both in-house and external developers.

    External Developer Networks

    Having a wide network of development partners can also help rapidly scale innovation and deliver solutions that exceed customer expectations with speed. By building a community of great minds around the mutual benefit of market-ready innovation, organizations can unlock far more valuable projects, with positive effects for them, their customers and their partners.

    An effective developer partnership works on the basis that there is a symbiotic relationship where each side has something to offer to the other. For example, the developer partner may come from a smaller company who has specific expertise and can benefit from the financial support and expertise of the larger organizations. While both sized organizations can benefit from the diverse thinking, agility and challenger mindset of the developer partnership. Whether you’re a large company looking to fast-track customer-centric innovation or a small company with a great customer-centric idea that needs a boost onto the next rung, a healthy developer network can be highly beneficial.

    As an example, the accounting industry is currently undergoing a major cultural change, with accountants increasingly seeking to provide customers with value-add strategic services to maintain their competitive advantage. Technology is driving a consultancy-first service model. Accountants have access to more data than ever before, and as a result they can help clients to make effective business decisions based on new analytical insights.

    However, making the jump to effective consultancy requires new tools and innovations to turn raw information into usable insights. By working in partnership with a network of cutting-edge developers, accounting software providers can help to conceive, road-test and launch user-friendly solutions.

    The value of developer networks is also evident as you look at domain specific needs that might span a particular industry or a particular region. One company may not have the expertise or bandwidth to do it all–but by building effective communities of developers, an organization can expand their reach to tap into niche areas of expertise. It also gives the company the ability to deliver innovations more rapidly. Every company has a limited set of resources, and additional developer networks can help scale this innovation. For the developers, the value lies in reaching markets and customers they may not previously had access to.

    Taking Innovation Out of the Lab and into the Market

    Whether your company uses in-house developers or a partner network (or both) the core goal should be to hone ideas into realities through collaborative working practices. When developers and customer-facing teams understand one another, they can achieve great things.

    It’s an exciting time in the industry. Technology is evolving faster every day, solving more problems and opening up new possibilities. For businesses of all varieties, harnessing the power of developers should be a core priority. It’s not a question of pursuing innovation for the sake of it–it’s about taking practical steps to ensure you maximize the positive impact of tech on your customer experience.

    — Shivani Govil

  • Data Monoliths Go Cloud-Native with Kubernetes

    Data Monoliths Go Cloud-Native with Kubernetes

    Breaking monolithic applications into independently scalable services is the core of the cloud-native application model. By making cloud-native apps easy to deploy and manage, Kubernetes has contributed to a big change in IT practices.

    Initial cloud-native conquests started with stateless services and quickly grew to encompass applications that manage data. Database management systems have long relied on centralized servers, but cloud-native architectures are now reaching even the largest monoliths. Among the most interesting cloud-native converts are analytic databases, which hold data arranged in storage formats optimized for extremely fast analytic queries over billions ,or even trillions, of rows.

    The transformation of databases to cloud-native models has profound implications for data-driven applications. Intersecting trends in analytic database architecture and Kubernetes promise substantial benefits to the users of such applications as well as companies that operate them.

    The Rising Importance of Analytics

    Over the last decade the quantity of data that businesses generate has decoupled from the size of the business itself. Ad networks, website analytics and service observability applications often ingest 1 million events per second (or far more) and may have tables running into trillions of records. Businesses need to analyze this information rapidly–for example, to diagnose faults in cloud applications or to retrain models for displaying online ads. Flexible, real-time analytics have gone from nice-to-have to business critical.

    Meanwhile, new analytic engines have emerged that eliminate dependencies on specialized hardware and complex networking required by legacy data warehouses. Druid, ClickHouse or Elasticsearch run on commodity hardware and depend on basic compute, network and storage. Everything else, such as high availability (HA), is built into the application. Newer analytic databases offer built-in horizontal scaling using fleets of server processes. The server processes themselves typically run well in containers.

    With these changes, analytic databases are increasingly cloud-friendly if not cloud-native. It is a short hop to deployment on Kubernetes.

    Kubernetes Operators for Databases

    At the same time, as analytic databases have shifted their operational model, Kubernetes itself has become more welcoming for databases.

    One particularly important innovation for the database is custom resource definitions, or CRDs, which extend the Kubernetes API to define new types of manageable resources. Using CRDs makes it possible to have a single definition that shows the desired state of a complex application such as a data warehouse.

    Another addition is Kubernetes operators, which manage custom resource definitions. Operators track custom resource definitions and take actions when they change. They can also track other information that might be relevant to managing the custom resource, such as changes to pods, storage or even related cloud-native applications. It’s like having a human operator who is constantly watching the application, except the rules are programmed into software that runs inside Kubernetes itself.

    The operator pattern is a perfect match for analytic databases, which are complex distributed applications and may require definition of dozens or even hundreds of individual Kubernetes resources in a running system. Operators encapsulate these details as well as important procedures such as upgrade, scale-out and other operations that require careful manipulation of resources to prevent outages.

    It’s not surprising operators are popular with database developers. The OperatorHub.io registry for database operators alone has 19 entries covering databases of every kind, including analytic databases such as ClickHouse and Elastic. Many more exist outside the registry.

    A Cloud-Native Way to Consume Analytic Databases

    New data warehouse architectures and Kubernetes operators means users can now use analytic databases very differently from legacy data warehouses. Our work on ClickHouse, including development of the ClickHouse Kubernetes Operator, prompts two observations.

    First, operators make spinning up analytic databases simple and fast for all users. Amazon Redshift kicked open the door to data warehouse-as-a-service in the public cloud. Kubernetes operators enable similar capabilities not just in public clouds but any Kubernetes cluster. Individual developers can cycle through dozens of configurations in a single day while testing new analytic applications. Thanks to quick pod spin-up new analytic databases can be running in minutes or even less.

    More profoundly, users can use Kubernetes to move to an on-demand model for analytics. Rather than forcing all applications to compete for resources on a single shared cluster, each application can have an analytic database tuned to its specific requirements. Analytic databases can integrate tightly with applications in the same locations where those applications already run. Businesses can scale databases flexibly to deliver quality analytics at exactly the time they are needed and at reasonable cost. This is the heart of the cloud-native programming model.

    Modern database architectures combined with Kubernetes features, such as the operator pattern, are creating a new path to manage enormous volumes of data. For readers who are hesitant about managing data on Kubernetes, it’s time for another look.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Robert Hodges

  • Red Hat Launches Holistic DevOps Training Initiative

    Red Hat Launches Holistic DevOps Training Initiative

    Red Hat this week announced it has created a Global Transformation Office focused on bringing about digital transformation via broad adoption of best DevOps practices.

    The Global Transformation Office initiative is comprised of Andrew Clay Shafer, vice president of transformation at Red Hat; Jabe Bloom, chief sociotechnical officer for PraxisFlow; Kevin Behr, general manager and chief science officer at PraxisFlow; and John Willis, CEO of Botchagalupe Technologies.

    Shafer said Red Hat is trying to encourage organizations to take a more holistic approach to transformation by simultaneously addressing the need to understand how new technologies impact the organization in terms of skills required and the business model around which the organization is built.

    Instead of focusing solely on IT skills, Shafer said the Global Transformation Office at Red Hat is also seeking to educate line of business executives how to engage with DevOps teams to advance digital transformation initiatives, many of which never come to fruition because the IT teams working on those projects don’t know what the end business goal really is. If line-of-business executives can communicate that goal in a way DevOps teams can appreciate, they soon discover how much faster those DevOps teams will be able to drive the organization toward that desired outcome, he said.

    Rather than simply telling IT teams they will have a new role, Shafer said business leaders need to back up those claims by investing in training. Otherwise, IT professionals who today often associate their identity within an organization to the process they manage will resist that transition out of fear of losing their position. Right now, too many organizations are also asking IT teams to embrace DevOps without ever really informing DevOps teams why any given project matters to the business strategy, added Shafer.

    Red Hat is investing in this initiative because the amount of demand for enabling technologies such as the Red Hat OpenShift application development platform will be constrained unless business and IT models evolve in tandem. It makes no sense for organizations to embrace containers only to try managing them using the same processes they previously employed to manage virtual machines and bare-metal servers, said Shafer. Organizations need to realize they will not get the maximum value out of such platforms unless they have the skills and expertise needed to apply them within the context of achieving a business goal, he added. In many cases, organizations may need to reorganize themselves because too many business leaders still value themselves in terms of the size of the organization they lead instead of the contribution the business unit makes to the company.

    It’s not clear to what degree the DevOps tail may be able to wag the business dog. In many ways, it’s only a matter of time before the business needs to be reorganized in a way that makes it as agile as DevOps teams are becoming. The real question is how long that transition will occur. Of course, many organizations that don’t make that cultural DevOps transition won’t be around to find out how it all turns out.

    — Mike Vizard

  • Learning Production Troubleshooting Techniques in Microarchitectures

    Learning Production Troubleshooting Techniques in Microarchitectures

    One of the great things about our industry is the ability for anyone to get started doing development with a minimal amount of tools. Anyone can learn to do development with just a computer of some sort, a text editor, a tutorial and their brain.

    In fact, the initial basics of how to think through a development problem don’t require a computer at all, reducing the cost of getting started even further. Having such an easy way to get started that doesn’t require a lot of tools lowers the bar for entry, enabling more people to join in on development teams and contribute new ideas and perspectives, which enriches the entire industry.

    Getting into containerization and building modern microarchitectures is a slightly higher bar, but only slightly higher. There are a number of wonderful tutorials out there for getting started with Kubernetes, Docker, OpenShift Origin and other free and open-source tools.

    The only differences in getting started between general software development and microarchitectures specifically are: needing a computer that has the processing power and memory to run containers, and enough of a grasp of a development language to have a basic application to play with. Even that last difference isn’t as crucial because there are so many sample applications people have built for the express purpose of teaching new folks to deploy microservices onto various systems.

    However, when it comes to learning from the DevOps, SRE or ops side of managing Kubernetes deployments or other modern architectures, there’s no solid free tutorial I could find that dumped a bunch of broken sandboxes in front of the student with the express purpose of building their troubleshooting knowledge.

    To gain insight into understanding how to bring microarchitectures back from failures or fall-overs requires access to larger applications and clusters and more senior, or more experienced, mentors–unless you are lucky enough to be able to build large interconnected applications and send traffic to them regularly enough to cause failures.

    This lack of strong tutorials and a requirement for more resources makes breaking into DevOps or SRE work on microarchitectures particularly difficult if no one is willing to take a chance on a less experienced candidate. I want to see that change. We are missing out on hearing from new voices, people who may look at a problem from a different perspective or who may have unique solutions to problems we all face.

    I decided to create a repository full of deliberately broken Kubernetes builds for new practitioners to build progressively more advanced troubleshooting skills. The basic idea is to build a pyramid of environments. The base level should be easy to understand and figure out. These environments will be bare-bones simple, with only one cause for failure.

    We need to build the baseline recognition of common causes in isolation. Many of us that have debugged production systems have an almost unconscious grasp of when a common cause has appeared simply because we recognize that faint signature in the pattern of the data we’re analyzing. If you’ve never seen that signature, though, and you don’t have a mentor with you on these debugging challenges, learning to find that signal in the haystack of your data is much harder. These initial environments should make it easier to learn what those signals look like.

    The next level of environments would have some additional factors unrelated to the actual problem causing additional logs, additional spikes in traffic or additional activity. In short, these environments would have noise. We can add another layer of complexity for some of these environments by adding another service or application that runs just fine on the same cluster. This level would train the ability to rule out sections of a system that aren’t broken, to identify systems that aren’t interconnected and to work on one system without disturbing the rest of the environment.

    Further environments up the pyramid would have multiple contributing factors, red herrings or complex structures. At this point, some of the environments will need to be hosted outside of the local system. I’m still debating how to do these environments so the tutorial could still be free and easy for new folks to work with–I’m open to suggestions.

    Ideally, there would also be a written guide with answers and tips a student wouldn’t look at until they’ve solved the problem on their own, but they certainly can look up and learn regardless of whether they solved the problem. After all, troubleshooting isn’t a closed-book exam by any stretch of the imagination.

    So how do you start? Find the repo at GitHub. It’s free–open source with an MIT license–and open to contributions. There’s a contribution section of the README on the repo that explains how to get started if you want to send commits. If you want to try using these environments yourself and practice or learn troubleshooting skills, the initial environments are built on Minikube, a tool to run small Kubernetes clusters on your local machine. Follow one of the many excellent tutorials for getting Mikube up and running on your local machine, then follow GitHub’s tutorial on cloning down a repo.

    Alternatively, you can download the repo directly. Once you’ve gotten Minikube up and running and you have a copy of the repo on your local machine, follow the instructions in each environment’s directory to deploy the environment to your local cluster, and start trying to figure out exactly what went wrong.

    Want to help make this project a reality? I would love to have more contributors! In addition to being available online for help, I’m going to be at KubeCon North America 2019 in San Diego. At the conference, I’ll have a Hackers’ Corner set up in LogDNA’s booth for most of the conference, and I’ll be sitting there working on this project, among others, waiting for you. Bring your ideas, your stories, your broken environments, your laptops, your willingness to write docs, your Kubernetes knowledge, yourself—anything that can help enrich this project is welcome.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Laura Santamaria

  • How In-Memory Analytics is Solving Data Woes

    How In-Memory Analytics is Solving Data Woes

    The importance of data to business decisions and operations cannot be understated. On a daily basis, 2.5 quintillion bytes of data is generated. Managing that much data so the necessary intelligence is easily accessible or to make sure insights are not missed, can be incredibly challenging. Luckily, technology does exist that can help to significantly ease the strain of this deluge of data: in-memory analytics.

    Initially adopted by the finance industry—where complex, on-the-fly decision-making is required–in-memory analytics is incredibly adept at handling the analysis of substantial amounts of data very quickly. And as data more ubiquitous, and the development of public cloud services has made specialized hardware affordable to companies of all sizes, in-memory analytics technology is being applied in a variety of verticals from retail, to supply chain to healthcare. In addition, given in-memory computing enables a range of analytics features that are very useful to businesses of all kinds and extremely difficult to achieve with disk-based platforms, the technology is really poised to expand rapidly.

    Slice and Dice Any Which Way

    One of the hardest parts of being a data professional today is leveraging all the data at your disposal. Unfortunately, business intelligence (BI) pros are often constrained in terms of what portion of the company’s data they can and cannot analyze depending on pre-defined data queries and the limitations of their BI tools.

    In-memory analytics, however, enables analysts to explore data without any of those limitations. This creates an experience of train-of-thought analysis that helps businesses make deeper analyses and uncover valuable new insights into their market and their own performance. This means retailers for example, can have a global, precise view of the market and cross-reference data from sales numbers, client receipts and loyalty programs, to develop a better understanding of what customers expect from them.

    Figuring Out “What If?”

    Due to the vast amount of data involved, and the limitations of physical storage discs, the calculation of the impact of any business decision on complex KPIs can take a long time to resolve.

    By utilizing in-memory analytics BI pros can continuously compare and contrast “What If” scenarios—such as running stress tests for financial institutions or setting up disruption test cases for supply chain businesses—with a clear, immediate view of the potential outcomes, so they can be as prepared for the future as possible. This means businesses are able to limit the amount of times they are caught off guard or mitigate the impacts of any unexpected turns.

    Up-to-Date Data

    With how rapidly the world moves today, having data that is as current as possible is pivotal. Static datasets only updated once or twice a day can lead to significant, costly misjudgments in some cases. In-memory analytics on the other hand can support a continuous stream of data from multiple sources so analysts always work with the most relevant set of data. This means supply chain businesses, for example, can have real-time oversight over operations to better cope with any disruption that occurs and ultimately save millions of dollars in potential losses.

    The boom of data science has opened a new field of use cases for in-memory analytics, ranging from the validation of new algorithms derived from ML/AI to industrializing the building of new analytics views and dashboards. If you have invested heavily in data science, in-memory analytics could be the pivotal cog in generating the maximum output from each piece of your organization’s data infrastructure.

    — Antoine Chambille

  • Five Tips to Approach DevSecOps Training

    Five Tips to Approach DevSecOps Training

    Developers are on the front lines when it comes to protecting their organizations from cyberattacks. As we’ve seen with the hacks of Monster.com or the Fortnite vulnerability, 2019 has been a trying year for organizations who have failed to protect their applications and websites.

    With an increase of cyberthreats today, it is vital that organizations and developers incorporate standard security protocols within DevOps, a practice known as DevSecOps. Whether exploring these practices around security training, skills, metrics and more in the industry, it’s important to understand how each group fits into the security process.

    Here are five top tips for security training to successfully implement a DevSecOps strategy.

    Recognize Security is a Shared Responsibility

    Many departments throughout an organization make the mistake of thinking the responsibility of keeping their organization secure lies solely with the security department. However, with nearly 200 cyberattacks happening every hour, it’s important everyone gets on board in protecting applications and networks. As the creators behind our modern-day websites and applications, it’s especially important for developers.

    DevOps and security teams need to be taught how to approach security jointly. When approached separately, it is impossible to achieve DevSecOps. Without DevSecOps, the applications and websites used by consumers everyday will not be secure, and there could be security, quality, legal and reputational implications. Security is now being seen as a fundamental aspect of a consumer’s decision, and consumers gravitate towards brands that are more secure.

    Ensure the Entire Development Team Understands Common Cyberthreats and Application Vulnerabilities

    In today’s world, it is especially important for development and operations specialists to understand application vulnerabilities, and best practices to avoid those vulnerabilities. Without this knowledge, developers and their teams won’t understand just how important security is until it is too late.

    Staying on top of the evolving threat landscape, however, can be a daunting task. After all, there is a lot of information out there. The first step to staying informed is to compile a short list of reliable, valid security news sources. Another step could be creating an internal method for the security team to share details on new vulnerabilities with the development teams.

    Unfortunately, hackers tend to exploit vulnerabilities that were accidentally created by DevOps. In order to fight this, DevOps specialists should be made aware and trained in secure programming, secure application configurations and use of secure frameworks.

    Incorporate Just-in-Time Learning into Your Strategy

    Traditional training courses, especially in the security sector, have some challenges. Many universities and free online resources will offer courses online in cybersecurity. While this can be beneficial in continued training, it can take a significant amount of time to complete the course, and some projects require a certain level of knowledge sooner rather than later. Most courses need to be taken well in advance of a project, where the acquired knowledge will then be applied.

    For example, say developer X just started a course in how to protect from vulnerability Y. At the same time, developer X has a project with code that could be vulnerable to Y due in the next two weeks. With the tight deadline, developer X may not be able to get the knowledge they need to be able to protect their application in time.

    In order to combat this, the security industry now offers real-time learning. Otherwise known as “just-in-time learning,” this approach is giving the employees the much-needed information exactly when they need it. These courses usually come hand-in-hand with using application security testing technology, enabling rapid remediation of a detected vulnerability. For example, when a new security technology is introduced to the organization for development purposes, just-in-time learning would involve developers learning to use it on current projects in real time.

    Avoid Making DevOps Specialists into Security Experts

    While it is important for DevOps practitioners to have knowledge of security best practices, it’s important to remember they should be focusing on their task at hand: creating applications. Security transparency is a critical condition for security adoption by the DevOps team, so it is important the technologies developers rely on are transparent. However, security technologies should not distract DevOps specialists from development and operation. DevOps specialists should be security aware and applying their best security development practices.

    The technologies should test applications and get results to those who need them. DevOps practitioners should then use those results, without having to worry about learning and running sophisticated security detection and protection technologies.

    Remember the Goal is to Educate, Not Embarrass

    Even when using the above tactics, it is still quite possible developers will miss a vulnerability and SAST, DAST, IAST and SCA technologies may discover it first. When this happens, it is easy for DevOps specialists to become embarrassed because their managers see the results first. No matter the experience level, developers will likely at some point make a mistake that can cause a vulnerability–which ultimately gets back to their supervisors and peers.

    Rather than feel embarrassed over the security technologies, developers need to learn how to utilize them to their advantage. For example, modern SAST allows developers to conduct tests out of an IDE and return the test results to the same one. The developer is the only one who will see these tests, and therefore, they can be the one to review and remediate a problem. This process helps eliminate the shame developers feel, and it will instead be replaced with education, security and confidence.

    Cyberthreats are not going away, and all signs are pointing to the fact that they are only going to get more advanced in the future. However, with adequate training and cybersecurity education for development teams, organizations can stay ahead of adversaries and fortify their application security.

    — Joseph Feiman

  • Identifying and Abstracting Business Intelligence from Kubernetes Workloads

    Identifying and Abstracting Business Intelligence from Kubernetes Workloads

    In the age of big data, businesses are inundated with data points. While they know there’s an abundance of valuable resources available to them, making sense of this data to derive actionable insights is often still a challenge.

    CNCF industry survey data shows workload complexity and monitoring remain top challenges for enterprises in terms of using and deploying Kubernetes. Many understand there are valuable resources within these environments, but struggle to best identify and extract meaning from machine data. While most of this data is readily available, it just takes the right tools to gather and view intelligent insights.

    The Benefits of Open-Source Data Collection

    Perusing the CNCF website, there are a wealth of tools built around Kubernetes to enable not only monitoring but networking, storage and security. There is even a Kubernetes-specific package manager, Helm, to make the deployment and management of these resources easy and consistent.

    There are a couple of key benefits to taking advantage of these open-source collectors:

    1. They stay up to date. Each of these tools benefits from deep community support. As new versions of Kubernetes are released, the extensive use of each of these tools ensures they are quickly updated.
    2. They integrate with everything. Prometheus, for example, has an impressive list of integrations and exporters.  Regardless of your unique stack, it is likely that there is support for what you might want to export data from. The importance of these integrations cannot be overstated, as they enable the flexibility needed to grow and evolve a Kubernetes deployment over time.

    Ensuring Complete and Organized Metrics Collection

    Prometheus–endorsed by the CNCF–is the de facto tool of choice for metrics monitoring with extensive support for anything you might use to collect metrics data.

    Prometheus works by pulling data from all of the components and jobs running in Kubernetes since every component of Kubernetes exposes its metrics in a Prometheus format. The processes running behind those components then serve up the metrics on an HTTP URL. For example, the Kubernetes API Server serves its metrics on https://$API_HOST:443/metrics.

    Prometheus is particularly good at auto-discovering the jobs and services currently running in a Kubernetes cluster. As pods are added, removed or restarted, the Kubernetes Service construct keeps track of what pods exist for a given service. This auto-discovery capability is one of the primary reasons for Prometheus’ popularity, ensuring that all new and existing components are monitored.

    The Importance of Cluster Level Logging and Event Collection

    Kubernetes does not define a single standard approach to log collection, but the most common method is called cluster level logging. Cluster level logging deploys a node level logging agent to each node which then funnels data to a separate backend for storage and analysis of logs. The primary benefit of this solution is that if a pod dies, the logs detailing what happened are retained. Implementing node level logging, without funneling data to a logging backend, will not retain log data if pods die or are evicted, while cluster level logging ensures that data is captured and retailed. A common tool for implementing cluster level logging is Fluentd—or Fluentbit, a lightweight version of Fluentd—which acts as the node level logging agent funneling data to a logging backend.

    Events provide insight into decisions being made by the cluster and unexpected events that occur in Kubernetes. Events are stored by the API server on the master node and collected using the same method as log collection—via a node level logging agent like Fluentd.

    Establishing a Continuous Intelligence Dashboard

    Finally, collectors for logs, metrics, events and security can be easily deployed using Helm—an open source Kubernetes package manager. Helm can significantly simplify the setup process, reducing hundreds of lines of configuration to one. These collection plugins can be used on any Kubernetes cluster, whether one from a managed service such Amazon Elastic Kubernetes Service (EKS) or a cluster you are running entirely on your own.

    According to Sumo Logic’s recent Continuous Intelligence Report, enterprise adoption and deployments of multi-cloud grew 50% year-over-year, and 80% of users across multi-cloud environments are now utilizing Kubernetes architectures. As Kubernetes continues to become mainstream, it’s essential that businesses understand how these workloads operate and how to best extract valuable insights to inform business decisions.

    — Katie Lane

  • Cloud Native Tracing and Observability: Why You Care

    Cloud Native Tracing and Observability: Why You Care

    Is observability the new monitoring? Or is observability, and tracing, fundamentally different? Like any IT industry trend, it can be difficult to discern as many jump on the trend bandwagon, appropriately or not. The Splunk .conf 2019 event presented Splunk with the opportunity to bring clarity to the terms and the reasoning behind the acquisitions of SignalFx and Omnition.

    Monitoring is mostly about taking events, telemetry data and established data points and thresholds into monitoring, alerting and problem management processes and tools. Its tried and true, and has progressed with the evolution of typical monolith, distributed apps and systems. Enter cloud-native applications, composed containers, microservices and service meshes. With great flexibility comes complexity, and cloud native isn’t immune to such an adage.

    What would have been a monolithic application is now shattered into hundreds or even thousands of smaller pieces of application and app technology software. The complexity of externally monitoring cloud native quickly surpasses the capabilities of most monitoring approaches.

    Observability, as recently acquired SignalFx CTO Arijit Mukherji shared this week, is built upon three pillars: metrics, tracing and logs. Metrics show when you have a problem, tracing points you to the problem and logs help find the root cause—a reasonable way to define and segment observability. It also, not surprisingly, fits well with the logic of Splunk plus SignalFx plus Omnition.

    To move beyond monitoring requires instrumentation, built into the software and APIs as part of the software creation process, a DevOps process. SignalFx in part brought Splunk auto-instrumentation, which during run-time, identifies frameworks and libraries in use within applications and can capture tracing instrumentation. Omnition brings even deeper tracing chops to perform tracing across large service meshes of microservices. Add Splunk’s capability to correlate data across the business, applications and operations data and you complete the picture with the ultimate goal of making observability easy for developers.

    The above might explain why developers, ops and DevOps care about observability and tracing, but should the business care? SignalFx CEO Karthik Rau connected the dots nicely during a conversation this week. Digital transformation strategies require speed and agility, but also demand more risk-taking. Confidence in risk-taking comes when accompanied by the ability to respond rapidly to changes and failures. A software deployed 10 minutes ago may need to be backed out or corrected rapidly when the users’ experience goes negative. That requires a rapid determination of what the problems is, and the ability to take immediate corrective action, including automated action.

    Cloud native and DevOps not only enable disruptive, digital transformation strategies but must be accompanied by rapid and automated responses when negative business and customer impacting conditions occur. Customers don’t care when CPUs are taxed, but they do care when a mobile app’s responses fail or are slow. The move to cloud native is served well when backed up by such capabilities to respond in near real-time when problems occur.

    — Mitchell Ashley