devopsy.kinsta.cloud

Category: DevOps Practice

  • Privileged Access Management at the Speed of DevOps

    Privileged Access Management at the Speed of DevOps

    Privileged access management (PAM) is clearly a very important element of effective cybersecurity. PAM has been ranked number one on Gartner’s Top 10 Security Projects for two consecutive years, and it is number four on the CIS Top 20 Critical Security Controls. The struggle for many organizations, however, is they have also embraced DevOps culture, and the traditional approach to PAM clashes with the principles of a DevOps environment. Thankfully, it is possible to effectively manage privileged access in a DevOps environment—it just takes a new approach.

    Challenges of Privileged Access Management

    Despite advances in identity and access management and the existence of things like two factor or multi-factor authentication and biometric authentication, most organizations still rely on a simple username and password to authenticate an individual and grant access to network resources, applications and data.

    The concept of least privileged access—ensuring that users only have access to things they really need access to—has been preached since the dawn of cybersecurity and, when done right, can be an effective strategy. The problem is it’s tedious to do right and it’s very easy to lose track of who has privileged access to what.

    That’s where a PAM solution comes in. The whole point of privileged access management is to provide a way to track and manage who has access to what—to grant access when it is needed and revoke access when the need no longer exists. Unfortunately, there is a lot of confusion about PAM and the effort to manage privileged access is challenging for many organizations.

    Enterprise Management Associates (EMA) conducted a survey on the state of privileged access management. The survey included responses from companies representing different company sizes and industries around the world and focused on IT and cybersecurity professionals—most of whom are familiar with PAM and are directly responsible for managing or granting privileged access to users.

    There were some concerning findings from the survey regarding the perception of PAM. Twenty percent of organizations that reported using a dedicated PAM solution believe it somewhat or greatly reduces user productivity. Many respondents also expressed frustration about the amount of time required to manage privileged access. Half of the survey participants cited manually granting temporary privileges as either very (29%) or extremely (21%) time consuming, and half also said manually revoking temporary privileged access as very (33%) or extremely (17%) time consuming.

    PAM and DevOps

    As challenging as traditional PAM solutions might be for businesses around the world, they pose an even greater hurdle for organizations that have embraced DevOps. A recent episode of the Defense in Depth podcast, hosted by David Spark and Allan Alford, focused on some of the challenges of PAM. A listener had written to complain that “the traditional approach to PAM breaks everything that is DevOps. Regardless of what your PAM provider says about their ability to scale with your dev cycles, they’re lying about the real level of effort.”

    That seems like a fair assessment. Pretty much every company is somewhere on the cloud journey—and working within a DevOps framework on some level. Companies often face backlash from the DevOps team—with some going so far as to boycott efforts to introduce security tools that can’t keep up with the pace of DevOps in the cloud.

    DevOps developers express frustration about trying to build an automatic scaling, horizontal infrastructure and being dragged down by demands from the security team to install agents and change authentication modules. They complain it often means rewriting all of the scripts, resulting in significant and unnecessary effort for a solution that ultimately can’t scale properly in the DevOps environment.

    That is, unfortunately, true when it comes to traditional PAM solutions. Next generation PAM solutions, however, are built with the cloud and DevOps in mind. They are lightweight, automated, scalable and capable of integrating seamlessly with common tools and platforms.

    Rethinking Privileged Access Management

    Based on the survey, organizations understand the importance of privileged access management, but they struggle with finding a way to implement it effectively. The question from the podcast illustrates how PAM is also a significant issue for DevOps environments because traditional PAM solutions do not integrate or scale effectively.

    At the same time, we all know that PAM is a crucial element of effective cybersecurity. It’s not acceptable to just not manage privileged access, but it also doesn’t make sense to suffer through the headaches and frustration introduced by traditional solutions. Organizations need to find new privileged access management tools that work in the cloud and can scale automatically to provide PAM at the speed of DevOps.

    — Tony Bradley

  • Extending Automation to Your First Code Commit

    Extending Automation to Your First Code Commit

    I am a big advocate for software reuse, and an even bigger advocate if that reuse comes in the form of software automation. My team and I rely on reusable software processes that accept simple inputs to produce predictable results. We have tight deadlines with requirements that often seem impossible. Automation makes us more efficient and our work more predictable.

    Once upon a time, we used glue and bubble gum to cobble together means by which to build, then test and then eventually deploy application versions. It worked until it didn’t, and industry opportunities arose for mechanization. Enter continuous integration (CI), continuous deployment (CD) and DevOps, in part.

    You likely know this and have already taken advantage of CI/CD platforms such as Jenkins, CircleCI and others. The simplicity of the as-code model allows us to turn easy-to-create declarations into the otherwise complex provisioning of servers, databases, caches and file systems, along with the execution of code to leverage what was provisioned.

    Life as a developer has been made better using these as-code tools in the tool chain. Spinning up a Kubernetes cluster on Google to (re-)deploy an image built, tested and pushed to DockerHub from a CloudBees/CodeShip pipeline is something I rely upon each time I commit to my Git host.

    In this article, I am going to talk about Project-as-Code to extend automation to the first code commit with a working app running through a functional CI/CD pipeline and deployed to a production-like environment.

    Barriers Keeping Enterprise IT Departments from Coming Along for the DevOps Ride

    Unfortunately, basic usage of all these great tools comes with the added complexity and cognitive load cost. Understanding Docker image creation, HashiCorp Terraform syntax for infrastructure-as-code, Kubernetes cluster deployment and more are just a few of the competencies developers and operators need to have.

    Learning a new set of tools every few years comes with the territory, but there’s no reason to ignore new potential avenues for automation. I’ve wondered if there is a better way to abstract and automate the setup of these common pieces in the modern software production pipeline.

    Another common issue I’ve noticed—these pipeline components often get set up sequentially, with the first meaningful code commit being the blocker to setting up all the other pieces of the CI/CD pipeline. Without this commit, these tools sit relatively idle while waiting to provide value.

    For those of us who work on new projects regularly, getting things started has always had its issues, especially for consulting scenarios where requirements and team dynamics differ between clients. We are disciplined enough to first reach in our toolbox to reuse as much as possible. Still, DevOps has yet to be adopted wholesale in a standard way by many enterprises.

    For these situations, I find the benefits of DevOps are easy to explain and understand. Developers like to reuse and fewer barriers, the business likes savings and an enterprise likes productivity and innovation. However, to inject those benefits is still a very manual, time consuming process, making the overall sell of DevOps harder.

    The purpose of DevOps automation is to simplify tedious error prone manual processes. I see another opportunity for automation: entire DevOps projects using the same as-code model we have become accustomed to.

    Generating Your First Meaningful Commit “As-Code”

    Why not use a straightforward declarative model-driven means of not only generating tech-stack specific application source code, but all the config files that lead to DevOps being automated. With this, the toolchain learning curve is minimized, and idle tools now are immediately called to action. The result is a self-sufficient operational DevOps environment working through an application code base that comprises a fully deployable executable first code commit. Developers then get their reusable assets and environment(s), the business sees cost and time savings, and the enterprise gets the productivity and innovation necessary to stay competitive.

    In an article written by Pete Hodge titled “Hello production,” he goes into great detail about the value of quickly standing up a flowing CI/CD pipeline leading to a functioning app deployed to production. He contends deploying even a useless application into production, as soon as you can, is the right way to start a new project because it pulls unknown risk forward, opens up parallel streams of work and establishes good habits. Project-as-Code standardizes this thinking with model-driven automation.

    The next generation of DevOps is unfolding with automation remaining a key component to expand scale, velocity and productivity. As automation has worked its way up the toolchain, automating the automation behind DevOps will shift the paradigm to improve how projects are created, how DevOps is implemented and the quality and features of the first commit into a Git repository. What now takes weeks to months could instead take hours to days, falling in line with our expectations now met by other successfully mechanized aspects of DevOps.

    Hopefully this article will serve as a starting point for your next DevOps project and how Project-as-Code automation can be a part of getting you started easier and faster.

    — Steven Randolph

  • Getting RPA Right the First Time

    Getting RPA Right the First Time

    Remember when every article you read about IT focused on digital transformation (DX)? Now the broad concept of DX seemingly has taken a back seat in the headlines to a number of hot technologies. One of these, Robotic Process Automation (RPA), is frequently positioned as the new next big thing of the tech world. But if you look closely, you’ll notice that DX never really went away–RPA is really just the latest piece being added to that larger digital transformation puzzle.

    Digital transformation is often explained as an ongoing journey that promises to help organizations break the traditional boundaries of enterprise IT, empowering business line workers to enable the delivery of faster outcomes and better experiences to customers. An important part of DX is eliminating the rote task work that plagues day-to-day operations. That can only be done with transformative automation technology. It’s no wonder that many organizations worldwide are focused on exploring, implementing or scaling RPA to help check that box.

    But as the RPA hype cycle starts to crest, don’t be fooled into thinking that RPA is the single turnkey solution to your enterprise problems. There are ways to avoid common pitfalls that are starting to emerge in RPA projects and ensure its successful implementation and achievement of business results.

    RPA projects need proper planning, a strategy and an implementation team, all tied into broader enterprise systems and process management. Without these, organizations are bound to face costly delays that can affect deployment plans and return on investment (ROI). In fact, a recent study from EY found 30% to 55% of initial RPA projects fail. In another industry survey, it was reported that 87% of respondents had experienced RPA bot failure while reporting that maintenance was the second biggest concerns users face.

    Here are the most common mistakes organizations should avoid to achieve real results.

    Thinking RPA is Easy Since It’s Everywhere

    This is perhaps the most common myth around RPA–enterprises expect to purchase a solution, train business users to build automation and see the ROI start to accumulate. It is reported that only 39% of bots are deployed on schedule, and it typically takes 18 months to push bots into live production. This is hardly an out-of-the-box solution.

    But we all know how difficult technology implementations can be. Very few things in our industry are truly plug-and-play–no matter what sales professionals might say. While RPA is pitched as a way to solve menial tasks, there is nothing menial about the process of automation and developing an application tailored to how work actually gets done. It is possible to train up employees to build simple automation, but the true results will not be realized without an end-to-end approach that takes into account the broader outcome the automated task is supporting.

    Remember, your business has rules on security, compliance and governance–all critical to the success of an RPA deployment. Are you considering these factors and all the affected stakeholders across the entire organization when trying to fix a problem within one department?

    There is a place for business users building automation, but to get to real scale and potential for transformation, IT needs to work together with stakeholders across different lines of business to design an end-to-end intelligent automation solution that ties into pre-existing business systems and accounts for their sophistication/complexity, while also accounting for where humans need to be involved.

    Attended or Unattended?

    RPA is the hottest trend since cloud computing, and it boils down to two basic forms: attended and unattended. Simply put, attended requires the help of human interaction on the desktop, while unattended takes over tasks without the need to interruption. Is one better than the other? Not necessarily. In the right situation, both can generate quick wins and task automation to assist the workers.

    Consider the desktop of call center agents. These workstations are known for having deep integrations with legacy technology that are geared toward repetitive, scripted and manual tasks. This provides the perfect opportunity for attended RPA deployments to address these types of functions that take place across the entire call center, delivering better customer outcomes and generating business value.

    There is a time and place for both attended and unattended solutions–it is important to understand which capabilities the RPA vendor is offering and how they will address the needs of your current situation. In many situations, attended can be an unexpected source of ROI opportunity, as the total volume of work automated in environments such as contact centers, can far exceed the amount of non-human assisted work that can be automated.

    RPA Will Solve All My Problems

    RPA is great—I wouldn’t be writing about it if it weren’t. But it’s not the end-all digital investment automation platform; it’s part of an organization’s digital transformation strategy and can create serious business value when used the correct way. Automating and streamlining the complexity of multiple, long-running processes that interlink internal and external systems, machine and human work, bespoke software and third-party apps and systems is not what RPA was created to do. That’s the work of a BPM solution  supported by  Case Management. Businesses need to realize how RPA fits into their overall digital transformation strategy and where specifically it can be applied across their organization to realize its true benefits.

    You’ve likely heard of the old saying, patience is a virtue. Let’s apply that to RPA deployments. There is so much promise and opportunity for RPA within the enterprise, but the major ROI numbers that some expect may not be there–at least not yet.

    Finding the right opportunities for RPA technology within the organization, getting company buy-in and developing and deploying the solutions all take time to create an impact. Businesses need to approach their projects with a strong understanding of where RPA can be applied and what it does well, like automating high-volume, low-complexity, rule-based tasks within an end-to-end enterprise process automation strategy, to realize the business benefits it brings.

    — Nolan Greene

  • A New Paradigm in Container Security with Service Mesh

    A New Paradigm in Container Security with Service Mesh

    Containers have been around for some time, but only recently has the technology become mature enough for large enterprises to adopt it both on-prem and in the cloud. The obvious benefits for developers include the ability to dramatically reduce the time to production and the ability to lower the operational burden. This is further supported by implementing a service mesh which provides fast, reliable and secure communication among containerized infrastructure.

    The Rise of Kubernetes and its Impact on Security

    Kubernetes (k8s), introduced late 2014, bring container orchestration at large scale to both cloud and on-prem environments. The benefits of using k8s is clear and stipulated mainly due to the ability to become agnostic to the infrastructure. An application based on k8s can easily scale out from your datacenter into multiple cloud providers, and back. However, the challenge with Kubernetes is it creates a huge problem for security operations. Trying to enforce micro-segmentation on a large k8s cluster using traditional firewall models is not feasible.

    As the move to containers rise, so will container-based cybersecurity attacks. Criminal hackers will always be on the hunt for new ways to exploit this area. As container operations are inherently dependent on the development pipeline (CI/CD) so should be their security attributes. Any other interaction point will be an afterthought resulting in slower operations or reduced security. Organizations moving to containers and k8s should act preemptively due to on-going and increasing complexity driven by scale.

    Enter Service-Mesh

    Implementing a service mesh layer ensures that communication among containerized infrastructure services is fast, reliable and secure. It is a configurable, low‑latency infrastructure layer that handles service-to-service communication. It is responsible for the reliable delivery of requests through the complex topology of services that make up a modern, cloud-native application. In practice, a service mesh is typically implemented as an array of lightweight network proxies called sidecars, that are deployed alongside application code without the application being aware. The mesh provides critical capabilities, including service discovery, load balancing, encryption, observability, traceability, authentication and authorization, and support for the circuit breaker pattern.

    Criteria and Security Best Practices

    • Pre-scanning of images isn’t enough. Enforcing runtime micro-segmentation for container services should be addressed at runtime as they introduce an easier, more prominent attack vector.
    • Easy integration with development pipeline. As noted, the alternative will impact business agility or security quality.
    • Micro-segmentation should be decoupled from the infra and network as otherwise, it will require some kind of application to IP/PORT/protocol mapping.

    Implementing security built for k8s environments and running both on-prem and in the public cloud that uses Istio, a well-known open-source service mesh developed by Google, is recommended. Vet solutions connect directly to the code pipeline with enhanced security capabilities, such as the signing of artifacts that will be deployed to the k8s cluster in the code stage with the ability to generate signed identities. Instead of looking at machines and IPs when workloads boot up, the administrator will see that services will be able to control where they run.

    This will also enable you to intuitively create identity-based micro-segmentation. This is an advantage since micro-segmentation was always a complicated task to achieve, requiring administrators to map an application to the network while keeping track of what is running in the environment.

    With identities and a service mesh layer, it is only necessary to focus on your business logic. The fact that identities were generated during the CI/CD pipeline, provides the ability to control the application without the need for service discovery. Identities are represented by their business-context names, originating in the CI/CD pipeline, so the operator can easily group them together and assign policies at different layers of granularity.

    K8s is a great technology that enables the harvesting of containerized environments. Adopting it can accelerate your organization’s digital transformation dramatically, but it can also require you to adopt new methods of application security. Contracting with a k8s-focused security organization will allow the business to accelerate its migration to cloud-native applications on-prem and in the cloud, in a most automated fashion.

    By providing the needed security and operational capabilities, k8s clusters admins are afforded an enterprise-level micro-segmentation solution with policy orchestration, single pane of glass to your clusters in a multi/hybrid cloud deployment and the ability to connect between containerized clusters and legacy applications, as well as other resources that are external to the cluster.

    — Ran Ilany

  • When Should You Go for Microservice Architecture?

    When Should You Go for Microservice Architecture?

    There is steady increase in the adoption of microservice architecture style since 2014 (Figure 1). Microservice architectural style structures an application as a collection of loosely coupled services that implement business capabilities.

    An architecture style is a family of architectures that share certain characteristics. For example, N-tier is a common architecture style. Architecture styles generally don’t dictate the use of particular technologies, but some technologies are well-suited for certain architectures. For example, containers are well suited for microservice since they go hand-in-hand for the deployment.

    In this article, you will learn about the typical considerations, challenges and benefits of the microservice architecture style.

    Figure 1: Microservice Google Trends.

    While microservice applications are built as a set of modular components, they are easier to understand, simpler to test and effortless to maintain over the life of the application. It enables organizations to achieve agility and be able to improve the time it takes to get working enhancements to production.

    It also enables the continuous delivery/deployment of large, complex applications and the evolution of its technology stack. Having detailed out on the benefits side there are quite a bit of challenges as well, which we will discuss in the next section.

    Constraints Imposed by Microservice Style

    By adhering to the following constraints imposed by microservice architecture style, what we get is a system where services can be deployed independently, faults are isolated, frequent updates are possible and it’s easy to introduce new technologies into the application.

    • Each service represents a single responsibility.
    • Every service is independent of the others.
    • Each service needs its own JVM (or equivalent) so instances are isolated.
    • Data is private to the service that owns it.
    • Services do not share data and need to implement queries that need to retrieve data owned by multiple services.
    • Well defined mechanism to implement service-to-service communication and scheme to deal with partial failure.
    • Requests that span multiple services and require careful coordination between the teams.
    • Testing to cover the interactions between services.
    • Deploying and managing a system comprised of many different services.

    Consider Challenges Versus Benefits

    Each architectural constraint has its own associated challenges/benefits, so it’s important to understand and balance both. Before adopting any style of architecture, you have to ask yourself if the benefits outweigh the challenges for the application you are considering.

    In the table below, we take a look at the benefits versus challenges of microservice style applications.

    Benefits Challenges
    Enables the continuous delivery and deployment of large, complex applications. There is an additional complexity of creating a distributed system.
    Improved maintainability: Each service is relatively small so it’s easier to understand and change. Implementing requests that span multiple services is more difficult.

    Maintaining data consistency between service(s) is a challenge.
    Better Testability: services are smaller and faster to test. Testing the interactions between services is more difficult.
    Better deployability: services can be deployed independently. Increased operational and deployment complexity of deploying and managing a system comprised of many different services.
    Each team can develop, test, deploy and scale their services independently of all of the other teams. Implementing requests that span multiple services requires careful coordination between the teams.
    Improved fault isolation. Inter-service communication and dealing with partial failure implementation is challenging.
    Eliminates long-term commitment to a technology stack. Overhead of multiple JVM runtimes (or equivalent) and increase in memory consumption needs to be taken care of.

    Scenarios Where You Can Consider Going for Microservice

    Consider an scenario where you are developing an enterprise application that must support the following characteristics:

    • Support for different variety of clients that includes desktop browsers, mobile browsers and native mobile applications.
    • Expose an API for third parties to consume.
    • Integration with other applications via web services or a message broker.
    • Run multiple instances of the application on multiple machines to fulfill scalability and availability NFR requirements.
    • Adopt emerging technology stack.
    • Planning to go for continuous deployment pipeline setup for the application.

    From the requirements above, you can very well say the application is 100% fit for a microservices architecture.

    You have to define an architecture that structures the application as a set of loosely coupled, collaborating services. Services can communicate using either synchronous or asynchronous protocols. Services can be developed and deployed independently of one another. Each service has its own database in order to be decoupled from other services.

    Let’s look at some of the typical scenarios where you can consider going for microservice style of architecture:

    1. Monolithic application migration due to improvements needed in scalability, manageability, agility or speed of delivery.
    2. Re-platform a legacy application by transforming functions/modules to microservices.
    3. Rewriting legacy application to modern languages, technology stack to meet the demands of modern business.
    4. Individual cross-cutting services that are independent in nature. For example: encryption services, authentication services, etc.
    5. Independent business applications or services reused across multiple channels. For example, payment services, login services, flight search services, customer profile services, notification services, etc.
    6. Commonly used enterprise applications. For example, time tracking application.
    7. Scenarios where a service provider makes computing resources and infrastructure management available to the customer as needed. For example, forecasting services, price calculation services, prediction services, etc.
    8. Back-end services for responsive client-side web application where data could be coming from multiple channels or different data sources.
    9. Highly agile applications or applications demanding speed of delivery or time to market or innovation pilots, etc.
    10. Applications that got polyglot, multi-language, cloud application development.

    Make the Choice

    Before going for microservice architecture, make sure you know the challenges and benefits. Otherwise, your application design could end up with a design that fits the style but does not achieve its full potential. Adopt microservices only if you see enough value for your applications. Also note sometimes relaxing a constraint is better than insisting on architectural purity.

    — Karthikeyan Shanmugam

  • Harnessing the Power of Developers

    Harnessing the Power of Developers

    In our fast-moving digital economy, organizations across just about every industry are faced with a stark reality: to remain competitive and deliver the best possible customer experience, they must evolve continuously. The fintech industry is a great example of this–the sector’s growth and focus on excellent customer experience has been driven by external developers rather than incumbent players.

    For many companies, particularly in the service industry, improving customer experience now means developing new apps and services that will maximize the customer journey, engender deeper trust and secure customer relationships for the future.

    The aim is to continuously produce innovations that create delightful customer experiences, which in turn will drive ROI. In its 2019 Technology Industry Outlook, Deloitte highlighted the fact that the “democratization of innovation” is set to be a key strategic factor in the coming years. As cloud infrastructure makes it easier to innovate at speed, “medium and small enterprises can harness powerful capabilities once limited to a select few.”

    That might be a new SaaS application, for example, or the implementation of emerging tech such as AI or VR–anything that helps the company provide a more efficient or accurate customer service. A simple example could be integrating complementary, adjacent solutions to add additional value to end customers.

    Companies need to innovate in order to create unexpected functions and capabilities that will delight. But how do companies go about making that a reality? In a nutshell, there are two groups that can help you innovate: your own developers and the wider developer community. Having the right people involved and the right programs in place for both of these channels will allow you to scale and deepen your innovation more effectively–and ultimately create experiences that your customers cannot live without.

    The more you engage with developers inside and outside your organization, the faster you can accelerate your time-to-market for new innovations and, crucially, ensure that those innovations meet real customer needs.

    In-House Developers

    Building an effective in-house development team to meet customer needs is about more than just hiring the right computer skills. Companies need to aim for a fearless, fail-fast culture that lets teams go above and beyond in answering customer needs. This need not apply to all developers–there will be some focused on refining the core product set, for whom fail-fast may not be the ideal working model. However, in the pursuit of unexpected offerings that will delight customers, a healthy attitude to both fail-fast and conventional development is essential.

    Often, organizations are held back from being the best they can be by ingrained, deep-rooted cultures. The power of “the way we do things ‘round here” can be hard to crack, but good leaders must help staff buy into change as an ongoing process that will help them and the business to succeed in the long run.

    That’s where the concept of experimentation and fail fast comes into play. In these environments, developer teams are given permission to try things out without having to go through multiple rounds of approvals and senior buy-in. That means they can go from an initial idea to a fully testable concept in, say, weeks rather than months or years. It also means they can run their blue-sky ideas through a crash course of reality, bumping up against unforeseen problems–what some people might consider failures, but I think of more as learnings–without being penalized for it.

    When developers are given this permission to get ideas off the ground quickly, the benefits to the organization are obvious. Improved access to innovation; a greater chance of discovering more effective ways of working; and a supportive, forward-looking culture that helps to enthuse non-technical staff as much as developers themselves. The same is true for customers–they will benefit from innovations that improve their experience, surprising and delighting them with the idea they didn’t know they needed, but which exactly addresses their needs.

    Practical Considerations

    There are other key issues to address, particularly senior buy-in and customer-centricity. First, it’s essential that developers know their work is valued by senior leadership. Senior leaders need to trust their experts and not just allow but also champion the fail-fast model if the company is to benefit from the successful innovations that come out of it.

    Second, developers need to work in close communication with customer-facing teams to ensure their efforts are directed toward innovations that will translate into improved customer experiences. In other words, it’s important developer teams innovate with the company’s core goals in mind–not just for the sake of it, however tempting that might be.

    There are also pragmatic issues to be addressed, such as ensuring your APIs or SDKs are well developed, and there are good documentation/guides to help use these APIs. This is required for both in-house and external developers.

    External Developer Networks

    Having a wide network of development partners can also help rapidly scale innovation and deliver solutions that exceed customer expectations with speed. By building a community of great minds around the mutual benefit of market-ready innovation, organizations can unlock far more valuable projects, with positive effects for them, their customers and their partners.

    An effective developer partnership works on the basis that there is a symbiotic relationship where each side has something to offer to the other. For example, the developer partner may come from a smaller company who has specific expertise and can benefit from the financial support and expertise of the larger organizations. While both sized organizations can benefit from the diverse thinking, agility and challenger mindset of the developer partnership. Whether you’re a large company looking to fast-track customer-centric innovation or a small company with a great customer-centric idea that needs a boost onto the next rung, a healthy developer network can be highly beneficial.

    As an example, the accounting industry is currently undergoing a major cultural change, with accountants increasingly seeking to provide customers with value-add strategic services to maintain their competitive advantage. Technology is driving a consultancy-first service model. Accountants have access to more data than ever before, and as a result they can help clients to make effective business decisions based on new analytical insights.

    However, making the jump to effective consultancy requires new tools and innovations to turn raw information into usable insights. By working in partnership with a network of cutting-edge developers, accounting software providers can help to conceive, road-test and launch user-friendly solutions.

    The value of developer networks is also evident as you look at domain specific needs that might span a particular industry or a particular region. One company may not have the expertise or bandwidth to do it all–but by building effective communities of developers, an organization can expand their reach to tap into niche areas of expertise. It also gives the company the ability to deliver innovations more rapidly. Every company has a limited set of resources, and additional developer networks can help scale this innovation. For the developers, the value lies in reaching markets and customers they may not previously had access to.

    Taking Innovation Out of the Lab and into the Market

    Whether your company uses in-house developers or a partner network (or both) the core goal should be to hone ideas into realities through collaborative working practices. When developers and customer-facing teams understand one another, they can achieve great things.

    It’s an exciting time in the industry. Technology is evolving faster every day, solving more problems and opening up new possibilities. For businesses of all varieties, harnessing the power of developers should be a core priority. It’s not a question of pursuing innovation for the sake of it–it’s about taking practical steps to ensure you maximize the positive impact of tech on your customer experience.

    — Shivani Govil

  • Data Monoliths Go Cloud-Native with Kubernetes

    Data Monoliths Go Cloud-Native with Kubernetes

    Breaking monolithic applications into independently scalable services is the core of the cloud-native application model. By making cloud-native apps easy to deploy and manage, Kubernetes has contributed to a big change in IT practices.

    Initial cloud-native conquests started with stateless services and quickly grew to encompass applications that manage data. Database management systems have long relied on centralized servers, but cloud-native architectures are now reaching even the largest monoliths. Among the most interesting cloud-native converts are analytic databases, which hold data arranged in storage formats optimized for extremely fast analytic queries over billions ,or even trillions, of rows.

    The transformation of databases to cloud-native models has profound implications for data-driven applications. Intersecting trends in analytic database architecture and Kubernetes promise substantial benefits to the users of such applications as well as companies that operate them.

    The Rising Importance of Analytics

    Over the last decade the quantity of data that businesses generate has decoupled from the size of the business itself. Ad networks, website analytics and service observability applications often ingest 1 million events per second (or far more) and may have tables running into trillions of records. Businesses need to analyze this information rapidly–for example, to diagnose faults in cloud applications or to retrain models for displaying online ads. Flexible, real-time analytics have gone from nice-to-have to business critical.

    Meanwhile, new analytic engines have emerged that eliminate dependencies on specialized hardware and complex networking required by legacy data warehouses. Druid, ClickHouse or Elasticsearch run on commodity hardware and depend on basic compute, network and storage. Everything else, such as high availability (HA), is built into the application. Newer analytic databases offer built-in horizontal scaling using fleets of server processes. The server processes themselves typically run well in containers.

    With these changes, analytic databases are increasingly cloud-friendly if not cloud-native. It is a short hop to deployment on Kubernetes.

    Kubernetes Operators for Databases

    At the same time, as analytic databases have shifted their operational model, Kubernetes itself has become more welcoming for databases.

    One particularly important innovation for the database is custom resource definitions, or CRDs, which extend the Kubernetes API to define new types of manageable resources. Using CRDs makes it possible to have a single definition that shows the desired state of a complex application such as a data warehouse.

    Another addition is Kubernetes operators, which manage custom resource definitions. Operators track custom resource definitions and take actions when they change. They can also track other information that might be relevant to managing the custom resource, such as changes to pods, storage or even related cloud-native applications. It’s like having a human operator who is constantly watching the application, except the rules are programmed into software that runs inside Kubernetes itself.

    The operator pattern is a perfect match for analytic databases, which are complex distributed applications and may require definition of dozens or even hundreds of individual Kubernetes resources in a running system. Operators encapsulate these details as well as important procedures such as upgrade, scale-out and other operations that require careful manipulation of resources to prevent outages.

    It’s not surprising operators are popular with database developers. The OperatorHub.io registry for database operators alone has 19 entries covering databases of every kind, including analytic databases such as ClickHouse and Elastic. Many more exist outside the registry.

    A Cloud-Native Way to Consume Analytic Databases

    New data warehouse architectures and Kubernetes operators means users can now use analytic databases very differently from legacy data warehouses. Our work on ClickHouse, including development of the ClickHouse Kubernetes Operator, prompts two observations.

    First, operators make spinning up analytic databases simple and fast for all users. Amazon Redshift kicked open the door to data warehouse-as-a-service in the public cloud. Kubernetes operators enable similar capabilities not just in public clouds but any Kubernetes cluster. Individual developers can cycle through dozens of configurations in a single day while testing new analytic applications. Thanks to quick pod spin-up new analytic databases can be running in minutes or even less.

    More profoundly, users can use Kubernetes to move to an on-demand model for analytics. Rather than forcing all applications to compete for resources on a single shared cluster, each application can have an analytic database tuned to its specific requirements. Analytic databases can integrate tightly with applications in the same locations where those applications already run. Businesses can scale databases flexibly to deliver quality analytics at exactly the time they are needed and at reasonable cost. This is the heart of the cloud-native programming model.

    Modern database architectures combined with Kubernetes features, such as the operator pattern, are creating a new path to manage enormous volumes of data. For readers who are hesitant about managing data on Kubernetes, it’s time for another look.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Robert Hodges

  • Learning Production Troubleshooting Techniques in Microarchitectures

    Learning Production Troubleshooting Techniques in Microarchitectures

    One of the great things about our industry is the ability for anyone to get started doing development with a minimal amount of tools. Anyone can learn to do development with just a computer of some sort, a text editor, a tutorial and their brain.

    In fact, the initial basics of how to think through a development problem don’t require a computer at all, reducing the cost of getting started even further. Having such an easy way to get started that doesn’t require a lot of tools lowers the bar for entry, enabling more people to join in on development teams and contribute new ideas and perspectives, which enriches the entire industry.

    Getting into containerization and building modern microarchitectures is a slightly higher bar, but only slightly higher. There are a number of wonderful tutorials out there for getting started with Kubernetes, Docker, OpenShift Origin and other free and open-source tools.

    The only differences in getting started between general software development and microarchitectures specifically are: needing a computer that has the processing power and memory to run containers, and enough of a grasp of a development language to have a basic application to play with. Even that last difference isn’t as crucial because there are so many sample applications people have built for the express purpose of teaching new folks to deploy microservices onto various systems.

    However, when it comes to learning from the DevOps, SRE or ops side of managing Kubernetes deployments or other modern architectures, there’s no solid free tutorial I could find that dumped a bunch of broken sandboxes in front of the student with the express purpose of building their troubleshooting knowledge.

    To gain insight into understanding how to bring microarchitectures back from failures or fall-overs requires access to larger applications and clusters and more senior, or more experienced, mentors–unless you are lucky enough to be able to build large interconnected applications and send traffic to them regularly enough to cause failures.

    This lack of strong tutorials and a requirement for more resources makes breaking into DevOps or SRE work on microarchitectures particularly difficult if no one is willing to take a chance on a less experienced candidate. I want to see that change. We are missing out on hearing from new voices, people who may look at a problem from a different perspective or who may have unique solutions to problems we all face.

    I decided to create a repository full of deliberately broken Kubernetes builds for new practitioners to build progressively more advanced troubleshooting skills. The basic idea is to build a pyramid of environments. The base level should be easy to understand and figure out. These environments will be bare-bones simple, with only one cause for failure.

    We need to build the baseline recognition of common causes in isolation. Many of us that have debugged production systems have an almost unconscious grasp of when a common cause has appeared simply because we recognize that faint signature in the pattern of the data we’re analyzing. If you’ve never seen that signature, though, and you don’t have a mentor with you on these debugging challenges, learning to find that signal in the haystack of your data is much harder. These initial environments should make it easier to learn what those signals look like.

    The next level of environments would have some additional factors unrelated to the actual problem causing additional logs, additional spikes in traffic or additional activity. In short, these environments would have noise. We can add another layer of complexity for some of these environments by adding another service or application that runs just fine on the same cluster. This level would train the ability to rule out sections of a system that aren’t broken, to identify systems that aren’t interconnected and to work on one system without disturbing the rest of the environment.

    Further environments up the pyramid would have multiple contributing factors, red herrings or complex structures. At this point, some of the environments will need to be hosted outside of the local system. I’m still debating how to do these environments so the tutorial could still be free and easy for new folks to work with–I’m open to suggestions.

    Ideally, there would also be a written guide with answers and tips a student wouldn’t look at until they’ve solved the problem on their own, but they certainly can look up and learn regardless of whether they solved the problem. After all, troubleshooting isn’t a closed-book exam by any stretch of the imagination.

    So how do you start? Find the repo at GitHub. It’s free–open source with an MIT license–and open to contributions. There’s a contribution section of the README on the repo that explains how to get started if you want to send commits. If you want to try using these environments yourself and practice or learn troubleshooting skills, the initial environments are built on Minikube, a tool to run small Kubernetes clusters on your local machine. Follow one of the many excellent tutorials for getting Mikube up and running on your local machine, then follow GitHub’s tutorial on cloning down a repo.

    Alternatively, you can download the repo directly. Once you’ve gotten Minikube up and running and you have a copy of the repo on your local machine, follow the instructions in each environment’s directory to deploy the environment to your local cluster, and start trying to figure out exactly what went wrong.

    Want to help make this project a reality? I would love to have more contributors! In addition to being available online for help, I’m going to be at KubeCon North America 2019 in San Diego. At the conference, I’ll have a Hackers’ Corner set up in LogDNA’s booth for most of the conference, and I’ll be sitting there working on this project, among others, waiting for you. Bring your ideas, your stories, your broken environments, your laptops, your willingness to write docs, your Kubernetes knowledge, yourself—anything that can help enrich this project is welcome.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Laura Santamaria

  • How In-Memory Analytics is Solving Data Woes

    How In-Memory Analytics is Solving Data Woes

    The importance of data to business decisions and operations cannot be understated. On a daily basis, 2.5 quintillion bytes of data is generated. Managing that much data so the necessary intelligence is easily accessible or to make sure insights are not missed, can be incredibly challenging. Luckily, technology does exist that can help to significantly ease the strain of this deluge of data: in-memory analytics.

    Initially adopted by the finance industry—where complex, on-the-fly decision-making is required–in-memory analytics is incredibly adept at handling the analysis of substantial amounts of data very quickly. And as data more ubiquitous, and the development of public cloud services has made specialized hardware affordable to companies of all sizes, in-memory analytics technology is being applied in a variety of verticals from retail, to supply chain to healthcare. In addition, given in-memory computing enables a range of analytics features that are very useful to businesses of all kinds and extremely difficult to achieve with disk-based platforms, the technology is really poised to expand rapidly.

    Slice and Dice Any Which Way

    One of the hardest parts of being a data professional today is leveraging all the data at your disposal. Unfortunately, business intelligence (BI) pros are often constrained in terms of what portion of the company’s data they can and cannot analyze depending on pre-defined data queries and the limitations of their BI tools.

    In-memory analytics, however, enables analysts to explore data without any of those limitations. This creates an experience of train-of-thought analysis that helps businesses make deeper analyses and uncover valuable new insights into their market and their own performance. This means retailers for example, can have a global, precise view of the market and cross-reference data from sales numbers, client receipts and loyalty programs, to develop a better understanding of what customers expect from them.

    Figuring Out “What If?”

    Due to the vast amount of data involved, and the limitations of physical storage discs, the calculation of the impact of any business decision on complex KPIs can take a long time to resolve.

    By utilizing in-memory analytics BI pros can continuously compare and contrast “What If” scenarios—such as running stress tests for financial institutions or setting up disruption test cases for supply chain businesses—with a clear, immediate view of the potential outcomes, so they can be as prepared for the future as possible. This means businesses are able to limit the amount of times they are caught off guard or mitigate the impacts of any unexpected turns.

    Up-to-Date Data

    With how rapidly the world moves today, having data that is as current as possible is pivotal. Static datasets only updated once or twice a day can lead to significant, costly misjudgments in some cases. In-memory analytics on the other hand can support a continuous stream of data from multiple sources so analysts always work with the most relevant set of data. This means supply chain businesses, for example, can have real-time oversight over operations to better cope with any disruption that occurs and ultimately save millions of dollars in potential losses.

    The boom of data science has opened a new field of use cases for in-memory analytics, ranging from the validation of new algorithms derived from ML/AI to industrializing the building of new analytics views and dashboards. If you have invested heavily in data science, in-memory analytics could be the pivotal cog in generating the maximum output from each piece of your organization’s data infrastructure.

    — Antoine Chambille

  • Five Tips to Approach DevSecOps Training

    Five Tips to Approach DevSecOps Training

    Developers are on the front lines when it comes to protecting their organizations from cyberattacks. As we’ve seen with the hacks of Monster.com or the Fortnite vulnerability, 2019 has been a trying year for organizations who have failed to protect their applications and websites.

    With an increase of cyberthreats today, it is vital that organizations and developers incorporate standard security protocols within DevOps, a practice known as DevSecOps. Whether exploring these practices around security training, skills, metrics and more in the industry, it’s important to understand how each group fits into the security process.

    Here are five top tips for security training to successfully implement a DevSecOps strategy.

    Recognize Security is a Shared Responsibility

    Many departments throughout an organization make the mistake of thinking the responsibility of keeping their organization secure lies solely with the security department. However, with nearly 200 cyberattacks happening every hour, it’s important everyone gets on board in protecting applications and networks. As the creators behind our modern-day websites and applications, it’s especially important for developers.

    DevOps and security teams need to be taught how to approach security jointly. When approached separately, it is impossible to achieve DevSecOps. Without DevSecOps, the applications and websites used by consumers everyday will not be secure, and there could be security, quality, legal and reputational implications. Security is now being seen as a fundamental aspect of a consumer’s decision, and consumers gravitate towards brands that are more secure.

    Ensure the Entire Development Team Understands Common Cyberthreats and Application Vulnerabilities

    In today’s world, it is especially important for development and operations specialists to understand application vulnerabilities, and best practices to avoid those vulnerabilities. Without this knowledge, developers and their teams won’t understand just how important security is until it is too late.

    Staying on top of the evolving threat landscape, however, can be a daunting task. After all, there is a lot of information out there. The first step to staying informed is to compile a short list of reliable, valid security news sources. Another step could be creating an internal method for the security team to share details on new vulnerabilities with the development teams.

    Unfortunately, hackers tend to exploit vulnerabilities that were accidentally created by DevOps. In order to fight this, DevOps specialists should be made aware and trained in secure programming, secure application configurations and use of secure frameworks.

    Incorporate Just-in-Time Learning into Your Strategy

    Traditional training courses, especially in the security sector, have some challenges. Many universities and free online resources will offer courses online in cybersecurity. While this can be beneficial in continued training, it can take a significant amount of time to complete the course, and some projects require a certain level of knowledge sooner rather than later. Most courses need to be taken well in advance of a project, where the acquired knowledge will then be applied.

    For example, say developer X just started a course in how to protect from vulnerability Y. At the same time, developer X has a project with code that could be vulnerable to Y due in the next two weeks. With the tight deadline, developer X may not be able to get the knowledge they need to be able to protect their application in time.

    In order to combat this, the security industry now offers real-time learning. Otherwise known as “just-in-time learning,” this approach is giving the employees the much-needed information exactly when they need it. These courses usually come hand-in-hand with using application security testing technology, enabling rapid remediation of a detected vulnerability. For example, when a new security technology is introduced to the organization for development purposes, just-in-time learning would involve developers learning to use it on current projects in real time.

    Avoid Making DevOps Specialists into Security Experts

    While it is important for DevOps practitioners to have knowledge of security best practices, it’s important to remember they should be focusing on their task at hand: creating applications. Security transparency is a critical condition for security adoption by the DevOps team, so it is important the technologies developers rely on are transparent. However, security technologies should not distract DevOps specialists from development and operation. DevOps specialists should be security aware and applying their best security development practices.

    The technologies should test applications and get results to those who need them. DevOps practitioners should then use those results, without having to worry about learning and running sophisticated security detection and protection technologies.

    Remember the Goal is to Educate, Not Embarrass

    Even when using the above tactics, it is still quite possible developers will miss a vulnerability and SAST, DAST, IAST and SCA technologies may discover it first. When this happens, it is easy for DevOps specialists to become embarrassed because their managers see the results first. No matter the experience level, developers will likely at some point make a mistake that can cause a vulnerability–which ultimately gets back to their supervisors and peers.

    Rather than feel embarrassed over the security technologies, developers need to learn how to utilize them to their advantage. For example, modern SAST allows developers to conduct tests out of an IDE and return the test results to the same one. The developer is the only one who will see these tests, and therefore, they can be the one to review and remediate a problem. This process helps eliminate the shame developers feel, and it will instead be replaced with education, security and confidence.

    Cyberthreats are not going away, and all signs are pointing to the fact that they are only going to get more advanced in the future. However, with adequate training and cybersecurity education for development teams, organizations can stay ahead of adversaries and fortify their application security.

    — Joseph Feiman