devopsy.kinsta.cloud

Category: Development

  • Building a Solid Foundation for Microservices Security

    Building a Solid Foundation for Microservices Security

    It’s often true that threats evolve faster than development practices and technologies can keep up with, which can be seen in the current microservices approach to application development. While microservices bring clear benefits to developers by enabling application deployment via modular, independently deployable services, they also open the door to new security vulnerabilities.

    Consequently, these changes in the approach to software development require a change in the mindset of developers regarding how they think about and implement security. Security now must be a core component of each service—security that can withstand the constantly changing nature of these interactive services. Each service must be a resilient and secure unit that is capable of interacting with other apps and services without compromising security. In addition, each service must be careful how it handles data at the edges to ensure every possible interaction can be secured, even when interacting with services that do not meet the required security standards.

    To fully address these security challenges, developers also need to change their mindset on how they approach privacy and security in their code. That means thinking of security from the genesis of the program and from within the code itself rather than relying on the infrastructure to secure the application. This approach enables developers to start their architectural decision-making by thinking of security as the foundation of their code, instead of as an accessory.

    Security from the Start

    In the bigger picture, DevOps teams must have full awareness and appreciation that there is no single approach to security, and that security and privacy considerations must be addressed case by case. Although there can be more, there are essentially five things that developers can do to make microservices security foundational to the development process.

    Compartmentalization

    First, developers must begin compartmentalizing the software development mission by attempting to make their microservices as essential as possible. Still, developers have to be careful not to go to extremes that result in the creation of a swarm of nanoservices performing a single task. The overall goal is to reduce responsibilities to the point where each single process lacks enough information to be a liability if it were to be breached.

    Encryption

    The role of encryption plays prominently in the new mindset for developers, so it’s imperative to utilize it on all of the data that each microservice needs and sending it on a perishable channel. This allows the cancellation of a compromised communication channel, rotation of the encryption keys used and retrying the message on a different channel or to a different node without worrying about a breach.

    A good example of this is a node where control has been lost in a high availability (HA) layout. By cancelling all message queues and resending the messages with a rotated key, it’s possible to continue functioning without worrying that the unresponsive node could be compromised and avoid a full stop.

    Disposable Channel Signing

    Developers will greatly improve their security posture in the software development life cycle (SDLC) by sending disposable channel signing and encrypting data to the services under their control. This will allow them to cancel credentials, and avoid data leakage, even when the control on the node is lost.

    Different Keys

    Developers are beginning to understand the efficacy in using as many different keys as may be needed since storage is a weak point in terms of encryption. The alternative is to use keys that are rotated regularly. This, however, requires going around the persistence solution and re-encrypting a lot of the data, which is not ideal. Besides, having layers of encryption will ensure that a single leakage from the storage keys will not grant access to plain data. Also, it’s important to use different keys for storage and transport, as re-encrypting on read is easy. This process avoids the need to spread keys and keep it tight.

    Key Access

    Finally, developers will benefit from making sure that all keys allow the minimum possible access while keeping in mind that they will need to tailor the time to live (TTL) as much as possible to the targeted use of each key. Although this approach may result in a complex permission schema, it delivers a massive reduction of the attack surfaces while adhering to security best practices.

    Conclusion

    Some of this may be common sense, and what is useful for one developer might not be sufficient for another. Generally, it’s good for developers to regularly challenge these suppositions on every service that they write, but the above formulated guidelines will serve almost every developer well by providing a stable foundation on which to begin their thinking. The universal viewpoint that they should all be taking is that security must be part of the DNA of their software.

    — Horacio Duran

  • One Team’s Microservices Journey and CD Growth

    One Team’s Microservices Journey and CD Growth

    Dev Manager Vadim Madison published a chronicle of his team’s move from a monolithic application to a microservices model. In the process, the tools and methodologies they used for continuous integration/continuous delivery (CI/CD) had to be expanded and even changed out in several stages. It’s a long read, but with the time, as there are not a lot of people documenting the case of full-on conversion.

    One of the more interesting pieces of his work is the software infrastructure they have to build up to cover CI/CD/monitoring of the application. This is a lot more complex than the original application was, and it all must be managed. One of the benefits of DevOps is to make management of these tools easier, and he does talk a bit about scripting it out and implementing APIs to take problem instances out of the pool and determine what went wrong so they can go through the Dev/CI/CD cycle again.

    Load Testing
    One of the graphics showing integration.

    What makes it interesting is where the team found limitations and switched tools midstream. This being a process, Madison at the end alluded to the fact that they have started eliminating TeamCity (or at least limiting what they’re using it for). I think that when most people begin to go high-rate DevOps, or even constant monitored low-rate DevOps, some of the items his team ran into escape their notice until they find an issue. The architectural growth will occur—you are automating more, building more, monitoring more and releasing more. The question is, how far ahead do you look? Until recently, the answer was pretty much, “Not far,” because we didn’t have cases like Madison’s to draw on and see parallels with what we were doing. But that is starting to change.

    CI/CD/ARA vendors can help, of course. Collectively they have a lot of knowledge about the lay of the land, and how people are using the tools to tackle the needs of new architectures. But more public documentation such as Madison’s would be great, too.

    Even though you’re moving at a much faster pace, looking ahead and increasing the steps taken in the CI/CD/ARA space can provide that added bit of resiliency, and monitoring at a level appropriate to your project’s maturity can add that extra bit of “we know, we can recover.” There have been some spectacular failures that were indicated by monitoring, but the volume of data flowing through monitoring was enough to mask it. Today, the best solution is to monitor at the appropriate level and then feed that monitoring back into the system.

    But you’ll be supporting more software infrastructure. The growth of tools is pretty inevitable: As you spread the functionality across machines/instances/containers, you will need to stretch CI/CD/ARA tools to multiple interrelated targets and monitor the overall system and its performance bottlenecks. And for now, the tools chosen will not always be widely known, so a process to bring new Dev/DevOps team members quickly up to speed on the more obscure tools should be on your list of to-dos.

    — Don Macvittie

  • DevOps Debates: Monoliths or Microservices

    DevOps Debates: Monoliths or Microservices

    From the time software came into existence, application software has been booming to leverage the benefit of the computing paradigm. During this cycle, it has transformed decades of design and architectural thinking, and has crossed all aspects of application development and management needs, be it configurability, reusability, maintainability, scalability or security (a good list is of these “ilities” is here).

    While many have been addressed, one “ility” that has turned a lot of eyes and attention in the last few years is deployability. More than that, it is the ability to do so continuously. This requirement has changed the thinking of building software for two main reasons: (a) Businesses have started asking to release software faster than ever, even in verticals unheard of before; and (b) Cloud and infrastructure as code has made faster deployment a easier possibility. When you look at application design with deployability in mind, given the need to be agile, what you end up getting is a microservice.

    Most applications of old style addressed most “ilities” in an architecture but did not consider “continuous deployability” as a core need as well. I call these monolith from a deployment perspective. They are tightly coupled, and for any small change a complete build of the application was necessary.

    Now the question is, Should you really look at microservice as a silver bullet to deploy all software? Should you consider that as the de facto standard to deliver any application functionality? Will applications cease to exist the way we know? Here are my thoughts. As usual, feel free to add yours or write to me personally, as some of you do as well.

    Applications as monoliths will stay as they are

    1. Monoliths have evolved over generations of design and architecture tests and stood the test of time. The focus on modular development is, in fact, a way of creating modules, which are built with separation of concerns in mind.
    2. Monoliths were built to run for a long time. It had addressed some of the “ilities” well before they became considerations.
    3. Applications will remain easier to debug, as most of the debuggable areas are in developer’s hand, instead of having to work with a team.
    4. Applications will perform better, as they can share data in memory; they don’t have to do out-of-process calls to get one task executed.
    5. Applications will be more secure, as they will be a black box from an end user perspective or even a hacker perspective, with little coming out from the system.

    Microservices will be a monolith-killer all the way

    1. Deploy, deploy and deploy to correct your problems away. Fail fast and fix faster is the mantra that you can practice.
    2. It helps bundle two-pizza teams owning microservices and hence embrace better ownership and faster response.
    3. The overall architecture to deliver an end-to-end use-case may become complex, but each piece of the jigsaw puzzle can be handled faster.
    4. In the faster-moving world, business wants to try a concept rather thinking through all the “ilities” ahead of time, and progressively make everything better. Microservices helps abstract all complexities much better.
    5. Microservices will help developers think out of box from day one, while helping business deliver smaller chunks, without having to make the system a “big ball of mud.”

    At the end of the day, irrespective of the approaches, the key is to ensure the applications are managed well, changes are refactored well, the code that is written is readable and maintainable, and significant test assets are created to regress the entire system quickly. So whether it is a large system or microservices, you are able to make any necessary changes quickly and can deploy fast.

    While microservices surely will help this new faster-deployment thinking, I—and I am sure you all—have also seen large monolithic systems that are making delivery at an astounding pace.

    Thoughts?

  • Microservices: Easier Than You Think

    Microservices: Easier Than You Think

    “Microservices are easier than you think and more valuable than you’ve heard.” Those are the words of Ben Gracewood, chief delivery officer at cloud POS and omnichannel retail provider Vend. His company has used microservices to trim a monolithic PHP application, improving performance and changing the way the company approaches updates.

    Approaching Microservices

    Vend is currently used by 15,000 retailers in more than 180 countries. It is aimed at inventory-based retailers—think housewares, fashion, grocery and so on. In addition to the usual POS and payments facilities, Vend handles loyalty schemes, gift cards and online storefronts. The company is based in New Zealand, with offices in London, Melbourne, San Francisco and Toronto. Engineering and DevOps is done in Auckland.

    Its global customer base means there is no maintenance window, and so live updates are essential.

    Vend’s HTML5 and iOS clients include caching and a local database to allow offline operation (the data is subsequently resynchronized), primarily to allow users to keep working in situations where there is no Internet connection. The main system was originally a monolithic PHP application.

    Vend executives had discussed the idea of migrating to microservices for about a year. But when the company’s vice president of Architecture experimented by writing one read­-only microservice that improved performance by orders of magnitude, “This was the ‘a-ha’ moment for us,” Gracewood says.

    Later, when Vend migrated from Rackspace to Amazon Web Services, it used a routing microservice that directed traffic to whichever cloud held that particular customer’s data at the time. That allowed a progressive migration with no disruption for customers.

    Those initial successes showed Vend it should move more seriously in that direction. Nine months later, the company had 12 microservices in production.

    Vend is now adding them at the rate of about two per month, and the monolithic PHP system has been slimmed and surrounded by a cloud of microservices. An Nginix-based routing layer allows existing URLs to be redirected to microservices where appropriate.

    The company has followed DevOps practices from the outset, but its first few microservices were deployed and managed as special cases. Now, both the PHP monolith and all the microservices are within the DevOps tent.

    Adoption Dos and Don’ts

    When it comes to adopting microservices, “It’s never too soon to start,” says Gracewood. “It doesn’t have to be all or nothing,” despite what you might hear from other microservices veterans—everyone has been partway through their journey at some stage.

    Gracewood’s advice is to build your first microservice and do what you have to in the development of your DevOps practice. Then keep iterating.

    Don’t fall into the trap of thinking a microservice has to be 100 percent complete before it can be deployed, he says. Start by developing the minimum functionality required at the outset, and then increasing its sophistication over time. Since a change to a microservice affects a relatively small part of the entire system, the continuous update model is easier to apply to microservices than monolithic applications.

    That point also has an organizational impact: A microservices architecture and DevOps support Gracewood’s philosophy of trusting engineering and development teams to implement the right things at the right time. With monolithic software, a mistake or a poor decision can have wide-ranging repercussions, so considerable oversight is required. But the nature of microservices keeps issues localized, so if an issue does somehow slip through the testing process, it is easier to revert to the previous version.

    Adopting DevOps and microservices does require a certain amount of education, Gracewood notes. Engineering teams must understand the command line, Docker and so on. Making the move calls for “lots of tooling or lots of training,” he says. To that end, Vend has built some custom tools.

    Gracewood’s final tip for organizations thinking of adopting microservices: “Log everything [and] apply metrics to everything,” right from the start. You need to know how people are using your system. In particular, identifying the most heavily used microservices will reveal the areas where efforts to improve performance are likely to yield the best returns.

  • Microservices, Hyper-Complexity, and Automation

    Microservices, Hyper-Complexity, and Automation

    Imagine building a human being one DNA codon at a time. Sure, you might be able to build a pretty awesome individual. But it would take a long, long time. And you’d only be able to build one. Worse yet, if you didn’t get your person exactly right on your first try, it would take you another long, long time to come up with Version Two.

    Unfortunately, this is exactly the position many companies are putting themselves in with microservices.

    Yes, microservices theoretically enable companies to adaptively recombine their digital DNA to build the increasingly awesome apps. But without better automation, they won’t be able to do so very quickly—or very reliably.

    Hyper-Complex DevOps

    Microservices introduce significant additional complexities into the DevOps lifecycle. Each microservice has its own data mappings, APIs, latency/performance dependencies, etc. So, while microservices may offer genuine advantages in terms of adaptability and re-use, they also multiply the total number of discrete DevOps tasks—especially vis-à-vis test/QA—by at least one full order of magnitude.

    This is a serious problem for shops that aren’t sufficiently well-automated, because any task—or, as is much more often the case, any handoff between tasks—that is not well-automated will quickly turn into a bottleneck.

    The resulting adverse consequences of insufficiently automated microservices DevOps can thus include slow delivery, excessive cost, and mistakes. Worse yet, chronic bottlenecking will likely create a pervasive organizational disincentive that fundamentally undermines the microservices initiative itself.

    What Automation Entails

    Thorough, effective DevOps automation is, of course, more easily aspired to than achieved. To implement the kind of end-to-end DevOps automation required for microservices success, organizations have to get three things right:

    • Process mapping. Effective automation requires an exhaustive definition of process. This exercise is not for the faint of heart—because most organizations have many pockets of ad hoc process at various points in the DevOps cycle. Test/QA tends to be especially variable based on factors such as the delivery pressure teams are under at any given time.
    • Automation technology. Development teams often use scripting and other technologies to automate various DevOps tasks. An overly fragmented automation environment, however, can be difficult to orchestrate and manage. So the move to microservices may necessitate a more holistic integrated automation environment.
    • Leadership and culture. A well-automated DevOps process alone is insufficient for microservices success, because it is unfortunately not true that “if you build it, they will come.” On the contrary, a strong leadership commitment is essential for changing the way people work.

    The application economy is putting development organizations under intense pressure to be more productive and responsive to ever-changing market dynamics. Microservices offer one possible way to respond to that pressure. But microservices won’t work without a much more rigorous approach to DevOps automation. IT leaders may therefore want to focus on automation first and microservices second, instead of the other way around.

    Unless, that is, you only want to build one app using microservices—and then maybe build a slightly better one a few months later.