devopsy.kinsta.cloud

Category: Blogs

  • How AI and Microservices Create a Reliable Enterprise of the Future

    How AI and Microservices Create a Reliable Enterprise of the Future

    The Rise of Microservices

    Enterprise adoption of microservices is continuing to rise. Its popularity is so strong that 86% of development professionals internationally expect microservices to become the default application architecture within the next five years. Looking out even longer, nearly all professionals see microservices becoming their company’s default application architecture.

    At its core, a microservice approach centers on the idea of breaking up complex enterprise software into smaller, more manageable pieces, containing everything from the operating system to the platform to the framework to run time and dependencies, packaged as one unit of execution.

    By dividing an application into smaller stand-alone services, each performing a single function and communicating over the network to work together, microservices offer increased modularity and make applications easier to scale and faster to develop. In a world of ever-increasing apps, websites and digital experiences, there are immense benefits to deploying microservices.

    However, the use of microservices is no silver bullet and, for all their benefits, they have their own limitations and challenges. When it comes to deploying microservices, scale, complexity and constant change are the new realities. Microservices architecture introduces complexity with respect to network latency, network communication, load balancing, fault tolerance and message formats. With great flexibility also comes a higher risk of something slipping through the cracks. Managing many services requires understanding deep structures, dependency graphs and more.

    Process scalability is another challenge. An enterprise can run one cloud-based service with relative ease, but simultaneously running dozens of services across multiple clouds quickly becomes a complicated juggling act. In fact, with all the moving parts that come with microservices, it is becoming increasingly difficult to manage these processes manually.

    No wonder that, along with the rise of microservices, we have seen the emergence of auto-remediation/self-healing capabilities. To the untrained eye, the terms auto-remediation and self-healing would seem to be the answer, but don’t be fooled. Many of these features, although making the management of microservices easier, use static thresholds for data points such as waiting time, throughput, message queue size and more. When the thresholds are passed, the system will correct or self-heal as necessary. Although helpful, they’re not necessarily smart or intelligent—and that is where AI can help.

    Advanced Self-Healing and Self-Defense

    AI can help solve some of the hardest problems facing microservices today. Below are a few achievements we’re already seeing AI accomplish:

    • Load Prediction: Understanding the load on the system and acting on it before it happens will allow systems to provision more resources before they’re needed. Provision of resources is not just limited to the network and number of tasks running; AI can trigger customer support representatives to be on hand when—and only when—needed.
    • Decay Detection: Anomaly detection is built into a lot of monitoring services and is efficient at detecting spikes, loads and abnormal behaviors; however, detecting a service decay, even gradual or subtle, is difficult. AI can help systems recognize whether services are reacting differently over time and what the reasons are for this decay. It also has the potential to be more effective in identifying bugs based on studying clusters of microservices having continuous issues.
    • Resource Planning: As the scale and complexity of microservices grows, so does the use of the cloud. The ability to plan ahead using smart resource management could save significant costs while maintaining a stable and robust service.
    • Security: Security is already leveraging AI in many places by discovering attack patterns based on known behaviors. Leveraging these capabilities at the network and database level can improve security for actions done by an authorized person. If you can understand the usage patterns and act on them, you can define a more secure system for all users.

    Benefits aside, using AI is not without pitfalls. For example, despite all its potential, the so-called state-of-the-art AI models are still quite primitive when compared to what they have the potential to achieve. Another issue is how to manage the possibility of bias creeping into AI models to avoid unintended consequences. Technology professionals are already working on ways to solve these complications. However, it will take time to develop the right models, data sets and algorithms.

    The Future

    If the future of application architecture is truly to be dominated by the ever-increasing use of microservices, then those architectures will need to also be dominated by AI. The reality is microservices are simply becoming far too complex for humans to manage on their own effectively. Because of the dynamic nature of their structure, their complex dependencies and their sheer scale, machine learning must be applied to assist with monitoring and management tasks. AI is best suited to meet this challenge and tackle it head on.

    — Martin Henley

  • Should Your Company Use DBaaS?

    Should Your Company Use DBaaS?

    From SaaS to IaaS, companies are rapidly moving to the cloud to gain huge agility and cost benefits. So, is it time for your company to take advantage of database as a service (DBaaS)? Here’s a high-level discussion to help you along your decision path.

    First, it’s important to distinguish DBaaS from simply running a database on a cloud-based server, such as Amazon AWS or Microsoft Azure. While you can certainly gain some cost and agility benefits from running your database on a virtual machine, you’ll still need to provision and maintain the database, that is, you will need server operations expertise.

    With DBaaS, the cloud provider also provisions and maintains the database, eliminating the need for your organization to maintain server operations expertise in-house. DBaaS manages the entire environment—physical and virtual hardware, operating system, networking, replication configuration, etc.—so you can stay focused on software development. With the click of a button you can set up the database and then connect your application with a simple API call. And you’re ready to go.

    The speed and cost advantages of this approach can be huge, but there are also some limitations, so it’s important to assess whether your company’s resources and your project’s requirements align with the current capabilities of the various DBaaS offerings.

    Reasons to Use DBaaS

    Lack of Expertise: As suggested above, if your technical team is mainly composed of software developers who won’t be able to manage the complexity of provisioning a database, you should consider DBaaS. In addition to handling the management of the database, the DBaaS provider will offer additional services, such as automatic backups of your data. In this worry-free environment, you can focus on optimizing your software instead of constantly worrying if the database is functioning properly.

    Short-Lived Databases: If you need databases to serve as back ends for continuous integration testing, DBaaS will allow you to rapidly provision the databases, populate them with test data and then delete them when they are no longer needed. Having the database provisioned for you makes it far faster and simpler to build the required automation for the testing environment.

    A Microservices Design: DBaaS can be particularly helpful, even in a production environment, if you are using a microservices design composed of a very large number of databases. Even if you have the internal expertise, you may not want to invest the time and effort in deploying all those databases. DBaaS can get you into production significantly faster.

    Reasons Not to Use DBaaS

    While the above benefits are appealing, you likely won’t want to use DBaaS in the following scenarios.

    Database-Centric Applications with Complex Topologies: To ensure ease of deployment, DBaaS providers limit the ability to deploy complex environment topologies. This reduces the flexibility DBaaS customers have. For example, let’s say that for disaster recovery, you have a primary synchronous replication cluster in one region and a secondary synchronous replication cluster in a different region, with an asynchronous relationship between them. This relatively complex topology is outside the scope of most DBaaS offerings.

    IO-Bound Workloads: If your database has IO-bound workloads, you may need very powerful hardware, including high-end CPUs, terabytes of memory and very fast storage. In this case, you may not find a DBaaS provider that offers a configuration that will meet your needs.

    Understand Your Options

    If your database requirements align with the capabilities of DBaaS, you now have the ability to eliminate significant operational complexity and develop, test and deliver your applications faster and at a lower total cost. However, make sure you spend time carefully investigating today’s DBaaS offerings, including Amazon RDS, Google Cloud SQL and Azure SQL Database, which all have different features and focus, including deployment models, performance trade-offs, instance types and more.

    If your needs don’t fit current DBaaS offerings, you’ll need to provision your databases yourself. If you don’t have the expertise in-house and can’t afford new hires, a database consultant may be a cost-effective strategy for getting the support you need.

    — Tyler Duzan

  • IoT Security Concerns

    IoT Security Concerns

    The Internet of Things (IoT) is being hailed as the next step in our technological development. It’s hard to argue with that. The abilities it provides are as captivating as they are numerous: smart cars and homes, automatized agriculture, advancements in medicine that help with after-stroke recovery and the list goes on.

    However, while the IoT is truly a huge topic nowadays, we must not let our fascination with it overcome us. It’s too easy to forget about security and privacy concerns when you hear about exoskeletons that can increase a person’s strength and dexterity–trust me, I know. But we can’t afford to forget about them.

    It does seem, though, in our pursuit of the interconnected device environment, the security of those devices was somewhat left behind. Let’s take a look at what the primary IoT cybersecurity hindrances are and what can be done to overcome them.

    When Security Struggles To Keep Pace

    One example of the quantity of devices outrunning their quality safety-wise is smart homes. They are very popular now, and according to the IDC forecast this trend is not going away:

    Data source: IDC Worldwide Quarterly Smart Home Device Tracker.

    While undeniably convenient, smart homes can pose a potential threat to their users. To do their job, they must collect information about those users. And the more interconnected the entire system gets, the more trouble can a single weak link cause.

    Take smart door locks, for example. Some of them come with facial recognition algorithms. That in itself is a pretty big privacy concern. Facial recognition can be used with an alarming accuracy to identify and track individuals. It can have various purposes, but very few of them (if any) are benevolent.

    To add to that, within the smart home system, a smart lock is going to be connected to other devices. Maybe it is safe from cyberattacks on its own, but is the smart thermostat safe? What about the smart water kettle? It only takes one vulnerability to get access to a lot of one’s private data.

    This risk can be somewhat mitigated by users taking certain precautions: installing AV software to protect their devices, updating the said devices regularly to prevent exploits, setting up a VPN on their routers to safeguard all of their internet traffic, etc. But how many users are actually going to go through with all this trouble?

    Physical Problems

    It should be noted that data breaches and cyberattacks are not the only threats the IoT faces. Some problems arise from poor design.

    For example, among other considerations presented in the June NIST publication, a non-descriptive appearance of an IoT device is mentioned. If it’s just a black box with no markings on it to indicate what software and hardware it runs on, updating it (as well as keeping track of its activities) becomes a very unpleasant chore. It is especially true if there are thousands of such devices located on the premises of a company.

    Alternatively, the support of a certain device may get discontinued well before the company plans to replace it. It is also possible the news of its discontinuation can be missed by the company, thus creating a possible vulnerability.

    Sheer Scale of the IoT Makes Threats Bigger, Too

    Another concern with the IoT is it’s so overwhelmingly difficult to maintain secure. If there is the need to install crucial updates on hundreds of millions of devices, the task becomes all but impossible.

    The hundreds of millions of devices example is not an exaggeration, either. Recently, several critical vulnerabilities were found in the VxWorks real-time operating system. Thankfully, this finding was not made by hackers but by a security firm. Some might think it should, therefore, be easy from now on for those vulnerabilities to be patched.

    Here comes the problem. There are 2 billion devices that run on VxWorks. Although not all of them can be affected by the vulnerabilities found, hundreds of millions still can. Moreover, there are difficulties in finding all the devices that need a patch, as it’s not obvious what devices even run on this RTOS.

    Many of the VxWorks devices are used in medicine. To update them, it is required they undergo the certification and testing processes, and that can take a long time. And before all the necessary testing is done, those devices remain vulnerable.

    Making the IoT Safer

    That’s why the approach to IoT device security must become more holistic than it is now. It is necessary for the IoT market itself to develop, as security and privacy issues were named one of the biggest problems hindering the implementation of smart devices.

    According to the recent Deloitte webcast, one of the biggest threats of the IoT today is security measures and systems are not incorporated into the core design of devices and networks. Another top risk is the lack of security awareness and training among engineers. Yet another one is the lack of proper incident response processes due to the wanting skill levels of security professionals.

    It’s obvious the IoT field can benefit immensely from the implementation of DevOps and, especially, DevSecOps. To ensure safety in the IoT, it is not enough to start worrying about it only after the fact, as an addition to other functions. On the contrary, it must become secure by design, and its safety must be a concern on all levels.

    — Dean Chester

  • GitLab Outlines Scope of DevOps Ambitions

    GitLab Outlines Scope of DevOps Ambitions

    At the GitLab Commit event in New York today, GitLab launched an ambitious effort to extend the scope of its DevOps platform to provide a view into the software development life cycle for not just developers but also designers, product managers and business leaders.

    Fresh off raising an additional $268 million in funding, GitLab now plans to create different ways to view a common pool of metadata and metrics generated by the GitLab DevOps platform.

    It may be a while before GitLab can deliver on that promise, but the core concept builds on the dashboards the company already has built for IT operations and cybersecurity teams. Rather than requiring each team participating in the software development process to acquire a separate platform, GitLab should be viewed as a DevOps platform delivered as a single application that can now be applied to everyone, said Mark Pundsack, vice president of product strategy for GitLab, in speaking with conference attendees.

    As part of that commitment, Pundsack noted GitLab is expanding its DevSecOps capabilities to include auto remediation. The scanning capabilities built into GitLab will be extended not only to detect issues based on the cybersecurity policies of an organization, but also, when allowed, automatically remediate routine issues, Pundsack said, adding GitLab also will automatically roll back applications if necessary. Once in place, GitLab will have pushed application cybersecurity much further left than anyone might have expected when DevSecOps was first broached as a concept, he said.

    At the conference, GitLab demonstrated a pipeline graph capability that will provide a foundation for many of these promised offerings. GitLab already can be employed to accomplish tasks such as code quality scanning, container scanning, dependency scanning and licensing dependencies.

    All these features are an extension of GitLab’s existing strategy that aims to rationalize the number of tools organizations need to stitch together to drive a set of best DevOps processes. However, other DevOps platform providers have similar ambitions.

    Thanks to its latest round of funding, GitLab—which also announced its plans to go public next year—is now valued at $2.75 billion. The company has not released any revenue results but does claim to support more than 100,000 organizations using its platform, including Ask Media Group, Charter Communications, Delta Air Lines, Goldman Sachs, Ticketmaster and Nvidia. GitLab also claims to have more than 4,800 active contributors to its open core platform. The company now has more than 800 employees in more than 55 countries across the globe, and there currently are 222 open GitLab jobs posted.

    It remains to be seen how much $268 million will enable GitLab to fend off rivals such as GitHub, which was acquired by Microsoft, and CloudBees, which has built a large DevOps community around the open source Jenkins continuous integration/continuous deployment (CI/CD) platform. Whatever the outcome of this arms race, the massive amount of funding pouring into the platforms soon should take DevOps platforms to places many might not have ever imagined possible.

    — Mike Vizard

  • How App Stores Are Addressing Fragmentation in the Linux Ecosystem

    How App Stores Are Addressing Fragmentation in the Linux Ecosystem

    According to DistroWatch, 273 Linux distributions are currently active, with another 56 dormant and 521 discontinued. While some of these have shared underpinnings, it still makes for an extremely varied landscape for companies and developers.

    It means developers must create multiple versions of their applications to be able to provide their software to all Linux users or just address a fraction of the market. Also, developers require multiple versions of build tools, which inevitably results in significant resource overhead.

    Desktop application distribution is complex across all operating systems in general; in Linux, this is further compounded by such fragmentation and inter-dependencies both in the packaging and distribution of software.

    For example, Fedora uses the RPM packaging format, while Debian uses the .deb format. Moreover, packages built for one version of a Linux distribution are often incompatible with other versions of the same distribution and need to be built for each version separately.

    Developers need to satisfy all these requirements at the same time to be able to successfully offer their products to their users, which can be difficult with limited resources.

    In a way, the choice of distribution and subsequent promotion of Linux software is a chicken-and-egg problem. Initially, companies and developers must decide what parts of the Linux desktop system they want to target. Then, once they have their products available to end users through one or more of the distribution channels, they still have to invest in separate efforts to promote their applications and capture usage.

    Unfortunately, there is insufficient data to assist in the selection and optimization of either the distribution or the promotion process. Linux software developers generally lack sufficient immediate visibility to the usage and popularity of their software.

    Traditionally, there have been multiple dimensions to how creators distribute their software: official websites, where companies and developers offer software downloads on their own domains; code repositories such as GitHub, GitLab and SourceForge; and application repositories in which Linux-based operating systems typically provide software through self-hosted storage locations.

    To achieve the highest level of penetration, software developers will often distribute their applications using some or all of the above methods. Likewise, most users will utilize multiple methods to obtain software they need. While this gives developers more freedom in how they deliver applications to the end-users, it creates additional fragmentation.

    In recent years, there has been significant effort to address the fragmentation in packaging and simplify the distribution of software through the establishment of software centers, in which some Linux distributions offer store-like destinations with a graphical frontend, software ratings, screenshots, descriptions and other user-focused data that allows better search and discovery.

    Currently, there are three of these in the Linux space—AppImage, Flatpak and snaps. They each approach and handle the various aspects of usability in different ways.

    AppImage and Flatpak technologies are architected to primarily support GUI desktop applications. Snaps are designed to work with servers, cloud infrastructure and IoT devices, in addition to desktop applications.

    Spotify, a snaps publisher, is an example of a company benefiting from this approach. On one hand, it requires the Spotify engineering team to maintain both the snap and the traditional Deb package. On the other, the snap version is significantly easier to release, which has also helped with the simplification of the Deb release, as well.

    The primary reason is the ability to use easy rollbacks, which have already been used. This feature removes uncertainty that a possibly broken version of the Linux desktop client application might be pushed out to the users. Spotify engineers first release the snap version, and after ascertaining that it works and behaves correctly, they build the Debian package, too. Snaps allows the Spotify team to more closely follow the weekly cadence of releases they have for the Windows and Mac versions.

    The emergence of self-contained packaging and distribution mechanisms addresses a major problem in the Linux application ecosystem–the difficulty in establishing a successful business model because of the packaging and distribution challenges. The new formats reduce the fragmentation, as they minimize the number of permutations developers need to support for their applications.

    The shift toward centralized application stores that are not specific to any particular Linux distribution offers higher discoverability and better consistency in the user experience.

    Developers need fewer resources to support multiple target platforms, which allows them to focus their efforts on the core product, with the understanding their software will be available to the wider desktop audience, without having to depend on distribution-specific dependencies.

    Since the aspects of new, refined models of promotion and distribution in Linux are still relatively young, we see improvements but there is still opportunity to develop and adopt better, data-driven practices.

    As this field continues to evolve, the store model will help drive growth and usage of the Linux desktop, offering software developers a refined, more tangible experience than ever before.

    — Martin Wimpress

  • How to Overcome Resistance to DevOps Implementation

    How to Overcome Resistance to DevOps Implementation

    The number of businesses adopting DevOps to make software releases happen faster without compromising on software quality continues to grow. Within the next five years, the DevOps market is expected to reach almost $13 billion. If you are among those who’ve decided to make a shift to DevOps, you are likely to experience resistance coming from the departments involved in a software development life cycle (SDLC), and even from end users, at the start of your DevOps journey.

    In this article, we will outline the reasons why your employees may resist the DevOps approach and show how to overcome their resistance.

    About DevOps Approach Implementation

    Among the benefits DevOps brings are the fast provision of new infrastructure, quick and reliable delivery of software updates, fewer post-release errors due to the implementation of continuous testing and the alignment of testing and production environments, etc. DevOps implementation requires efforts in establishing tight collaboration between the development, QA and operations teams, as well as adopting new tools and methodologies, such as containerization, IT infrastructure automation, continuous integration and continuous delivery (CI/CD) tools.

    A new approach calls for project managers, developers, infrastructure management, QA and operations teams to leave their comfort zones and get used to new working conditions. However, each stakeholder has reasons to resist DevOps implementation and requires a tailored approach to overcome them.

    The Resistance of Project Managers

    Why They May Resist

    Project managers (PMs) are the first to face the changes when the DevOps approach comes into force. Their resistance originates from a range of new duties, new responsibility and knowledge they need to add to their playbook, such as:

    • Persuading development, QA and operations teams involved in SDLC in the DevOps necessity.
    • Establishing constant collaboration between developers, the QA team and the operations team.
    • Getting familiar with DevOps-specific tools and techniques (CI/CD, continuous monitoring, version control tools, etc.) as well as cloud computing technologies like AWS, Azure, Google Cloud Provider (GCP).

    How to Overcome Resistance

    PMs should become DevOps evangelists as they will be the ones who will contribute the most to overcome the resistance of other parties involved. A CIO should provide a PM with sufficient training on all DevOps-related aspects. Such training should include the management of the development, the QA team and the operations team in the context of close collaboration between them. It should also provide the PM with the understanding of DevOps methodologies and tools–containerization, infrastructure as code (IaC), continuous integration and continuous delivery (CI/CD), cloud technologies, etc.

    The Resistance of Developers

    Why They May Resist

    Developers may resist DevOps adoption due to the following reasons:

    • They are not excited by the necessity to work with infrastructure engineers closer and deal with operational issues if the operations team is not competent enough to address them on their own.
    • They are comfortable with the technology stack they’ve chosen for a particular project and not accustomed to caring about how familiar the other project team members are with these tools and technologies.

    How to Overcome Resistance

    PMs should embrace their soft skills to establish a mutual understanding between developers and infrastructure engineers. They should also make sure infrastructure engineers can work with the tools and technologies developers may use within the DevOps approach, e.g., Docker, Jenkins, by providing them with relevant training.

    The Resistance of a QA Team

    Why They May Resist

    A QA team may feel concerned about a faster SDLC DevOps implementation, since it entails less time for testing. Test engineers now need to acquire new skills prescribed by automation across the entire SDLC. In particular, they face the need to learn new tools for automated testing (Selenium, Appium, fMBT) and API testing (JMeter, Postman).

    How to Overcome Resistance

    To make QA engineers less risk-averse, QA managers in cooperation with PMs, should provide them with appropriate training (tools for automated testing and API testing) and show the advantages DevOps may bring. An opportunity to increase test engineers’ qualification level, less effort on testing activities than required with manual testing are among such benefits.

    The Resistance of an Infrastructure Department Manager

    Why They May Resist

    In the pre-DevOps times, an infrastructure department manager was responsible for controlling the design, configuration and delivery of infrastructure for development, testing and production. With DevOps on board and the IaC approach implemented, the infrastructure manager may feel as if they lose control over the IT infrastructure provisioning and management. What’s more, the scope of the infrastructure manager’s responsibilities extends greatly since they’ll need to run the infrastructure department as a competency center.

    How to Overcome Resistance

    A company’s CIO should explain the infrastructure manager their new responsibilities upfront. The infrastructure manager should stay assured that they change the area of control, not lose it. Within DevOps, the infrastructure department manager has to transform their approach to work. They must be able to help a PM correctly arrange the development, QA and operations team for each particular project taking into account project complexity and size. The infrastructure manager must be able to proactively deal with the cases when the team members involved in software development lack qualification for a project, and not wait until the lack in teams’ competencies results in the project prolongation, for example.

    The Resistance of Operations Engineers

    Why They May Resist

    With DevOps in place, the operations team will have to coordinate their efforts with developers and the QA team from the very beginning of the project, as well as regularly report to the PM on the tasks completed, which is not the usual work approach. To be able to work in the DevOps environment, operations engineers will also have to get familiar with a range of IT automation, version control and other tools. What’s more, the operations staff’s responsibilities will now include the provision of the information on the IT infrastructure functioning to business and technical stakeholders.

    How to Overcome Resistance

    To help an operations team adapt to the DevOps culture, the infrastructure department manager should deliver training to make operations engineers familiar with new tools and technologies they will need to use, e.g., version control systems (Git, Team Foundation Server), software deployment automation tools (Chef, Puppet), etc. A PM, in their turn, will need their soft skills to increase the operations team’s willingness to work in close alignment with development and QA teams.

    The Resistance of End Users

    Why They May Resist

    It may sound surprising, but even the primary beneficiaries of the DevOps approach implementation may resist the changes DevOps brings. There are two reasons for it:

    • The time available for software acceptance testing gets reduced due to a faster SDLC.
    • Acceptance testing becomes automated, and end users hardly trust automated tests.

    How to Overcome Resistance

    To dispel end users’ DevOps-related doubts, PMs must do their best to earn their trust by establishing strong communication with end users. If, for example, end users know software testing covers critical software functionality and the probability of post-release errors is minimized, their confidence in software reliability and the overall trust in the DevOps approach rises.

    In a Nutshell

    Each party involved in software development has its reasons to be suspicious when the DevOps approach is coming on board. New duties, close collaboration, the necessity to apply new tools and other DevOps-related changes may make project managers, developers, QA engineers, IT infrastructure managers, an operations team and even end users resist the new approach.

    To overcome the resistance, a company’s CIO should maintain open communication with each party involved in SDLC, explain their new duties and listen to their concerns. It’s essential to make sure each team member realizes what changes and advantages DevOps brings into SDLC. To ensure they have the necessary knowledge and skills, appropriate training should be provided to project managers, developers, QA team, IT infrastructure manager and the operations team.

    — Andrei Lipnitski

  • An Inside Look at GitOps

    An Inside Look at GitOps

    At a time when variations on DevOps abound, get ready for the latest: GitOps. This new term promises to bring some order to what can be operational chaos and empower operational teams to leverage best practices learned from their developer counterparts. It’s early days yet for GitOps, but here’s what we think you need to understand about this new facet of DevOps.

    What is GitOps?

    Everyone we spoke with has a slightly different perspective on GitOps, but this is our take: In its most basic form, GitOps is infrastructure and operational procedures defined as code, and living in a source control system, such as Git. Git becomes the single source of truth for what is running in production (or staging, or development).

    Declarative systems, such as Terraform, AWS CloudFormation and Kubernetes help enable this practice; but operations is much more than Infrastructure as Code. True operations requires a way to plan, review and approve moves, adds and changes to the infrastructure. It also requires well documented procedures, or runbooks, to be successful. This is where GitOps differentiates itself. We believe GitOps is the full operations lifecycle of infrastructure management, living as code in a git repository.

    GitOps may be used for managing both application deployments and base infrastructure made up of systems needed regardless of the applications deployed (i.e. networking, security, etc.). Both will follow the same procedures to plan, review, approve and release code into production. Done right, GitOps will let engineers constantly apply updates to infrastructure meet all the needs of the applications in a continuous delivery format. The version control system ensures everything is recorded and visible and an audit trail keeps teams compliant. GitOps will make it easy to revert problematic changes, becoming a single source of truth about what is happening in the system from both the software development and infrastructure perspective.

    What Technology is Required?

    In order to kick off GitOps, you’ll use a Git-based version control system. Then you’ll need the ability to create a declarative cloud infrastructure, using something such as Terraform and Kubernetes, with the help of Helm or Flux. Because both Terraform and Kubernetes are declarative systems, they make it simpler to create and deploy infrastructure as code in a manner that is both replicable and transparent. And you’ll also want to implement a solution such as GitLab that wraps operational procedures around these tools. GitLab, being an end-to-end DevOps solution can manage the backlog of requests, assign those requests to the appropriate subject matter expert to create the change in Git, facilitate the review and approval process, and kick off the execution of the changes using its flexible CI/CD system.

    While it is possible to achieve this sort of workflow by using different solutions to merge and ship code to Kubernetes, GitLab is certainly a straightforward path to GitOps. It allows developers and operations administrators to work from the same source of truth and with a common set of procedures and tools.

    What Are the Benefits?

    In the DevOps world, GitOps is perhaps unique in that it can potentially benefit both operations team members and developers. For starters, GitOps aims to address the sometimes cave-like nature of operations (a single person working largely in isolation) by taking all of the best practices from software development and letting them loose on the deployment side.

    Full oversight, architectural review, project management and all the other pieces of the process are brought together in GitOps in an effort to end ops versus everyone else and get all the stakeholders involved in the process. The basics don’t change–an ops engineer will still create an infrastructure–but that will just be one part of a much larger process that everyone can contribute to and collaborate on. No more worrying about an engineer creating a cloud infrastructure that can’t be replicated; with GitOps everything is tracked making repeatability easy. Suddenly it will also be simple to answer the “what changed?” question because no one will have to hunt for an answer. The bottom line is a successful GitOps program will benefit the actual bottom line because operational processes will be dramatically streamlined.

    Possible Challenges

    With any collaborative effort, change can be tricky and GitOps is no exception to that. It is usually easier to create one-offs and not follow a process, so the first thing to consider about GitOps is it will require discipline from all participants. It will be vital for teams to write everything down.

    At GitLab we live and die by this thanks to our culture of collaboration, but we know it’s not necessarily something that comes naturally to any individual or organization. It will be vital for everyone on the team to record what’s going on in merge requests and issues. The temptation to edit something directly in production or change something manually is going to be difficult to suppress, but the less “cowboy engineering” the better GitOps will work.

    If GitOps sounds like something your organization might benefit from, our best advice is to start small. Pick a tiny area to begin with, get a win and then build on that. At GitLab we call that iteration and it’s worked well for us.

     

    This article was co-authored by Priyanka Sharma, director of technical evangelism at GitLab.

    — Brad Downey

  • To Build a Healthy DevOps Culture, Look to Your Ecosystem

    To Build a Healthy DevOps Culture, Look to Your Ecosystem

    How do you do DevOps?  Here’s a hint: Don’t fixate on finding the right tools or implementing the right processes. Instead, think about the broader ecosystem in which you operate, the culture that prevails in that ecosystem and which parts of the culture you can borrow to help enable your DevOps transformation.

    This is true no matter which type of ecosystem you work in–even one that depends in part on mainframes. In other words, to quote from Rouan Wilsenach, writing on Martin Fowler’s blog, “Even with the best tools, DevOps is just another buzzword if you don’t have the right culture.”

    Keep reading for tips on what this means in practice, and how to build a successful DevOps culture by engaging your ecosystem.

    Tools and Processes in DevOps

    To be sure, tools and processes are a key part of DevOps. We’ve yet to see an organization that achieves continuous delivery successfully without the help of a continuous integration server or a continuous feedback process.

    But the reality is tools and processes are not what ultimately separates DevOps success from DevOps failure. Tools and processes help you to implement DevOps; but on their own, they don’t deliver the cultural priorities that allow DevOps organizations to thrive.

    This is an important point to digest. The market is teeming with tools pitched as critical enablers of DevOps, and there is no shortage of articles about why processes such as shift-left testing or de-siloing your team are important for DevOps.

    It would be a mistake, however, to assume that doing DevOps is as simple as implementing one of these tools and processes. If it were, we wouldn’t be spending so much time talking about DevOps in the first place, because it would be a trivial thing for anyone to achieve.

    DevOps Culture

    What matters above all in DevOps is culture. By culture, we mean the values and viewpoints that shape how your team approaches software delivery.

    If your team members don’t believe in constant collaboration and transparency as cultural values, no amount of continuous integration servers or real-time communication tools are going to enable you to achieve the type of team-wide collaboration necessary to support continuous delivery. Similarly, if your developers, software testers or admins don’t recognize the importance of doing continuous delivery in the first place, having the tools and processes in place to perform continuous delivery is not enough to ensure it actually happens.

    This is why instilling key cultural values is critical for DevOps. This isn’t an exhaustive, universally applicable list of essential DevOps cultural values, but most folks would include the following items in this category:

    • Collective accountability. Everyone on your team should assume responsibility for ensuring that software is delivered on time and meets expectations. Software delivery is everyone’s responsibility in DevOps.
    • A blameless environment in which the team as a whole, rather than individuals, owns the system.
    • In a DevOps culture, all team members should have constant visibility into what other members of the team are doing.
    • DevOps places a priority on minimizing manual effort. It promotes automation over allowing some team members to perform tasks manually in order to protect their turf or increase their individual value.
    • Collaboration and shared roles. In DevOps, no single engineer should exclusively own a particular task. Some people might bear primary responsibility for certain tasks, but at the end of the day, everyone should be permitted, qualified and willing to assist with any task.
    • Support from management for all of the above. Executives need to embrace the concept of collective ownership for IT processes, while also giving IT teams the flexibility to scale and change tools and processes as needed.

    Your Ecosystem and Your Culture

    How do you actually construct a DevOps culture founded on principles like those described above? Part of the answer lies in evaluating your current internal company culture, of course, and taking steps to change it.

    But another important part of the answer lies in your business’s ecosystem. After all, culture is not something that develops only on the basis of internal factors. Culture is influenced to a tremendous extent by what is going on around you.

    So, in order to build a healthy DevOps culture, think about what other companies you engage with are doing. In many cases, you can jump-start your DevOps cultural transformation by borrowing ideas from partners who have already been there.

    To be clear, I’m not suggesting you look only for partners who have won accolades as DevOps-first organizations, or startups that have had the benefit of coming of age when DevOps was already a big deal. Think also about established companies that have deep experience working with an array of different technologies, integrating those technologies and bringing them up-to-speed with modern software delivery practices.

    Those are the organizations that have sophisticated, organic cultures. They’re also the ones that know what it takes to update existing cultures to support DevOps. And they may be able to help you identify the specific tools and processes that will enable DevOps for your business.

    Building a Healthy DevOps Ecosystem

    As an example of what the process of constructing an effective DevOps culture might look like in practice, consider how you might approach this challenge if your ecosystem (like many today) includes a mix of modern, cloud-based infrastructure on the one hand, and mainframes on the other.

    This might seem like a difficult circle to square. Conventional wisdom tells us that mainframes may be difficult to integrate with modern DevOps processes.

    But that is not the case. IBM offers a range of solutions to help integrate mainframes into the rest of your ecosystem. For example, Z Open Development makes it possible to standardize your SCM and build processes for mainframes using modern, Git-based tooling. Likewise, Z Open Unit Test makes it possible to bring modern unit testing to mainframe applications. And you can even test and build mainframe apps on commodity x86 servers using Z Development and Test Environment.

    Of course, integrating mainframe infrastructure into the rest of your ecosystem using tooling like this also requires a culture and attitude adjustment. Developers and ITOps admins must move past the cliché that mainframes have no role to play in modern DevOps shops, or that they are too difficult to integrate into modern ecosystems. That’s not at all true and building a successful DevOps culture that embraces the mainframe means discarding this inaccurate viewpoint.

    Conclusion

    In short, succeeding at DevOps means adopting the right cultural principles to turn a DevOps toolset into an actual DevOps workflow. The good news is you don’t have to invent those principles from scratch. Look for an experienced ecosystem partner or partners who can help you in your DevOps cultural journey.

    — Chris Tozzi

  • 5 Best Practices for Managing Open-Source Components

    5 Best Practices for Managing Open-Source Components

    In today’s connected world, software development environments focus a lot on faced-paced development. Organizations that adopt agile development practices such as DevOps and use Open-Source (OS) software and components to their advantage have a much better chance of keeping up with demand and shorten the Software Development Lifecycle (SDLC).

    However, incorporating OS components into applications does not come without risks, as they often contain bugs or vulnerabilities, and are highly exposed to exploits. In this article, we will cover what is Open-Source Software (OSS) and OS components, discuss the importance of security for open source and cover five best practices for managing open-source components.

    What Is Open-Source Software?

    Open source refers to any software with accessible source code that anyone can modify and share freely. You can head over to the Open Source Initiative (OSI) and see a detailed list of criteria for the definition of OSS. While OSS allows many more modifications over proprietary software, you must still accept the terms and conditions defined in its license.

    Some estimations suggest OSS adoption saves $60 billion dollars for consumers per year, while about 96% of all commercial applications contain some components or elements of OSS.

    The Benefits of Open-Source Components

    Open-source components are software units that meet the definition provided by the OSI. Modern development environments put a lot of pressure on developers to build and deploy applications more quickly. To successfully achieve their goals within short software release cycles, developers frequently use OS components.

    The Importance of Security for Open-Source Components

    OS components are incorporated into the vast majority of software and applications. Even though they allow developers to be more efficient and get more done, they can also introduce new vulnerabilities and jeopardize the integrity of the entire code.

    Since OS components are developed in a distributed system, they are often more vulnerable to exploits than proprietary code. Some of the reasons for this are:

    • Distributed development: Compared to a distributed OSS developed by many individuals, it is much easier to track and manage a code developed in-house and ensure you follow policies and protocols.
    • Fast release cycles: OSS development is much quicker and agile, which means that very often, by the time a new software release passes all the quality and vulnerability checks, it has already become obsolete with the release of a newer cycle. This makes it almost impossible and in most cases useless to run the necessary tests and ensure the safety of using the code.
    • Lack of quality standards: OSS elements do not pass the same quality and standard checks as proprietary code. Thus, unless you evaluate each OS component before implementing it, you can often incorporate low-quality code that is exposed to vulnerabilities, lowering the overall quality of your code.
    • The double-edged sword: When members of the open-source community discover a vulnerability, they publicly share the details of the problem in online forums to help developers learn about the risk and how to patch it. Unfortunately, this solution is often a double-edged sword. While this is a step in the right direction, it exposes the details of the vulnerability to threat actors as well, who can exploit it against systems that have not yet patched the vulnerability.
    • Lack of security measures: The problem with OSS is even greater for smaller companies, who prefer to use it to save time and resources but lack the necessary security measures to ensure the components they are implementing are secure enough to be implemented into the code.

    Now, let’s take a look at the five best practices for OS components management.

    Form a Policy

    Establish a clear policy on how to incorporate and manage OS components. This step will give usage guidelines to your developers and ensure OSS elements are incorporated responsibly without damaging code quality. Additionally, if you are facing problems with OSS elements, this policy can act as a playbook and give instructions to your security teams on how to handle the threat.

    Carefully Choose Software

    It is important to choose the right software for you. While it might be tempting to use many kinds of software and enjoy all their possible benefits, you have to consider your requirements and ensure the software you use can answer all your demands. You should consider the amount of support you require and the risks you might face. One of the most crucial aspects is how secure the software is. A good way to measure the software’s security is the Open Web App Application Security Project (OWASP) dependency-check, a utility tool that identifies project dependencies and checks if they contain any open-source vulnerabilities.

    Track and Update OS Components Regularly

    Track your OS components throughout the entire product life cycle and ensure transparency to find critical security vulnerabilities. Updating OS components versions frequently is typically the best way to deal with bugs and vulnerabilities.

    Forking

    One of the best features of OSS is (as long as you keep up with the policies) you can modify the source code and customize the software to fit your needs. Thus, you can take software intended for specific use-cases and modify it to provide a better solution for your situation. However, this process can be intensive and take effort and time. Thus, before you attempt forking, consult with other community and team members to ensure you have the right idea and execution.

    Use Automated Tools

    Cybersecurity is highly time-sensitive—the first moments of dealing with an attack are the most critical. Automated tools, such as Security, Orchestration, Automation and Response (SOAR), can help you identify threats quicker and make mitigation efforts much faster to minimize and dramatically minimize the attack’s potential damage to your system. You can also use automated tools to enforce policies and check your OS components for vulnerabilities, making the process much faster and more secure.

    Wrap Up

    OSS and OS components have many advantages for developers and facilitate agile development. However, the flip side of their ease of use is the danger they pose in terms of open-source vulnerabilities. To mitigate this risk, you should incorporate the right practices for managing OS components into a clear strategy. This will allow you to enjoy all the benefits of OS components without exposing your application, network and systems to unnecessary risks.

    — Ilai Bavati

  • Why COBOL Still Matters After 60 Years

    Why COBOL Still Matters After 60 Years

    In the world of DevOps, the Agile Manifesto and countless other application delivery frameworks, one has to wonder how an anachronism such as the programming language COBOL (common business-oriented language) has managed to remain relevant. After all, COBOL has its roots back in the 1940s, and was built upon Rear Admiral Grace Hopper’s work on the FLOW-MATIC programming language (a text-based programming language). Hopper, who worked as a technical consultant on the FLOW-MATIC project, is sometimes referred to as the grandmother of COBOL.

    Last century, COBOL quickly became one of the most used programming languages in the world and it is still quite popular, claiming more lines of code in active use today than any other programming language. It offered a huge advantage over the other programming languages of the era. One that can be summed up as portability. From the outset, COBOL was the first popular language designed to be operating system-agnostic. In other words, it could be run on numerous different systems with little change made to the source code. What’s more, as a text based language, COBOL used English words and phrases to make it easier to understand and also made it somewhat self-documenting. 

    While much of the programming world has moved onto JavaScript, Python, PHP and even C and C#, COBOL can not be ignored. It continues to evolve and remains relevant, even in the age of the cloud. Simply put, it still has some chops to it. Take for example:

    • COBOL is a Language of Business: It is still used by the majority of major banks and insurers. The language can be found in widespread use across numerous other sectors, such as retail, healthcare, government and automotive. For most of those businesses, no other programming language has displaced COBOL and it remains as the back-office business language of choice.
    • COBOL is Extremely Portable: From the outset, COBOL was designed to be operating system agnostic, meaning the same applications developed under it could run unchanged across many different platforms. That allows COBOL developers to focus on building applications, instead of worrying about the intricacies of operating systems. That fuels DevOps and agile by allowing developers to test and deploy across a wide range of supported platforms.
    • COBOL Continues to Evolve: It supports and integrates with numerous platforms and works with operating environments such as Linux, mainframes, Windows and UNIX. What’s more, COBOL also integrates with Java, C#, Docker containers, mobile, .NET and JVM as well as Azure and AWS cloud environments. It can be used today to build cloud, containerized and managed code apps. And continues to evolve to meet the challenges of the future.
    • COBOL is Easy to Learn and Understand: It was created with the objective of making it easier for programmers to work with computers and each other. That objective has translated into a language easy to learn and understand by using English phrases and commands. The readability aspects of COBOL make it self-documenting, along with an ease of use not found in other languages. What’s more, the language costs less to maintain and its readability means it is easier to maintain.
    • COBOL is Future-Proof: It has continually evolved and now works with industry-standard IDEs, meaning programs can code with COBOL using familiar tools that support teams. COBOL developers are able to use the language alongside contemporaries, without having to forgo the latest IT tooling and processes.

    While many in the DevOps world sell COBOL short, there are many reasons why the language remains popular today. It continues to meet the needs of business and embraces many of the changes the cloud and new techniques can throw at it. COBOL is even now embracing APIs, micro services, the cloud and distributed platforms, making it a worthwhile tool for business today.

    — Frank Ohlhorst