devopsy.kinsta.cloud

Category: Blogs

  • DevSecOps: 10 Best Practices to Embed Security into DevOps

    DevSecOps: 10 Best Practices to Embed Security into DevOps

    For companies that employ the agile approach, DevOps seems like a natural extension. Traditionally, enterprises started with integration, development and test automation early in the product lifecycle. Gradually, the agile delivery team took care of iterative development and monitoring practices that increase code quality.

    Today, organizations work in unison to bridge the gap between development and operations (DevOps). They want to offer a faster market delivery with the least human interaction. 

    But what about security integrations? Is there a way to mitigate vulnerabilities early in the development lifecycle?

    The answer is DevSecOps.

    Exploring the Concept of DevSecOps

    DevSecOps is short for development, security and operations. It brings together people, processes and technology to pursue a shared objective. 

    The objective of DevSecOps is to implement security decisions on the same scale as development and operations and make everyone in the product lifecycle accountable for security. 

    Why Should You Adopt DevSecOps?

    People adopt DevSecOps because they are seeking:

    •  Modern alternative to traditional security engagement.
    • Transparent collaboration and workflows during development.
    • Security that’s built into the product, not applied at the final stage. 
    • Reduced expenses and faster delivery rate. 
    • Faster recovery speed in case a threat is detected.

    Steps to a Typical DevSecOps Workflow

    1. A developer starts by writing code within a version control system.
    2. Any required change is committed to the version control system.
    3. Another developer analyzes the code to identify any security defect that may weaken code quality. 
    4. An environment is created to deploy and apply security configurations to the system.
    5. Next, a test automation suite is executed to evaluate the newly deployed application.
    6. After it passes the automation test, the application is deployed to a production environment.
    7. This new production environment is actively monitored for security threats. 

    While there is no right way to transform organizational culture, below are a few components necessary to sustain a DevSecOps environment: 

    Let Developers Get Security Right: Developers are responsible for security. Therefore, you must keep them on top of cybersecurity best practices through continuous training and learning activities. 

    Promote an Open Culture: Openness in communication within the enterprise environment can drastically improve development and security. One way to keep information transparent is by using metrics and dashboards wherever possible. 

    Get Experts on Board: It is extremely difficult to transition from DevOps to DevSecOps without the supervision of expert security professionals. Hire people who understand security within the development and operations environment and let them train your DevSecOps team for the big transition. 

    Tempted to embed security into DevOps? How do you ensure that the best practices are followed? We have answers.

    The Best Way to Implement the DevSecOps Process

    Gather a single group of professionals (admins, developers, security engineers and testers) that are aware of your product from start to end. They should know your requirements and should be experts in deploying, monitoring and implementing new changes. 

    Once you have your team ready, here’s what you need to do next.

    Plan

    Planning is crucial. Do not just stick to feature descriptions. Instead, go for detailed user stories that include:

    • Functional and nonfunctional requirements (e.g., security and performance).
    • UI and UX designs.
    • Acceptance test criteria.
    • Threat models.

    Develop

    Start by evaluating your existing practices. Choose the best resources to build a development model in coherence with security guidelines. 

    Build

    Automated build tools can do a lot more than compile codes. Use them to conduct test-driven development, enforce quality standards and ensure that the best security practices are implemented through static code analysis. 

    Test

    When it comes to a DevSecOps environment, test automation is not limited to UI-focused Selenium tests. Optimally, your security practice should include the following:

    • Unit testing.
    • Front-end testing.
    • Back-end testing.
    • API testing.
    • Database testing.
    • Passive security testing.

    Secure

    Because development, operations and security go hand in hand, only a few issues are left unattended toward the end of the development process. 

    When vulnerabilities are identified, there is a better chance of determining if they are potential exploitations or false positives. 

    Deploy

    Automated provisioning and deployments can be utilized to accelerate product delivery and add consistency in the development process. Using an infrastructure-as-code tool, one can audit properties across the IT infrastructure and enforce secure configurations in a system. 

    Operate

    Routine maintenance and upgrades should be an indispensable component of your operations team. Leverage infrastructure-as-code tools to patch zero-day vulnerabilities and apply updates to the entire organization’s infrastructure. 

    Monitor

    A continuous monitoring plan should be in action to generate real-time stats of how your system is performing. In case any exploitation is recorded, it can be addressed immediately. 

    Scale

    Traditional data-center operations cannot entirely replace a compromised environment. Today’s ability to scale infrastructure through virtualization and the cloud, while addressing the demands of modern-day IT user base, should go a long way. 

    Adapt

    When it is about sustaining an agile practice, continuous improvement is key. This is also true for DevSecOps practices, as you improve and adapt throughout the software development lifecycle.

    Conclusion

    DevOps isn’t going anywhere, anytime soon. It is the new phase of developing, releasing and updating products in a software lifecycle. 

    That’s why high time security professionals let go of the traditional security stack and embrace security solutions at the speed of DevOps.

    — Deepak Gupta

  • A Developer’s Task: Minimizing Challenges Created by Fragmentation

    A Developer’s Task: Minimizing Challenges Created by Fragmentation

    Over the past decade, there has been a wellspring of apps, services and platforms to help developers build custom solutions for businesses. Some are great, some not so much. Regardless of quality, the quantity of options and each one’s varying level of sophistication has made it difficult for developers to separate the wheat from the chaff.

    A recent survey from IDC supports this notion. It contends that the current landscape of software development languages and frameworks is highly fragmented. Having too many options—as opposed to a single, unified environment—leads to holes and disconnects in the development process, not to mention the potential for significant complications in the long-term support of today’s apps.

    Here are some ways developers can navigate the challenges created by this high level of fragmentation. 

    Don’t Be Afraid of Low-Code (or No-Code) Solutions

    The IDC survey found that less than half of developers—44%—have used low-code development tools professionally at one point or another. As one would surmise, low-code programming brings with it a host of advantages, such as easy-to-use and intuitive interfaces, and the ability to deliver applications faster. 

    Some low-code platforms also enable developers to build applications that can be used on any device while also reducing dependency on outsourced IT. A built-in script builder can allow even relatively inexperienced developers to turn complex business logic into functional workflows—such as automating a repetitive task—within a matter of minutes. Even after apps are launched, workflows can be updated and customized to fit specific business needs nearly instantly.

    Achieve Scale with Serverless Applications

    When building on a serverless platform, developers prevent infrastructure maintenance from inhibiting applications from scaling globally. This also allows developers to build mobile apps without managing the backend infrastructure. Cloud computing using public clouds typically involves renting virtualized servers across the technology stack. But when looking to scale, the onus is on developers to deal with the operating system and database while also managing necessary updates. This can lead to a work-intensive process on the app infrastructure level.

    A serverless infrastructure, on the other hand, abstracts databases and file servers, shifting focus to the app logic level. This allows developers to build and manage microservice-based apps that integrate with other services, making it easy to scale applications.

    Make Things Easy on Your Team

    According to the IDC survey, just 20% of developers claim they are “extremely familiar” with containers and microservices. With such expertise hard to find, using flexible tools that allow teams to quickly deploy code in scalable, compartmentalized components is vital.

    Organizations should aim to help their developers focus more on designing top-class user experiences of serverless applications, rather than spending (or wasting) time and resources on setting up and maintaining components on the server side. This gives developers the ability to focus on doing what they love—developing—as opposed to other tedious and mundane activities such as manually managing servers and storage solutions, or tweaking the underlying infrastructure of their microservices.

    Of course, making things easier for developers means deploying a powerful tool built on a stable foundation that can offer SDKs and APIs for both mobile and web apps, the ability to test early and often in a sandbox environment and analytics that monitor and report performance in real-time.

    Developers know just how challenging their jobs can be, and the fragmentation that can occur due to the vast number of options on the market can make it even worse. There is nothing wrong with using multiple solutions, of course, but having a sturdy and unified underlying infrastructure goes a long way toward avoiding unnecessary headaches.

    Finally, organizations should look to those who know best when deciding which company-wide tools to implement: the developers themselves. As the IDC study states, developers should be viewed as key stakeholders in IT purchasing within any organization moving to the cloud accompanied by an internal digital transformation.

    — Raju Vegesna

  • The Disservice of Cloud Consolidation

    The Disservice of Cloud Consolidation

    There’s no secret that over the past five years especially, the major (“tier 1”) cloud infrastructure players—Amazon, Microsoft and Google—have been snatching up smaller specialty providers, particularly cloud migration and security services.

    While this acquisition strategy immediately aims to strengthen and expand the provider’s direct and tangential cloud infrastructure offerings, the long-term goal is customer attraction and retention.

    In consolidating cloud infrastructure and related services, cloud vendors are targeting a basic human desire that often bleeds over into business decisions: convenience.

    • “Finding a cloud provider seems hard.”
    • “Switching to the cloud seems hard.”
    • “Managing more than one cloud seems hard.”
    • “Keeping up with more than one vendor seems hard.”

    The play is transparent; the marketing is already in your psyche. People don’t like doing more than they have to, whenever possible. So, why not apply that tick to the cloud infrastructure experience? Why not use a provider where everything is already in one place?

    Truthfully, the move toward cloud consolidation is a step in the opposite direction from where we should be going. The future lies in a multi-cloud approach, not a consolidated one.

    Multi-cloud approaches offer both businesses and their customers several advantages over consolidated infrastructure.

    • When a business’ cloud infrastructure exists solely with one provider, every cloud-related business activity—and every customer interaction with those activities—is beholden to the provider. A provider outage is a business-wide outage, and a business-wide outage is unacceptable.
    • Multi-cloud approaches allow businesses to take advantage of best-of-breed cloud infrastructure. A business can implement databases, containers and other virtual infrastructure in a more highly optimized way.
    • Data held in disparate systems can be better partitioned and more secure than data stored with a single provider. Also, these disparate systems can bolster adherence to existing (and likely coming) data privacy laws and compliance regulations both nationally and internationally.
    • Utilizing cloud infrastructure housed physically closer to a business or its key users can drastically influence business activities in industries where parts of seconds count (e.g., finance).

    Aside from the standard reasoning, I’d argue that there’s additional economic health in taking a multi-cloud approach. Competition naturally drives innovation, stabilizes costs and encourages improvement. Where one provider may falter, three others might specialize. If the majority of businesses consolidate their cloud resources with a single provider, though, the likelihood increases of services gradually worsening with no alternatives toward which to turn.

    Multi-cloud takes planning and coordination to properly implement, and careful monitoring to maintain. An additional tool or two, such as a cloud management platform or cloud integration solution (which, truthfully, many businesses are likely already utilizing), may be needed.

    But, the benefits far exceed the challenges. After all—to call upon Ken Jenkins’ iconic chief of medicine, Bob Kelso—nothing worth having comes easy.

    — Zack Busch

  • DevOps Versus ITIL: How to Win the Battle Over Change Management

    DevOps Versus ITIL: How to Win the Battle Over Change Management

    Ever since DevOps came on the scene a decade ago, ITIL has been under siege.

    ITIL lays out a number of IT service management (ITSM) best practices, including release management, change management, incident management, problem management and many other aspects of running an IT shop so that its priorities align with the business’s. For many enterprises, ITIL is essential for governance and regulatory compliance–especially in heavily regulated industries.

    The basic idea of DevOps is that by leveraging better automation technology across the software lifecycle, it’s possible to rework the culture and organization of all the teams involved in software production and deployment. By breaking down the numerous silos and fostering greater collaboration, it should be possible to dramatically accelerate the ability to deploy quality software at scale.

    From the DevOps perspective, ITIL is nothing but a set of speed bumps and roadblocks, getting in the way of its continuous integration/continuous delivery (CI/CD) priorities.

    DevOps wants to speed up software release cycles, while change management often slows them down. Does this dichotomy mean that enterprises have to choose one or the other? Or perhaps we have to water down one of these approaches sufficiently so it’s marginally compatible with the other one?

    Is ITIL Part of the Problem?

    ITIL skeptics abound. “Frameworks like ITIL create the illusion of control that results in [the] organizational delusion that IT is efficient and manageable,” said Greg Ferro, long-time consultant and co-founder at Packet Pushers Interactive. “The single biggest problem in ITIL [is that] the creation of silos creates ‘not my problem’ attitudes.”

    One such silo is the change advisory board (CAB), a team that has the responsibility for making thumbs-up or thumbs-down decisions on numerous different activities across the IT organization. “Having the CAB review every single change request isn’t efficient,” said Kaimar Karu, head of ITSM at AXELOS, “and it’s definitely not common sense, especially when their costs can run to tens of thousands of deployments per hour in some organizations.”

    One organization that has struggled to transform their ITIL process as it moved to a DevOps model is financial services firm ING. In fact, its CABs were a point of contention for it. “We were just like an insurance company, always rejecting your first claim,” said Mark Heistek, IT specialist at ING. “And then come back if you have good arguments.”

    ING found it was able to modify ITIL to work within its new DevOps approach. “Don’t do everything the ITIL book says,” advised Jan-Joost Bouwman, ITSM process owner at ING. And yet, following ITIL for practices such as incident management are “still the best way to do it, because everybody knows what to do and you don’t get confused about the rules.”

    Or Is ITIL Part of the Solution?

    In spite of the skepticism, there are plenty of proponents of using ITIL with DevOps. “The DevOps Movement fits perfectly with ITSM,” according to Gene Kim, DevOps thought leader, author of The Unicorn Project and coauthor of The Phoenix Project. “ITIL and ITSM still are best codifications of the business processes that underpin IT operations, and actually describe many of the capabilities needed into order for IT operations to support a DevOps-style work stream.”

    Kim, however, focuses more on human capabilities than on the ITIL processes and procedures themselves. “ITSM practitioners are uniquely equipped to help in DevOps initiatives, and create value for the business,” said Kim. “ITSM skill sets are ever more important in a world where there is an ever-quickening business tempo.”

    Phil Tee, CEO and co-founder of Moogsoft, agrees with Kim–and also echoes the experience of Heistek and Bouwman at ING. “ITIL is not in conflict with DevOps,” said Tee, “but it must be adapted to DevOps practices in order to define clearly where issues arise and how to solve problems efficiently.”

    Adapting ITIL to work within the context of DevOps–without slowing it down–has become the crux of the discussion. “ITIL is not explicitly opposed to Agile and DevOps,” explained Charles Betz, principal analyst at Forrester Research. And yet, he found that “the overall ITIL narrative is still sequential, plan-centric and deterministic.”

    Rock and a Hard Place

    Part of the DevOps way of thinking is to reevaluate how the organization makes the decision to deploy software into production. “By combining developers and operations into smaller, more agile DevOps teams, organizations can build software more quickly than ever, enabling them to respond faster to changes in the market,” said RJ Jainendra, general manager and vice president, IT business management and DevOps business unit at ServiceNow. “But, that’s a difficult challenge for larger companies, however, that haven’t been able to realize the full benefits of DevOps for a variety of reasons, including the complexity of manual processes they manage, and a shortage of DevOps expertise.” 

    There are many business contexts where internal process or regulatory compliance concerns are the issue, introducing requirements beyond what current DevOps approaches can satisfy.

    Jumping to the opposite extreme and requiring rigid change management procedures, however, can easily be overkill. After all, even for the most heavily regulated, quality-conscious enterprise, some of the changes they put into production don’t have the same stringent requirements as the rest. Not all software changes carry the same level of risk.

    The challenge, therefore, is telling the difference–and putting into place proactive, automated ways of handling different change scenarios that are subject to different levels of rigor.

    Judicious Use of Automation

    In spite of what many DevOps zealots may say, the approach is inherently flexible and human-driven. People still create software, after all–and DevOps is more about encouraging new ways for people to cooperate to achieve a common goal than it is about offloading human tasks to automated processes.

    As a result, DevOps is not nearly as black and white as those zealots might have you believe. For example, the “continuous” in CI/CD doesn’t mean “all the time” or even “as fast as possible.” In practice, continuous means “as fast as practical given the particular business requirements that apply in this situation.”

    Sure, Amazon may deploy several times a minute and be able to easily integrate compliance and governance into their technologies, but that doesn’t mean you can, or even that you must. After all, enterprises have a very different business and technology context from web-scale, born in the cloud companies such as Amazon.

    You may simply have different requirements–and even more importantly, different software will likely have different deployment cadences, even within the same organization.

    These principles of flexibility and human-centeredness apply beyond deployment to all aspects of change–and thus, they apply to the change management and release management practices of ITIL more broadly. 

    Sometimes you need a CAB, and sometimes you don’t. Sometimes you need to change windows, and sometimes you don’t. Sometimes fully automated quality checks are sufficient for giving the green light for deployment, and sometimes they’re not–and so on.

    And perhaps most importantly, sometimes humans should make approval decisions, and sometimes those decisions should be automated. 

    The more you can automate these approvals, the faster your deployment will be. But if you are able to identify those changes that require human approval and route those accordingly, then you’ll have optimized how you handle change in a way that pulls together DevOps and ITIL practices.

    Bringing ITIL and DevOps Together in Practice

    Traditionally, ITSM products are ticketing systems. When some incident occurs or someone calls for a change, they must create a ticket that then kicks off a set of manual tasks, often by different people.

    DevOps can bring automation to this traditional ITSM context. “One of the modern miracles of DevOps principles and practices is that many parts of ITIL processes can now be automated, solving significant challenges, especially in the configuration, release and deployment management processes,” Kim explained.

    Such manual workflows are typically well-suited to supporting ITIL practices, but fall short of the speed that DevOps requires. “ServiceNow DevOps leverages the data and metadata it already has on its platform, combined with new information from the DevOps pipeline to create, automate, and manage ITSM changes from within DevOps workflows on an enterprise-wide scale,” said Jainendra.

    In fact, ServiceNow can oversee the full automation of the deployment, integrating with other DevOps tools as necessary to effect the change, eliminating the need for any additional action from a developer. On the flip side, when the policies in place require a human approval of a change, then ServiceNow routes the change to the appropriate person. 

    The Intellyx Take

    Automation, for all its benefits, has never been about taking humans out of the decision-making loop. On the contrary, automation takes low-value tasks off of people’s plates, allowing them to spend more time and effort on more valuable activities. Decision-making is often among those high-value activities.

    That being said, automation is getting better at making decisions, both because of the breadth of machine-understandable policy representations and other configuration metadata, as well as the increasing prevalence and sophistication of machine learning.

    For DevOps professionals, it’s essential to understand this increasing level of sophistication for the automation available in today’s tools. The choice between automated or manual is no longer a black-and-white affair. Instead, there are different levels of capability for automated decision-making depending upon the situation at hand.

    — Jason Bloomberg

  • GitLab Open Sources Monitoring Tools

    GitLab Open Sources Monitoring Tools

    GitLab has decided to make its monitoring and observability tools available as part of its core open source platform to help reduce the total cost of DevOps. Previously, the tools were available as part of the commercial tools in GitLab’s DevOps platform.

    Kenny Johnston, director of product/Ops products for GitLab, said that as responsibility for observability shifts left, it is clear developers are now exercising more influence over which monitoring tools are employed. As developers begin to employ monitoring tools early in the application development process, most of them prefer open source tools because it’s unclear whether the application being developed will ever make it into a production environment, he said.

    However, if the application does make it into a production environment, it can be challenging for developers to make a case for standardizing on whatever monitoring tools they’re using in place of an existing commercial offering employed by an IT operations team. In addition to reducing costs, the GitLab approach could serve to eliminate any potential friction in the DevOps environment, as applications move from development into production environments.

    GitLab’s move to make its monitoring tools available as open source software comes at a time when more telemetry applications being employed to collect data are also based on open source software. In addition, developers of modern cloud-native applications based on containers are taking advantage of open source Prometheus monitoring software developed under the auspices of the Cloud Native Computing Foundation (CNCF) to monitor their applications. GitLab already provides integration with Prometheus.

    GitLab is also making it clear it expects its monitoring tools to benefit from the contributions of a larger pool of developers. Specifically, GitLab is calling for help in the areas of ongoing integrations with Prometheus along with improved logging, tracing and alerting capabilities.

    It’s not at all clear just yet to what degree DevOps teams will prefer to rely on open source versus commercial monitoring tools. While open source tools have some obvious advantages in terms of cost, many organizations have already standardized on commercial monitoring tools for a wide variety of existing applications. Open source tools often are more difficult to deploy and don’t tend to provide as much observability depth for IT operations teams. In addition, most of the providers of commercial monitoring tools are integrating their offerings with both open source telemetry software and Prometheus to bridge any divide that might occur between when an application is developed and deployed in a production environment.

    The one thing that is clear is a lot more individuals within the IT environments will be accessing these tools now that developers increasingly are being held accountable for the performance of their applications in production environments. In addition, modern applications require monitoring tools to make sense of all the dependencies that existing between microservices. Previously, IT operations teams tended to limit usage of monitoring tools based on commercial software to their most critical applications because of the costs involved. With the rise of open source monitoring tools, the opportunity to apply monitoring more broadly becomes more economically feasible.

    Regardless of the path forward chosen, the more visibility into IT environments provided, the easier it becomes to embrace best DevOps practices.

    — Mike Vizard

  • Best of 2019: Breaking Down the OWASP API Security Top 10, Part 1

    Best of 2019: Breaking Down the OWASP API Security Top 10, Part 1

    As we close out 2019, we at DevOps.com wanted to highlight the five most popular articles of the year. Following is the fifth in our weeklong series of the Best of 2019.

    As a result of a broadening threat landscape and the ever-increasing usage of APIs, the OWASP API Security Top 10 Project was launched. From the start, the project was designed to help organizations, developers and application security teams become more aware of the risks associated with APIs. This past September, the OWASP API Security Top 10 release candidate (RC) was finalized and published on OWASP.

    In my previous article, I provided a high-level view of the interaction between API endpoints, modern apps and backend servers, in addition to how they’re different from their traditional browser-based counterparts. I also discussed why this project was so important to the contributors and the industry overall.

    In this article, I aim to clarify the first five risks by highlighting some of the possible attack scenarios to help organizations and end-users understand the dangers associated with deficient API implementations. The following discussion follows the same order as found in the OWASP API Security Top 10.

    API1:2019 – Broken Object Level Authorization

    Attackers can exploit API endpoints vulnerable to broken object level authorization by manipulating the ID of an object sent within the client request. What this means is the client can request information from an API endpoint that they are not supposed to have access to. This attack normally leads to unauthorized information disclosure, modification or destruction of data.

    Example attack scenario:

    Say for instance there is an e-commerce platform that provides financial and hosted services to a group of different online stores (shops). The platform provides an API used to gain access to revenue charts for each of their hosted stores, and each store should only have access to their own revenue charts. However, while inspecting the client request from a single store who wants to gain access to their own revenue charts, an attacker can identify (find) the API endpoint for those revenue charts and identify the URL in use, for example /shops/{shop name}/revenue_data.json. Using the names of other stores being hosted on the e-commerce platform, an attacker can create a simple script to modify the {shop name} ID object in subsequent requests, and gain access to the revenue charts of every other store.

    API2:2019 – Broken Authentication

    Being different than authorization discussed above, authentication on the other hand is a complex and confusing mechanism concerning APIs. Since authentication endpoints are exposed to anyone by design, the endpoints responsible for user-authentication must be treated differently from regular API endpoints and implement extra layers of protection for credential stuffing attempts, in addition to brute force password and token guessing attacks.

    Example attack scenario:

    Suppose an attacker obtained a list of leaked username/password combinations as the result of a data breach from another organizations. If the API endpoint handling authentication does not implement brute force or credential stuffing protections such as CAPTCHA, rate-limiting, account lockout, etc., an attacker can repeatedly attempt to gain access using the list of username/password combinations to determine what combination(s) work.

    API3:2019 – Excessive Data Exposure

    By design, API endpoints often expose sensitive data since they frequently rely on the client app to perform data filtering. Attackers exploit this issue by sniffing the traffic to analyze the responses, looking for sensitive data that should not be exposed. This data is supposed to be filtered on the client app, before being presented to the user.

    Example attack scenario:

    Image an IoT-based camera surveillance system allows administrators to add a newly-hired security guard as a system user, and the administrator wants to ensure the new user should only have access to certain cameras. These cameras are accessible via a mobile app the security guard uses while at work. The newly hired security guard’s mobile app makes an API request to an endpoint in order to receive data about the cameras and relies on the mobile app to filter which cameras the guard has access to. Although the mobile app only shows the cameras the guard can access, the actual API response contains a full list of all the cameras. Using the sniffed traffic, an attacker can manipulate the API request to show all cameras, bypassing the filtering on the mobile app.

    API4:2019 – Lack of Resources and Rate Limiting

    It is common to find API endpoints that do not implement any sort of rate limiting on the number of API requests, or they do not limit the type of requests that can consume considerable network, CPU, memory and storage resources. The amount of resources required to satisfy a request greatly depends on the user input and endpoint business logic. Attackers exploit these issues causing denial-of-service attacks and associated endpoint outages.

    Example attack scenario:

    Let’s say an attacker wants to cause a denial-of-service outage to a certain API that contains a very large list of users. The users’ list can be queried, but the application limits the number of users that can be displayed to 100 users. A normal request to the application would look like this: /api/users?page=1&size=100. In this case, the request would return with the first page and the first 100 users. If the attacker changed the size parameter from 100 to 200,000, it could cause a performance issue on the backend database, since the size parameter in use is so large. As a result, the API becomes unresponsive and is unable to handle further requests from this or any other client.

    API5:2019 – Broken Function Level Authorization

    Although different than API1 above, exploitation of this issue requires the attacker to send API requests to endpoints that they should not have access to, yet are exposed to anonymous users or regular, non-privileged users. These types of flaws are often easy to find and can allow attackers to access unauthorized functionality. For example, administrative functions are prime targets for this type of attack.

    Example attack scenario:

    To illustrate this further, imagine during the registration process to a certain application that only allows invited users to join, the mobile app triggers an API request to GET /api/invites/{invite_guid}. GET is a standard HTTP method used to request information from a particular resource. In this case, the response to the GET contains details about the invite, including the user’s role and email address.

    Now, say an attacker duplicated the request and manipulated the HTTP method by changing GET to POST. POST is an HTTP method used to send information to create or update a resource. The URL would look like this: POST /api/invites/new/{“email”:”hugo@malicious.com”,”role”:”admin”}. In this case, the attacker easily exploits this issue and sends himself an email invite to create an admin account.

    In the context of the five risks above, one could easily image many similar attack scenarios. Those provided were just examples of the nearly unlimited possibilities when attacking vulnerable API implementations. As you can see, the risks above are primarily caused by errors or oversights. I believe these risks could easily be managed or nearly eliminated when organizations improve their secure coding practices, especially when it comes to the way they’re utilizing APIs.

    — Erez Yalon

  • Predictions 2020: Infrastructure and Ops Trends to Watch in 2020

    Predictions 2020: Infrastructure and Ops Trends to Watch in 2020

    Today, modernizing systems is more than simply moving technology to a new location. It requires an IT stack of a new generation of technologies and tools to work.

    As we head into the new year, let’s take a look at the top eight trends that will shape how we will develop and deliver software. 

    Kubernetes

    Yes, it’s big and getting bigger. The pace of enterprise adoption of this leading container orchestration solution will only gain momentum in 2020. Look no further than the 2,300% attendance increase from KubeCon 2015 (500) to KubeCon 2019 (12,000). As more companies begin to see the direct financial and operational benefits to the flexibility, scalability, automation, high availability and portability of Kubernetes, the demand for developers and engineering experts in this complex software will grow accordingly. Case in point: A 2019 national job search on LinkedIn turned up 16,744 open positions for Kubernetes-related roles. Given the already large and still developing Kubernetes community, we should also see an increase in the speed and volume of feature releases. 

    Microservices

    It’s predicted that by 2022, 90% of all apps will feature microservices architectures. The growing business imperative for software engineering teams to produce and release software in ever shorter cycles—continuous delivery—has paved the way for a huge surge in microservices, the ideal architecture for CD. As enterprises continue their accelerated move to the cloud in 2020 and beyond, they increasingly prize the flexibility to try out the new technology stacks that microservices provide. All businesses want to optimize resources, minimize downtime, reduce infrastructure costs and avoid technology or vendor lock-in. In 2020, microservices architectures will continue to offer up more and better of all these benefits as the migration from monoliths picks up speed. 

    Service meshes

    The rise of microservices has led to a parallel boom in service meshes. This will only accelerate, given how integral the current class of service meshes—Istio, Linkerd, Consul, Envoy—is to the successful functioning of microservices. As a software designed to enable the thousands of microservices that make up an application to talk to each other, it’s quickly become an essential component with a vendor ecosystem poised for substantial growth. Feature-rich Istio, backed by IBM, Google and Lyft, is the leading Kubernetes service mesh, and that shows no signs of changing in 2020, especially given the relative ease of setting up and running it. If your enterprise has embarked on the journey from monolith to microservices, it will soon be crossing paths with service meshes. 

    Observability

    In the recent past, the sheer complexity of modern distributed systems has outstripped the tools designed to allow teams to see into them. Enter observability—which isn’t monitoring. Like good helicopter parents, software engineers, application developers, SREs and DevOps teams want to know and understand what their systems are doing at all times. LightStep, Datadog and Honeycomb, to name a few, are vendors poised to offer logging, tracing and metrics solutions to the rapidly growing number of enterprise teams building cloud native systems. Keep a close eye out for OpenTelemetry, an open source observability framework currently a member of the CNCF Sandbox. 

    SRE

    One of the great challenges ahead for this still-emerging field is for enterprises of varying sizes and work cultures to find a form of SRE that works. What succeeds at Google may not be the recipe for SRE implementation at smaller tech enterprises or startups. The role of SRE and the expectations that go with it remain fluid. The SRE’s constant imperative to automate will only add to the profile of critical SRE and DevOps tools such as Terraform, Ansible and Jenkins. Ponder Amazon’s Prime Day 2018 losses of up to $99 million for a clear understanding of the business-critical importance of SREs in the years to come. The job market reflects this new reality. 

    Security

    The shift-left movement in the software development lifecycle is on. The days of bolted-on post-production security are rapidly fading. DevSecOps, though still a buzz term, is here to stay, with more practice and use cases added to the theory. There are huge opportunities here for automated application security tools that support the software development workflow. 

    Then there’s privacy. The GDPR took effect last May and the California Consumer Privacy Act takes effect January 1, 2020. On top of these, there’s HIPAA and the regulatory compliance pressures on businesses in the area of security, which are already enormous—plus the immense challenges of cloud security. A 2019 Symantec survey on cloud security threats reports that 65% of businesses in the cloud still aren’t using multi-factor authentication. The same survey also notes that an astounding 85% of Symantec’s cloud customers aren’t employing security best practices. Look for businesses to tend to security earlier in the process. 

    Serverless

    It’s a boom time in serverless, with Google’s Functions, Microsoft’s Azure Functions and IBM’s OpenWhisk all offering their serverless solutions following the 2014 rollout of AWS Lambda. Look for more businesses to adopt a serverless-first approach as they upgrade legacy systems and see the substantial cost saving and application development benefits. Startups, too, will see the cost-cutting upside and the agility that serverless brings. The changes in application development and architecture in the cloud as a result of serverless are hard to overstate. With the vendor solutions now on offer, developers are freed up to do what they most like to do: run code and add functionality to their application, faster, without worrying about scaling and availability. Like microservices, serverless will see a huge upsurge in usage as the age of the monolith fades. 

    DevOps

    With the demands of cloud computing and far greater enterprise investment in dedicated teams, DevOps in 2020 is poised to make the transition from policies and principles to actual practice. The addition of baked-in security to the DevOps realm—DevSecOps—adds urgency to the automation of building, provisioning and deploying secure software. ML- and AI-driven apps have arrived, and DevOps practices will play a large role in the development workflow. The goal of zero-touch automation looms large, and this adds urgency to the need for clean data, organizational buy-in and more integrated systems. AIOps, though getting plenty of attention, is still young and its benefits to IT operations are still more theoretical than actual–but stay tuned.

    As we head into 2020, it will be exciting to watch these trends unfold. DevOps teams can expect the way we develop and deploy applications–in any architecture–to continue to evolve. By keeping a pulse on new developments in Kubernetes, microservices and other new technologies, they’ll be well prepared to bring their businesses into the new decade.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit  featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Kelsie Pallanck

  • Predictions 2020: 7 DevOps Predictions Shaping 2020

    Predictions 2020: 7 DevOps Predictions Shaping 2020

    2019 was an exciting year for DevOps practitioners, with DevOps adoption continuing to grow at a fast pace. Most organizations were either executing or evaluating their DevOps strategies. So, what is 2020 going to bring in DevOps, infrastructure and application automation? Here are seven DevOps predictions for emerging trends in the new year.

    Customer Experience

    CX will become the guiding light of the DevOps journey, and not necessarily just technology. This is a big mindset shift from the early days of pure-technologist enthusiasm. As DevOps matures, we have to remind ourselves that it’s all about making business faster, not just the cool tech.

    Take advantage now:

    • Tie DevOps achievements to business needs and results to get executive buy-in and budgets.
    • Successful QA teams will start looking at their role as guardians of customer experience and not just testers.

    Trust

    Everyone will be talking about trust—not only in the context of zero trust networks but also in the context of making it possible for Dev and Ops to trust one another as the foundation for DevOps.

    Take advantage now:

    • Proactively invest in creating a culture of trust. Identify causes for lack of trust in your value stream and make trust a 2020 goal.
    • Encourage product-level conversation and define methodology and ownership when it comes to value stream management, application and infrastructure automation. It is the foundation that everyone is working on.
    • Invest in staging environments—Staging environments are needed to ensure things work before you put them in production. For many of us, jumping directly to automatically deploying to production is a big step. The risk can be mitigated by investing in staging environment automation, which will be the first step toward deployment to production.

    Value Stream

    The DevOps toolchain is going away. In 2020 we will think value stream. The industry is maturing beyond the notion of chaining together dozens of open source tools and trying to make it work together. In previous years many felt comfortable saying that if they have a CI system in place it means they “do DevOps.” This is no longer the case. DevOps is not just CI, and DevOps cannot be established in silos. Mature DevOps needs to take into consideration the entire value stream—from planning a new capability and the business requirement, all the way to the customer getting more value and a wonderful experience.

    In 2019, Gartner announced it’s the last year of the ARO MQ, as it’s not the right perspective. In 2020 we finally gain a system-level view on the value stream. More players are announcing end-to-end value stream management including Atlassian’s new Forge platform, GitLab and CloudBees. Companies are maturing and adopting system-level solutions. 

    Take advantage now: 

    • Plan for platform operations. System-level value stream perspective requires roles that take ownership of orchestration and automation throughout the value stream and not just in silos.

    The SRE Role Will Continue to Mature

    The SRE role that started with Google bringing software development practices into Operations will continue to evolve. Increasing volume, velocity and variety of data will make AIOps solutions thrive. Also, increasing levels of automation will require smart ways to handle dynamic infrastructure and applications without losing control, and being able to track changes all the way back to the coder. 

    Take advantage now:

    • If you haven’t implemented the SRE role yet, consider how SREs can be part of your teams in 2020. Define SRE areas of responsibility and KPIs, and how the SREs communicate with the teams. Is it a centralized position or are the SREs part of the team?
    • Make sure you provide SREs with the tools they need to succeed including:
      • What are the observability solutions you use? Who uses them today? How do you analyze it and how is it communicated?
      • What are the production environment orchestration, deployment and configuration tools that you use today?  How do you connect them to application and infrastructure automation that happens before production?

    Seamless Infrastructure

    We will have to make infrastructure seamless for our teams. Internal customers are also customers, and their experiences will be just as important. Self-service and empowering teams will be the main themes in the bridges that need to be built between Dev and ITOps. Public cloud, Kubernetes and serverless technologies will grow. And while it makes automation more feasible than ever, the emphasis will shift to self-service and to the end users. With more companies adopting similar approaches, the need to develop internal expertise will decrease, and we will see more off-the-shelf solutions.

    Take advantage now:

    • Map how infrastructure is used throughout your value stream, and work on unifying the approach in product-level.
    • Make sure you build self-service solutions with strong focus on the end-user in order to enable and balance both speed and control. Make sure developers and ITOps are equally represented when determining an approach to infrastructure automation.

    Infrastructure Automation

    Infrastructure automation is going to get smarter, with a lot of AI and ML. Making sure automation is properly governed will also increasingly be important. Take into consideration security and compliance, as well as cost.

    Take advantage now:

    • Assess infrastructure automation practices throughout your value stream. Who builds the automation? What teams and systems consume it? How do you enforce security best practices (e.g. no passwords in public repositories, using proper data sets for testing, compliance of automation blueprints)? How do you enforce cost control (e.g. tagging, reviewing cloud spend reports)?
    • Invest in control mechanisms before you lose control, especially in public cloud automation. If you don’t have governance in place, look at baking governance into automation, rather than adopting traditional cloud management approaches. Invest in automatic de-commissioning of infrastructure, automatic tagging, analysis of infrastructure consumption data and connecting it back to the business requirement, secrets and key management and access control.

    SaaS

    Open source is nice. SaaS is great. While big open source projects set the pace (K8S, Github) and are strongly backed up and funded by the industry’s giants, smaller projects are increasingly difficult to scale and glue together, especially when trying to assure enterprise readiness. The need to “do it yourself” is gradually fading, as more vendors offer products and services for supporting DevOps with clear business and support models and, as important, with OOB integrations with the ecosystem. Just Rest API is no longer enough.

    Take advantage now:

    • If your enterprise-readiness requirements make it difficult for you to scale DevOps initiatives, consider evaluating new SaaS offerings that eliminate much of the heavy-lifting and gluing that were typical to the skills-heavy early-days implementations. If you were concerned with security, cost and skill requirement, today you can find tools that are purpose-built to support DevOps for enterprise at scale. The good news is that most of the vendors are well-aware of the criticality of integrating to your ecosystem, making it easier to achieve your end-to-end value stream vision.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Maya Ber Lerner

  • IBM Z Open Editor Support for Language Server Protocol is a Game Changer

    IBM Z Open Editor Support for Language Server Protocol is a Game Changer

    The integrated development environment (IDE) is an indispensable tool for software developers. Before it came along, coding was a laborious, detail ladened undertaking. We’ve become accustomed to the syntax checking and code completion features than even the most basic IDEs provide. These days we tend to forget how hard it was programming with nothing but a rudimentary text editor. Something as simple as finding a missing comma or a misplaced curly bracket that was causing a compilation error could take hours, maybe days should the codebase be big enough. When it came to tracing your way through a seemingly endless chain of functions and classes in order to find the culprit of a runtime error, well…fuggedaboutit! Without the modern IDE, we’d be sunk.

    Well, that was then and this is now. A few years back, a new technology emerged on the technical landscape that’s boosting the power of the IDE, and subsequently the scope of work a developer does and the decisions he or she makes by an order of magnitude. This new technology is Language Server Protocol (LSP).

    LSP Takes the IDE to a Whole New Level

    LSP is a specification developed by Microsoft that standardizes the way certain typical features are implemented within a programming IDE. LSP attempts to consolidate the implementation of IDE commonplace features such as code completion, hover-over tips, jump-to-definition, find-references and syntax checking into a standard format that can be applied to any IDE in an agnostic manner.

    LSP is a technology designed to address the following problem. Imagine that a tools vendor has an idea for implementing a new plugin that will automatically do code completion for programs written in the COBOL programming language. Such a feature might be useful to any number of COBOL programmers. However, not every COBOL programmer is programming using the same tool. Some might be using an IDE such as IBM Developer for z/OS. Another might be using a text editor such as Sublime Text, Atom or even something more basic such as vim. In earlier times, in order to get the feature to market the vendor needed to implement different versions of the plugin in a way that was special to the given development environment. There’d be a team working on a Sublime Text version of the plugin, another team working on the vim plugin so forth and so on. Not only did the tool developer need to accommodate different operating systems, but each team might come up with its own, particular way to implement the feature. As team members left the company and new members arrived to replace them, those new developers needed to learn each particular way the feature was realized. As you can see, not only did the variety breed complexity, but things got very expensive very quickly.

    LSP does away with the need for such specialized tools development. LSP provides a standard protocol that can be implemented in any tool. The protocol defines a set of standard methods, objects and communication methods that provide a common set of services in any IDE.

    Thus, the only work that needs to be done to get the previously described COBOL code complete feature to market is to express it in a way that is compatible with the LSP. The actual work of realizing the feature then involves writing a client that supports the standard. It’s a “one ring to rule them all” process so to speak.

    The LSP specification was originally published by Microsoft specific. But the standard has been embraced as an industry-wide. There are implementations in Java, C++, C#, Python, Scala, Swift and PHP, to name a few. Also, there is an implementation of LSP for COBOL under z/OS, which brings us to IBM Z Open Editor.

    Bringing the Use of LSP to the Mainframe

    IBM Z Open Editor is a free Visual Studio Code extension that provides a state of the art programming experience for COBOL and PL/1 developers. Also, IBM Z Open Editor integrates with Eclipse Che and Eclipse Theia. Integration with Eclipse Che and Theia is particularly interesting because it enables browser-based editing. It’s a powerful tool, even more so given that historically COBOL and PL/1 programmers have lived in the backwaters of modern IDE development. IBM Z Open Editor makes IDE-based software development a general programming experience. Developers can use the same tool to program COBOL as they do in Java or even HTML for that matter.

    LSP Enabled Features in IBM Z Open Editor

    The following is a list of tool features available to mainframe programmers using Visual Studio Code with IBM Z Open Editor enabled:

    • Code completion.
    • Hover over instructions and snippets.
    • Jump to definitions.
    • Workspace symbols.
    • Find references.
    • Diagnostics and syntax error detection.
    • Outline view.
    • Previewing of included copybooks on hover.
    • Navigate to copybooks.
    • Code template snippets.
    • Refactoring such as “rename symbol.”
    • Variable completion.

     Using a single IDE to program in a variety of languages is a trend that’s been evolving for a while. However, adding LSP into the mix allows any IDE to present a consistent set of design time features. For example, one IDE is now able to do code-completion, hover-over reporting and variable tracing regardless of the programming language in play.

    For example, Figure 1 below shows the COBOL code completion feature of IBM Z Open Editor plugin for Visual Studio Code. IBM Z Open Editor uses LSP to facilitate code completion.

    Figure 1: IBM Z Open Editor uses the Language Server Protocol to facilitate context-sensitive code completion.

    Figure 2 shows the search and sort features of IBM Z Open Editor, which again implemented according to the LSP.

    Figure 2: Using LSP to implement IBM Z Open Editor’s sortable and searchable outline view for navigation.

    Finally, Figure 3 shows how the IBM Z Open Editor takes advantage of the extensibility capabilities of LSP to implement a feature that resolves and loads copybooks for preview. A copybook is code that can be copied from a master source and inserted into several different programs or into a variety of places in a single program. A copybook is used to define the physical layout of program data, variables for example, as well as instances of procedural code.

    Figure 3: IBM Z Open Editor extends LSP to review and load a copybook within Visual Studio Code.

    Clearly, LSP brings a lot of power to the mainframe programming experience. Now tools developers can create a unified programming experience using a common methodology. But where things really get interesting is when LSP is used to report data that goes beyond standard programming and code management activities, to a place where the real-time behavior of those using deployed code is being reported. In other words, LSP makes it possible not only to get information about code that is local to the developer’s desktop but also about the code as it is being used at runtime out on the internet.

    Beyond the Mainframe: The Long Term Significance of LSP

    LSP is extensible by design. As you read above, IBM Z Open Editor extends LSP to allow programmers to preview and load copybooks. This is done by extending the protocol.

    The typical IDE to server interaction under LSP is that the LSP client reports to the LSP server when a code document has been opened, changed, saved and closed. Then, once the LSP server is aware that a code document is in play, the server can inspect the document and report back events that are relevant to the programming language (Figure 4).

    Figure 4: LSP uses a predefined set of commands and request/response objects to support programming IDE is a language and operating system agnostic manner.

    For example, the LSP server can inspect the document to make sure the code is syntactically correct. If an error is encountered the server reports back the starting and ending location of the error, along with a message. In the case of the copybook review feature, the client reports back the location of the cursor in the code window and the server determines an opportunity for a copybook review and informs the client accordingly.

    While the copybook preview feature is powerful, it’s confined in scope to the immediate document. This is good. But, imagine what it would be like if the server had knowledge of all instances of the code deployed and running on computers connected to the internet?

    Suppose for example, that your code was powering an API running on the internet and you had the ability to learn about the number of calls to a particular function backing the API or better yet, imagine that you had the ability to learn about function calls that encountered particular problems, such as a pattern of mistyped parameters or high latency calls. Then imagine if that global runtime behavior was reported right in the IDE? Such information would change the very nature of how a programmer approaches his or her work. More importantly, it would change the way an enterprise manages programmers.

    What if a programmer could decide what to work on based on global runtime behavior reported into the IDE via an LSP extension? For example, going back to the imagined pattern of mistyped parameters, let’s say that when a developer hovers over the related function in IDE a warning icon appeared informing the developer of 1000 instances of mistyped called were made at the global level. The developer decides that such an error rate is unacceptable. Then, he/she examines the developer documentation and discovers that the parameter type definitions for the function are poorly described and that the document needs to be corrected. The developer then makes updating that developer documentation a critical path, part of the day’s work that needs to be delivered immediately.

    Given the power that LSP brings to the programming experience, such a scenario is entirely possible. In fact, there are companies out there such as Apollo Graph Manager that presently allow developers and third parties to register their production APIs in order to collect the sort of global information described above and have it reported in an IDE via LSP. As those of us who have been in the software business for a while have come to learn, once a feature such as global reporting catches on, it’s only a matter of time until it becomes an industry-wide practice. It’s only a matter of time before globally collected runtime metrics delivered to a programmers LSP become commonplace.

    Again, the ramifications are significant. Once a programmer has the ability to make fundamental decisions about feature development and deployment schedules, what then becomes the role of the project manager and the product manager?

    To take the issue one step further, how much power should a programmer be given? And in terms of general business practices, how much should the programmer know about a company’s products? These questions might seem fantastic, but keep in mind at one time companies guarded their source code as a trade secret. Today, more code than not is shared between companies as part of the open-source movement. As a result, the way code is made and a developers work has changed. The same possibility exists due to the growing trend to extend the capabilities of the LSP. The ramifications of LSP go well beyond simply reporting syntax errors and doing code completion. LSP will change the way the entire industry works, from mainframe programming to developers working in the cloud or on commodity hardware.

    Putting it All Together

    Given current trends, there’s little doubt that LSP will make even the most rudimentary code editor a powerhouse tool for software development. Standardizing IDE features into a common format allows new features for any development tool to be implemented faster and at a reduced cost. In no time at all we might even figure out a way to use LSP to make is so large bodies of code can be written automatically with just a few guiding keystrokes from the developer’s keyboard no matter the programming language. It’s a real possibility.

    Not only will LSP making coding easier and faster, but it’s also going to change the role of the software developer. Once all the code running on the internet becomes LSP aware and accessible, developers will get runtime information about how their code is performing at runtime on a global level. The trend is already in play. As history has shown, the further down a technology goes into a given population, the more powerful that population becomes and the greater the ramifications when the power is exerted. There’s a very real possibility the information that LSP enabled IDEs provides will make the day-to-day programmer more prominent in the product decision-making processes that are presently the responsibilities of program and project managers. Once that shift takes place we’re likely to see another industry-wide transformation take place in the way software is made and delivered. History has shown that such transformations have long term benefit, but in the short term are disruptive and disturbing. We’ll do well to be prepared.

    — Bob Reselman

  • The Use of Feature Flags in Code: Too Many Flags?

    The Use of Feature Flags in Code: Too Many Flags?

    It is interesting to watch the growth of an idea over time. The idea of using flags in code to do different things is as old as programming itself. Old-school configuration files used them, and they worked, so they were adapted.

    Thing is, given an idea that works, humanity inevitably takes it too far, and flags to control software behavior seem to have reached that point. If you are using one set of flags for A/B testing, another to segregate code in development at runtime, another to turn functionality on for some users but not others and yet another set of flags to hide data based upon user…You’re buried in unnecessary code branches. If your organization isn’t great about cleaning up after flags are no longer needed, you are building spaghetti code that future developers will not thank you for.

    The best answer to this growing burden is focus. What is the best use of feature flags/toggles in your environment? They certainly speed the delivery process by having the code there, in nightly builds, just not executed until the flag is flipped. This helps in rollout also by allowing operators to turn a feature on when it is ready by changing config rather than redeploying the application.

    For A/B testing, in many cases it makes more sense to handle switching at a load balancer, where again it is a config, but separate instances serve the UI in question. This, like all decisions regarding flags, should be based upon the volume of change. Changing the contents of a list is likely easier to test with flags, but large volume of change is probably better run separately and switched off of a load balancer.

    The same is true with the volume of change in source. A rollout might be the best option for handling large volumes of change, and feature flags may simply create more complexity.

    As with everything, approach the problem with a plan that works best, in this instance, in your environment. There is no denying that the ability to turn on a new feature and then turn it off if something goes wrong is appealing, just use the idea judiciously. And clean up after yourself. No shop that uses flagging extensively will say “yeah, we don’t clean it up,” because the mess of nested checks for flags would slow the code and make it passingly difficult to even read, let alone maintain, over time. So approach it from the beginning with “It is a tool, and when it no longer serves its intended purpose (the feature is fully live or abandoned), we remove it.”

    And keep rocking it. You are the engine of business, and this is what you do. Make it great and keep it running.

    — Don Macvittie