devopsy.kinsta.cloud

Category: Doin’ DevOps

  • Predictions 2020: 7 DevOps Predictions Shaping 2020

    Predictions 2020: 7 DevOps Predictions Shaping 2020

    2019 was an exciting year for DevOps practitioners, with DevOps adoption continuing to grow at a fast pace. Most organizations were either executing or evaluating their DevOps strategies. So, what is 2020 going to bring in DevOps, infrastructure and application automation? Here are seven DevOps predictions for emerging trends in the new year.

    Customer Experience

    CX will become the guiding light of the DevOps journey, and not necessarily just technology. This is a big mindset shift from the early days of pure-technologist enthusiasm. As DevOps matures, we have to remind ourselves that it’s all about making business faster, not just the cool tech.

    Take advantage now:

    • Tie DevOps achievements to business needs and results to get executive buy-in and budgets.
    • Successful QA teams will start looking at their role as guardians of customer experience and not just testers.

    Trust

    Everyone will be talking about trust—not only in the context of zero trust networks but also in the context of making it possible for Dev and Ops to trust one another as the foundation for DevOps.

    Take advantage now:

    • Proactively invest in creating a culture of trust. Identify causes for lack of trust in your value stream and make trust a 2020 goal.
    • Encourage product-level conversation and define methodology and ownership when it comes to value stream management, application and infrastructure automation. It is the foundation that everyone is working on.
    • Invest in staging environments—Staging environments are needed to ensure things work before you put them in production. For many of us, jumping directly to automatically deploying to production is a big step. The risk can be mitigated by investing in staging environment automation, which will be the first step toward deployment to production.

    Value Stream

    The DevOps toolchain is going away. In 2020 we will think value stream. The industry is maturing beyond the notion of chaining together dozens of open source tools and trying to make it work together. In previous years many felt comfortable saying that if they have a CI system in place it means they “do DevOps.” This is no longer the case. DevOps is not just CI, and DevOps cannot be established in silos. Mature DevOps needs to take into consideration the entire value stream—from planning a new capability and the business requirement, all the way to the customer getting more value and a wonderful experience.

    In 2019, Gartner announced it’s the last year of the ARO MQ, as it’s not the right perspective. In 2020 we finally gain a system-level view on the value stream. More players are announcing end-to-end value stream management including Atlassian’s new Forge platform, GitLab and CloudBees. Companies are maturing and adopting system-level solutions. 

    Take advantage now: 

    • Plan for platform operations. System-level value stream perspective requires roles that take ownership of orchestration and automation throughout the value stream and not just in silos.

    The SRE Role Will Continue to Mature

    The SRE role that started with Google bringing software development practices into Operations will continue to evolve. Increasing volume, velocity and variety of data will make AIOps solutions thrive. Also, increasing levels of automation will require smart ways to handle dynamic infrastructure and applications without losing control, and being able to track changes all the way back to the coder. 

    Take advantage now:

    • If you haven’t implemented the SRE role yet, consider how SREs can be part of your teams in 2020. Define SRE areas of responsibility and KPIs, and how the SREs communicate with the teams. Is it a centralized position or are the SREs part of the team?
    • Make sure you provide SREs with the tools they need to succeed including:
      • What are the observability solutions you use? Who uses them today? How do you analyze it and how is it communicated?
      • What are the production environment orchestration, deployment and configuration tools that you use today?  How do you connect them to application and infrastructure automation that happens before production?

    Seamless Infrastructure

    We will have to make infrastructure seamless for our teams. Internal customers are also customers, and their experiences will be just as important. Self-service and empowering teams will be the main themes in the bridges that need to be built between Dev and ITOps. Public cloud, Kubernetes and serverless technologies will grow. And while it makes automation more feasible than ever, the emphasis will shift to self-service and to the end users. With more companies adopting similar approaches, the need to develop internal expertise will decrease, and we will see more off-the-shelf solutions.

    Take advantage now:

    • Map how infrastructure is used throughout your value stream, and work on unifying the approach in product-level.
    • Make sure you build self-service solutions with strong focus on the end-user in order to enable and balance both speed and control. Make sure developers and ITOps are equally represented when determining an approach to infrastructure automation.

    Infrastructure Automation

    Infrastructure automation is going to get smarter, with a lot of AI and ML. Making sure automation is properly governed will also increasingly be important. Take into consideration security and compliance, as well as cost.

    Take advantage now:

    • Assess infrastructure automation practices throughout your value stream. Who builds the automation? What teams and systems consume it? How do you enforce security best practices (e.g. no passwords in public repositories, using proper data sets for testing, compliance of automation blueprints)? How do you enforce cost control (e.g. tagging, reviewing cloud spend reports)?
    • Invest in control mechanisms before you lose control, especially in public cloud automation. If you don’t have governance in place, look at baking governance into automation, rather than adopting traditional cloud management approaches. Invest in automatic de-commissioning of infrastructure, automatic tagging, analysis of infrastructure consumption data and connecting it back to the business requirement, secrets and key management and access control.

    SaaS

    Open source is nice. SaaS is great. While big open source projects set the pace (K8S, Github) and are strongly backed up and funded by the industry’s giants, smaller projects are increasingly difficult to scale and glue together, especially when trying to assure enterprise readiness. The need to “do it yourself” is gradually fading, as more vendors offer products and services for supporting DevOps with clear business and support models and, as important, with OOB integrations with the ecosystem. Just Rest API is no longer enough.

    Take advantage now:

    • If your enterprise-readiness requirements make it difficult for you to scale DevOps initiatives, consider evaluating new SaaS offerings that eliminate much of the heavy-lifting and gluing that were typical to the skills-heavy early-days implementations. If you were concerned with security, cost and skill requirement, today you can find tools that are purpose-built to support DevOps for enterprise at scale. The good news is that most of the vendors are well-aware of the criticality of integrating to your ecosystem, making it easier to achieve your end-to-end value stream vision.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Maya Ber Lerner

  • The Benefits of Test-Driven Design

    The Benefits of Test-Driven Design

    At John Hancock, we use Test-Driven Design (TDD) for all our new initiatives. TDD has three simple steps:

    1. Before you write functional code, write a test that proves if the implementation works or fails.
    2. Write the code and watch the test pass.
    3. Check if the added functionality changes your holistic approach and if you want to consolidate/refactor some of your code. If yes, refactor these sections.

    That sounds simple enough, so why write an article about a three-step process? Why do we, at John Hancock Digital Enablement, get so excited about TDD? We have seen major impacts to our developers, products and culture:

    • Increased motivation and self confidence.
    • Shorter Release-Cycles.
    • Increased Code Quality.
    • Massive Reduction of Bugs in Production.

    The Foggy Path

    Imagine your development process like a foggy path, where you can see directly in front of you, but anything further away becomes foggy and imprecise. Imagine it like a journey into the unknown.

    Obstacles may appear suddenly, forcing you to rethink your approach (refactoring), or you may get above the fog after a long journey and realize that there is a much simpler way to reach your goals (also refactoring).

    This uncertainty and constant change may appear to be frightening and trigger anxiety, but TDD is the safety net, because you know that what you’ve built holds up against all the risks that have been identified. As a developer, I can take much higher risks and try different approaches and get quick validation.

    The test cases built out as part of the TDD flow are ideally integrated into a continuous integration/continuous development (CI/CD) Platform, giving fast and reliable feedback to the engineers about unexpected behaviors their code change may have caused. Because the feedback to the engineer is near instant, the developer is familiar with the change that caused the error and is able to resolve it rapidly. 

    TDD Is the Opposite of “Never Change a Running System”

    A well-known issue is that the later a bug is found, the more costly it is to resolve. TDD enables fast feedback loops so bugs can be resolved quickly and cost efficiently.

    As an application grows, TDD gives you a solid foundation of everything that has already been built. You can be confident that what you have built is solid and that new functionality can be added to it easily and without taking the risk of breaking everything. In other words, TDD is the opposite of “never change a running system” and enables us to build even better software. 

    TDD Always Provides Up-to-Date Code Documentation

    TDD provides excellent documentation toward the requirements and behaviors of the application. While documentation always lags behind the functional code and typically is written after a functionality is implemented, in the world of TDD everything starts with a test. The test documentation is always up to date and gives developers an easy and understandable way to figure out what the application is achieving without dissecting the entire code.

    The following quote from Terry Pratchett explains it best: “If you do not know where you come from, then you don’t know where you are, and if you don’t know where you are, then you don’t know where you’re going. And if you don’t know where you’re going, you’re probably going wrong.”

    As an application grows, evolves and is built up over time, and as it’s passed down and handed over, TDD gives it documentation and describes its solid foundation. The act of writing a test case forces a developer to stop and think about the functionality that is to be implemented and describe its desired outcomes.

    TDD Provides Early Feedback 

    Through TDD we receive early feedback about the path we are about to set foot on. Early indicators can help us correct our assumptions or help rephrase or rewrite a requirement so that we deliver customer value. 

    Early indicators that we’ve encountered: 

    • Functionality cannot be measured: Why are we implementing this if we cannot measure its outcome?
    • Functionality does not add any value.
    • Functionality cannot be tested: Similar to the first bullet, if a functionality cannot be tested, how can we accept it’s delivery? 
    • Functionality can be easily embedded into an existing part of the application that needs refactoring: This can trigger the refactoring before we start the implementation.

    As described, it is very helpful to think about the steps ahead before jumping in, which can be difficult for engineers to do. TDD does so in a language that is natural to engineers.

    TDD Embraces an Ever-Changing World and Removes Fear of Change

    TDD as a part of eXtreme Programming (XP) accepts and embraces an ever-changing world we cannot control. Changes may cause discomfort and anxiety as planning becomes difficult/different.

    In an unstable, disorganized software project, changes are thrown at it, a solid foundation is missing, testing is skipped or postponed to the end and refactoring is skipped all in the name of on-time delivery. The first delivery of the application may be on time, but as time goes on the anxiety grows, there are random bugs everywhere, the more bugs that are fixed, the more issues pop up and you essentially end up playing a game of whack-a-mole in which you will never be in control.

    In a project that embraces XP principles, such as TDD, we try to remain in control from the beginning, facing challenges early so they do not bite us later on. A solid test base gives developers, and everybody else on the team, confidence that the delivered solution will work, but most importantly it removes the fear of change.

    Without fear of change, we can try all sorts of crazy things, test out ideas and see them turn out or fail without the code failing, without losing a customer, without getting that phone call in the call center. 

    Traditional QA and Release Cycles Are too Slow for this New Era of Ever-Changing Needs

    As software grows, its testing becomes more intense. So, with each new piece of functionality there must be a corresponding test case. The manual execution of those tests becomes lengthy, the feedback loop to the developers slower, releases get held up, customers do not get new functionality in time and, in the end, money is lost.

    Traditional QA and release cycles are too slow for this new era of ever-changing needs. What if I want to release daily or weekly? Do we have to execute the test procedures every time? What if bugs are found and repaired? Do we need to rerun the entire set? 

    The answer is yes—but if a machine does it in a few minutes, who cares? 

    While I am not a proponent of no human testing, what I am saying is that 95% of manual QA can be automated and executed by a machine, and TDD provides a fantastic test set to get started with. Humans may catch errors that are hard for a machine to identify or imagine since human behavior is hard to predict, so spot testing as well as exploratory and behavioral testing are still needed.

    We Are Left with a Team that Is Hungry for More and Only Afraid of the Enormity of the Possible

    What are we left with at the end of a TDD project? We have a well-documented, easy to understand, easy to change and stable code base lead by a team that is not afraid of change. We are left with a team that is hungry for more and only afraid of the enormity of the possible.

    — Niklas Studer

  • Predictions 2020: The Future of the Mobile Industry

    Predictions 2020: The Future of the Mobile Industry

    The mobile device industry continues to grow in leaps and bounds. Statistics show that, today, over 5 billion people have mobile devices and there are over 9.2 billion mobile connections worldwide. Add to these statistics the fact that the number of consumers who shop and bank online continues to grow as well–presenting hackers with increasing numbers of mobile applications and end-points to target. As developers of software that protect mobile apps from reverse engineering and hacking, at Guardsquare we have a unique purview into the mobility space, and therefore have some pertinent 2020 predictions for the mobile industry.

    Mobile Payments Apps on the Rise

    As consumers’ comfort with, and reliance on, personal technology increases, we expect to see purchases via mobile payment apps grow rapidly throughout 2020. Great news for the global economy, but the downside of this trend is it will serve to exacerbate opportunities for hackers to take advantage of mobile applications and devices that are not properly secured.

    Fake Apps Abound

    Fake apps or hidden apps have historically been a problem and we don’t see that changing any time soon. In December of last year alone, almost 65,000 new fake apps were detected among the various app stores, and we expect this to become increasingly common and problematic. Consumers need to be alert when downloading new apps and developers should be wary that fake apps can negatively affect their own reputation and their company’s. 

    Developers Beware Kotlin’s Hidden Vulnerabilities

    The new development language, Kotlin, is catching on now that it has been designated Google’s preferred language for Android development. However, many developers using the language still do not fully understand security best practices, including how to protect Kotlin code against OWASP’s well-known Mobile Top 10 risks. As with many apps written in Java, apps written using Kotlin must be protected against both static and dynamic attacks using a combination of code hardening and RASP.

    iOS Is Vulnerable Too

    We believe that 2020 will be the year in which awareness increases around the security shortfalls of iOS. Many application developers still believe iOS apps are virtually immune to reverse engineering, but, in reality, all iOS versions have already been jail-broken at some point.  We anticipate developers will take better security measures to ensure the integrity of their apps by employing a multi-layered approach to security to ensure the protection of their iOS applications.

    We continue to see the more sensitive an organization’s data, the more likely it is the mobile app will become a target for hackers. According to OWASP’s Mobile Top 10, reverse-engineering and tampering ranked the eighth and ninth most prevalent mobile security risks. Verizon’s Mobile Security Index 2019 highlights that more than 80% of organizations said they are at risk from mobile security threats, and 69% said those risks increased in the last year.

    It is our hope that, in 2020, all apps—but especially apps housing sensitive customer data—evolve from going unprotected to incorporating key security measures that protect the users and the organizations.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit  featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Roel Caers

  • The Role of DevOps in Custom Software Development

    The Role of DevOps in Custom Software Development

    Hundreds of tech giants and big retail companies such as Etsy, Netflix, Amazon and Target have adopted an ecosystem called DevOps. It establishes a good collaboration between cross-functional teams such as development and operations which simplifies the processes. DevOps has calibrated a new standard for building custom applications.

    Do you know any predefined technique to check whether developers have adhered to all coding principles? A quality check is not the right answer. What if you want to check and optimize their entire system of writing code, understanding the project and following coding guidelines?

    You must have dug into the details of not being able to accomplish the projects before the deadline. Well, there is not a single reason for this failure. There is a need for restructuring the entire ecosystem of project development, which will include polishing of coding rules, eliminating duplicates of code, unit tests, bug removal and understanding and improving the project architecture. DevOps aims for such a practice that includes neat coding, bug-free testing and the fastest deliverables.

    This blog explains everything about how DevOps plays a significant role in custom software development and makes a super-productive project development environment.

    The term “DevOps” is the combination of two different terms: development and operations. Thanks to Patrick Debois and Andrew Clay Shafer for originating the idea of DevOps in 2008. They have altered the traditional techniques of software development. Gone are the days when the software was written once and used for years. For the era of rapid software development and extreme personalization, you need smart software that updates in seconds.

    Custom Software and DevOps

    Now is the time where generic needs seem limited and are absolute off-the-shelf in the software industry.

    DevOps is the process of creating custom software that fulfills everyone’s demands in no time. It chooses to streamline the project development phases by automating the process, making faster deliveries and upscaling the software quality in the world of high-end customizations.

    How Does Automation of Software Development Plays a Predominant Role?

    Software development life cycle comprises of various components such as time, manpower, expenses, knowledge resources and problem-solving capabilities. It continuously evolves around bringing a unique solution that stays reliable over time.

    The optimization of the expenses, time and manual efforts is possible using the automation of SDLC. Especially when customizing the features, manual efforts make the tasks tedious and slow. Automation overcomes these drawbacks and makes customizations super easy to fulfill.

    Automation testing has carried out the responsibility of making project development glitch-free, as it detects the errors at a much faster pace. For instance, a custom software/application, such as the social media platform Instagram, carries a lot of features and functionalities to add. To make this a glitch-free development, automated test runs stand as the rock-solid solution.

    DevOps not only improves the programming practice of the coders but also scrutinizes the testing. It streamlines communication among the coders, front-end engineers, clients and project managers. Thanks to its immediate nature of validation and rectification.

    The automated workflow of SDLC saves a lot of business expenses as it requires less time for rebuilding the applications. Also, it optimizes the time required for running the manual tests. It brings in the high quality of work that decreases the glitch fixing need and hence, reducing the overall project costs of custom software.

     How Is Continuous Delivery or Faster Delivery Important?

    Continuous delivery means delivering the small chunks of code a developer has built, in short cycles. It lowers the project development costs significantly of custom software as the hidden fixed costs with the primary release of projects will not exist.

    Smaller release, smaller risks. Yes, there will be nothing like once the project delivery has been done, you can not opt for a small change request as a part of the suggestion. Because the deliveries won’t stop when you opt for custom software development. This sort of development is required to adopt changes very fast and hence, needs deliverables to be delivered at a faster pace for the rapid feedback implementation and live changes.

    So, hypothetically, you will not need to be afraid of how it will turn out. For instance:

    1. Amazon deploys code at every 11.7 seconds.
    2. Popular social media platform Instagram follows the same practice 30-50 times a day.
    3. Netflix pushes its code thousands of times a day.

    What led them to this?

    • To avoid bad commits.
    • Freedom to deploy the code whenever possible.
    • Avoiding the mess for going through the thousands of deploys for one error.
    • To speed up the work and make changes in seconds.
    • To respond to users unpredictable queries and turn it into a feature.

    This can be easily achieved through DevOps in custom software.

    There are many reasons for more occurrences of deployments. All the reasons lead one focus (i.e., faster delivery for custom software and small but speedy updates for the custom software that is used by millions of people online).

    Faster bug fixes and updates are the new demand for sure. All thanks to DevOps methodology. The most valued part is, developers and delivery managers will come to know users’ preferences at the very early stage of development. As the project goes on, there will be zero chances of bug generation and a smooth operation will be established. 

    Thus, the successful glitch-free customized software development is possible with a DevOps strategy. The teams will get to learn from peers and end-users about the project at an earlier stage using DevOps.

    Responsiveness Is the Best Software Quality 

    The main advantage of the DevOps technique is its quick responsiveness to clients’ feedback and rapid execution.

    If you can quickly look into the client’s feedback in the small chunk of code you’ve submitted, you can respond to them faster. Thus, can lead the fastest execution possible.

    The integration of a quality assurance team will add a lot of potential in improving product quality.

    Responsiveness plays a crucial role in decreasing the frequency of a module collapsing (i.e., maintains the module dependency because of the continuous delivery and quick implementation of feedback).

    Purpose and Role of DevOps in Custom Software Development

    You need a super key to stay ahead of competitions and it always comes by streamlining the process. Be it a business revenue, cross-selling of products, increasing the target market and anything that comes up with competitors.

    DevOps is that key here. Yes, you read it right. Bring in this methodology at your information technology firm to witness the enormous growth.

    Custom software development has become pivotal in the IT industry. To create the best-customized experience, consistency is required. That is the reason why DevOps is gaining popularity so fast. DevOps is the master key to connecting resources with the same thread. 

    You might have wondered how Netflix updates and personalizes the experience for everyone so rapidly. Netflix does thousands of deployments per day. That’s the reason it can give its users a top-quality personalized experience. 

    “The rate at which this entertainment game-changer has adopted new technologies and implemented them into its DevOps approach is setting new standards in IT,” said Coman Hamilton, editor at JAXenter.

    Netflix had started nailing this modern approach from 2015 itself. Now you can understand how technology has come so far approaching 2020. So, what next doable thing is up for today’s need for deployments using DevOps?

    DevOps is an exemplary technique that enables IT to showcase its power. It empowers the dynamic nature of custom software and stabilizes the work environment by keeping everyone on the same page at all instances of time.

    For a company to be responsive and relaxing throughout the year without overloading the work, adopting DevOps is a must. The purpose of enabling development and IT operations together serves as time-saving and an in-budget procedure of building applications with the utmost quality.

    Where to Start?

    DevOps is not a new software to learn or a new framework to set in. It is the way of working and adopting it—everyone must be involved. Educating resources is important when you are planning to adopt a big change. Conduct a conference, state the pain areas and showcase the possibilities with DevOps. You can start with teams that have great adoption skills, the ones who can take challenges and are ready to learn.

    Essentials to Consider for the Best Practices of DevOps Implementation

    Breaking the Silos

    If you want to make DevOps an organizational practice, you will have to first break down the functional silos. As DevOps will make everyone come closer, the communication gap among the teams will not be questioned. In an information technology organization, the more communication engineers keep, the more successful they will be in project development.

    Make this a daily practice, “exchange of ideas is an exchange of solutions.”

    Using the Right Tools

    There are several tools to help your team automate the process and track down the work done. You will need a tool such as Jira when entering a DevOps environment. Also, choosing the right tools for the right needs is necessary. For example, Jira is a very good tool for tracking the issues and for managing the projects, while Querysurge is used to perform continuous testing and data analysis purposes.

    As per the project and company needs, one can go for the right set of tools to use. Make your organization resourceful with the automation and project tracking tools that are already overcoming manual efforts.

    User-Centric

    Adopting DevOps is a cultural change and it listens to its end-users. The end-user of any particular industry will want the best features and robust functionality. Firms will have to step in with the altering demands and stay committed to user satisfaction. Start making customer-centric applications.

    Automation at Its Best

    As we know, DevOps detects diseases at an earlier stage. It has the lowest percentage of risks involved as it uses automated testing of a small snippet of code. Keep it in a practice to code and test. It will automatically follow the test-driven development practice used in agile teams. Use the most of automation practices that eliminate the slow process of software development. This way the teams get faster development completions without any hampering of deadlines.

    Conclusion

    You may end up thinking DevOps works for IT firms and is not going to help non-IT industries or businesses. It’s not like that.

    Think about the software you have used for years in your non-IT firm. Are you using a software that drives your business but it’s not up to date? Think about customizing it to improve the way it works. These customizations come from experienced custom software development companies. Prominent companies are nailing DevOps nowadays.

    — Hardik Shah

  • Predictions 2020: Ushering in 2020 Data Predictions

    Predictions 2020: Ushering in 2020 Data Predictions

    2019 brought us more data organizations running more advanced analytics, AI and ML workloads than ever before. 2020 is the year where we’ll see a spike in both the number of technologies and data teams that support these types of workloads internally. We’ll see AI and analytics teams merge into one as the new foundation of the data organization, focused on areas such as moving to the cloud while maintaining on-prem Hadoop, “Kubernetifying” the analytics stack and Hadoop compute. These are the trends I believe we’ll see come about in 2020.

    Rise of the Hybrid Cloud

    We’ve been hearing people talk about the hybrid cloud for the past three years now. For the most part, that’s all it’s been talk—but in 2020 it gets real. We are seeing large enterprises refusing to add capacity on-prem to their Hadoop deployments and instead invest in the public cloud. But they are still not willing to move their core enterprise data to the cloud. Data will stay on-prem and compute will be burst to the cloud, particularly for peak demands and unpredictable workloads. Technologies that provide optimal approaches to achieve this will drive the rise of the hybrid cloud. 

    One Machine Learning Framework to Rule Them All

    Machine learning with models has reached a turning point, with companies of all sizes and at all stages moving towards operationalizing their model training efforts. While there are several popular frameworks for model training, a leading technology hasn’t yet emerged. Just like Apache Spark is considered a leader for data transformation jobs and Presto is emerging as the leading tech for interactive querying, 2020 will be the year we’ll see a front-runner dominate the broader model training space with pyTorch or Tensorflow as leading contenders. 

    Kubernetifying the Analytics Stack

    While containers and Kubernetes works exceptionally well for stateless applications such as web servers and self-contained databases, we haven’t seen a ton of container usage when it comes to advanced analytics and AI. In 2020, we’ll see a shift to AI and analytic workloads becoming more mainstream in Kubernetes land. Kubernetifying the analytics stack will mean solving for data sharing and elasticity by moving data from remote data silos into K8s clusters for tighter data locality. 

    Hadoop Storage (HDFS) is Dead but Hadoop Compute (Spark) Lives Strong

    There is a lot of talk about Hadoop being dead, but the Hadoop ecosystem has rising stars. Compute frameworks such as Spark and Presto extract more value from data and have been adopted into the broader compute ecosystem. HDFS is dead because of its complexity and cost and because compute fundamentally cannot scale elastically if it stays tied to HDFS. For real-time insights, users need immediate and elastic compute capacity that’s available in the cloud. Data in HDFS will move to the most optimal and cost efficient system, be it cloud storage or on-prem object storage. HDFS will die but Hadoop compute will live on and live strong.

    AI and Analytics Teams Will Merge Into One as the New Foundation of the Data Organization

    Yesterday’s Hadoop platform teams are today’s AI/analytics teams. Over time, a multitude of ways to get insights on data have emerged. AI is the next step to structured data analytics. What used to be statistical models has converged with computer science to become AI and ML. So data, analytics and AI teams need to collaborate to derive value from the same data they all use. This will be done by building the right data stack—storage silos and computes, deployed on-prem, in the cloud or in both, will be the norm. In 2020 we’ll see more organizations building dedicated teams around this data stack.

    Talent Gap Will Inhibit Data Technology Adoption

    Building the stacks that enable data technology into practice is hard, and this will only become more obvious in 2020. As companies discuss the importance of data in their organizations, they’ll need to hire the data, AI and cloud engineers to architect it. But there aren’t enough engineers who have expertise in these technologies to do that. This super-power skill is the ability to understand data, structured and unstructured, and pick the right approach to analyze it. Until the knowledge gap closes, we’ll continue to see a shortage of these types of engineers—many companies will come up short on their promises of “data-everywhere.”

    China Is Moving to the Cloud on a Scale Much Larger than the US and Will Leap Frog From On-Prem to Massive Cloud Deployments for Advanced Workloads

    Over the past five years, while enterprises in the U.S. have been moving in leaps and bounds to public clouds, enterprises in China have been investing mostly in on-prem infrastructure primarily for data-driven platform infrastructure. 2020 will be the inflection point where this changes. China will leapfrog into the cloud at a scale much larger than the U.S. by adopting the public cloud for new use cases, bursting in the cloud for peak loads and over time move existing workloads. Public cloud leaders in China will see dramatic growth that might outpace the growth of the current cloud giants.

    2020 is the year where the rubber meets the road when it comes to advanced analytics and AI. Companies that back the types of technologies that enable and support this kind of data and workloads will emerge as leaders in the space. On the other side, companies that structure their data teams to meet the requirements of the new data stack will emerge as leaders as well. I’m excited to see how advanced analytics and AI opens up new and innovative applications and use cases.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit  featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Haoyuan Li

  • The Automated Data Center Emerges

    The Automated Data Center Emerges

    Hyperscale and automation seem to go hand-in-hand, at least when it comes to the data center. Synergy Research reported that, at the end of 2017, there were 390 web-scale data centers worldwide, up from 300 a year earlier. IDC identified that hyperscale data centers “have a minimum of 5,000 servers and are at least 10,000 sq ft in size but generally much larger.”

    Today, hyperscale data centers are driving a sense of urgency for automation, but many smaller corporate data centers are just as complex, even if they are less physically expansive. In either case, automation takes data from across the data center, applies analytics and machine learning to that data and helps to identify both inefficiencies and trouble spots in the data center–as well as their origins. 

    Some data center functions are well on their way to becoming automated, making new deployments and changes to existing deployments simpler, faster and more efficient than they are today. Among those functions, orchestration, change management, software-defined architectures and DevOps stand out. 

    Meanwhile, another data center function is lagging: problem resolution. The challenge revolves around the siloed nature of the data center, where the multiple, disparate tools and methods used to sleuth out inefficiencies and pinpoint trouble spots lack a unified view of the data center infrastructure. In addition, silo-centric infrastructure monitoring tools lack application context. Short of automation, today’s data centers are stuck in the past. Especially as a data center grows and must interact with cloud-based services, no amount of human intuition in using those tools and methods can overcome the barriers of siloed systems. 

    However, data center infrastructure planners are making strides in identifying the requirements for accelerating the move to the automated data center. 

    Three Keys to the Automated Data Center

    For years, the automated data center was just a dream. And as recently as late 2018, Forrester spoke of islands of automation.

    “For the enterprises I’ve talked to, it’s not like any of them are completely manual at this stage, but they’re all at varying degrees of trying to integrate all these pieces,” said Chris Gardner, Forrester senior analyst. He added that fragmentation was due to a lack of holistic solutions that could connect all aspects of the data center.

    But now, three forces are shaping up to help make the automated data center a reality. Let’s look further. 

    Real-Time Visibility 

    Real-time infrastructure visibility has become a critical element of automating the data center. Why? Because an issue can occur anywhere on the virtualized, hybrid infrastructure, and the issue can ripple to create problems elsewhere, making issue resolution a perennial challenge. 

    For that reason, your best defense is to build deep, real-time visibility into the compute/virtualization stack and the storage/storage network stack. This will help you quickly ensure the performance of mission-critical applications accessing the data from those stacks.

    For ITOps, it is about proactively pinpointing the source of the issue and resolving it before application users are impacted. That requires stitching data across the full stack of technologies supporting the business: application services and infrastructure services, both on-premises and in the cloud. 

    Real-Time Analytics

    Why is real-time analytics crucial in the data center? In short, it’s because real-time responsiveness is critical for supporting anomaly detection, root cause analysis, remediation, prevention and planning use cases, where processing and correlating massive amounts of information is critical.

    For ITOps, it means distributed real-time architectures must combine cloud-based and on-premises analytics that leverage AI and machine learning. It’s the analytics and correlation that helps you make sense of all that real-time data so it becomes actionable.

    Closed-Loop Systems

    IT organizations are acutely aware of the perils of introducing changes to critical production environments. In fact, they take great pains to establish a governance of change. Closing the loop is about humans delegating decision-making to technology.

    For ITOps, that means ensuring integrated control of change processes across the entire technology stack. Data and analytics should lead to optimal real-time decision-making that empowers the automation for closed-loop operation.

    How Automated Is Automated?

    In 2016, researchers at the University of Pisa argued that hyperscale data centers would have to rely completely on analytical, automated systems to have to any chance to manage the huge number of day-to-day issues their servers generate. They added that “large computer systems in general, and data centers in particular, will ultimately be managed using predictive computational and executable models obtained through data-science tools, and at that point, the intervention of humans will be limited to setting high-level goals and policies rather than performing low-level operations.”

    Now, more than three years later, critical technologies and methodologies are in place to make the automated data center a reality, even if humans are still required for guidance, if not for sleuthing out infrastructure issues.

    — Len Rosenthal

  • Untapping the Potential of DevOps, Part 1: Why Is It so Critical Today?

    Untapping the Potential of DevOps, Part 1: Why Is It so Critical Today?

    Apps are making the world a simpler, faster place. Mortgages can be verified in minutes. Food shows up at doorsteps with a few clicks. A safe ride is just a pin-drop away. For consumers, having a smooth, flawless and enjoyable app experience is the status quo. However, what goes on behind the scenes to make it happen is infinitely more complex.

    From developers to architects and QA to test automation engineers, the software we interact with requires connected teams to ensure that security, reliability, speed and quality are consistently being delivered. For years, technical teams operated in silos that were worlds apart. Application and software delivery processes were slow, painful and error-prone—a dangerous mix for failure. 

    About a decade ago, DevOps was introduced as a remedy.  

    While definitions of DevOps can vary, it is essentially about aligning people, tools and processes to make software delivery better, faster, cheaper and more enjoyable for those involved (which tends to lead to happy customers, too). However, with the boom in DevOps technologies and resources, cutting through the noise and honing in on what will make the biggest impact can be something of a struggle.

    In this series, we will address why DevOps is a key enabler for success, the trends impacting the software delivery industry, DevOps challenges, tips and more. First, let’s dig into why DevOps should be a part of an overall business strategy. 

    Why is DevOps Critical Today?

    Times are changing, and so are IT best practices. So, what makes DevOps great for business success? 

    As mentioned, customers expect new product features to be delivered continuously—with quality and security guaranteed. In order to deliver on this promise, the entire IT organization must work in unison toward the same goals, and these goals should be centered around better customer experience. Much of a DevOps transformation is focused on culture, and when everyone from developers and business leads to legal and marketing teams are collaborating and happy, it leads to positive business outcomes. In fact, according to the findings in the 2019 Accelerate State of DevOps Report, organizations that invest in creating a culture of psychological safety and collaboration are more likely to be high performing. 

    Being a high performing organization sounds like a no-brainer, but what does it really mean? According to the same report, the highest performing organizations have 208 times more frequent code deployments, recover 2,604 times faster from incidents and deploy 106 times faster than low performing organizations. DevOps is the foundation that makes this high performance possible. This means that organizations that aren’t adopting DevOps will be left in the dust.

    DevOps is a set of practices that combine software development and information technology operations to shorten the systems development lifecycle while delivering features, fixes and updates frequently in close alignment with business objectives. The practices help deliver features to end-users much faster and with higher quality, testing and documentation. Processes should be reproducible and auditable, making the technical infrastructure more secure and reliable as a result. Ultimately, DevOps leads to happy and productive teams, satisfied customers and keeps the budget in the black. 

    There’s much more to uncover when it comes to DevOps—in the next part of this series we will look at some of the common mistakes made along the DevOps transformation journey. 

    — Jonathan Fries

  • How Combining DevOps and Agile Can Ease Black Friday Pressure

    How Combining DevOps and Agile Can Ease Black Friday Pressure

    Every year, seasonal peaks such as Black Friday and Cyber Monday bring a higher volume and faster pace of work, as well as a considerable amount of pressure. This comes with a number of challenges, including managing traffic influx, ensuring top-notch digital experience, rebuilding catalog infrastructure and carrying out load and performance testing.

    Development teams need to navigate increasingly complex tasks in time spans that are shorter than usual. That’s where Agile comes in: With its fast-fail nature, Agile fills the gaps between limited resources and market needs. By prioritizing self-organizing teams and robust feedback, its flexibility allows teams to rapidly refocus as business requirements inevitably shift.

    DevOps, on the other hand, offers a more systematic approach. Despite its various overlaps with Agile, its signature benefit is its heavy reliance on the collaboration between development and operations throughout the entire product lifecycle. With continuous testing and delivery, DevOps ensures the delivered product is fully in line with customers’ needs.

    Both approaches offer benefits to teams looking to navigate the complexities of the holiday season. Research by Cutter Consortium revealed that combining DevOps and Agile can lead to the reduction of the time needed to get a product to market by 20-40% and to significant cost reductions as well.

    Why should teams leverage the marriage of DevOps and Agile to relieve the pressure of the holiday shopping season?

    Automation as the Centerpiece

    The last quarter of the year is traditionally seen as a trial that often stretches organizations to their full capacity. However, if organizations are still using obsolete manual processes, standards will slip. Exposing high-availability systems to human error during seasonal peaks that bring ten-fold higher traffic than normal may have dire consequences for a business. Automation, on the other hand, ensures that processes are less error-prone and vulnerable to human errors.

    DevOps puts emphasis on automation. Not only does it focus on maximum efficiency when deploying software, but it also advocates for close process documentation. Automation facilitates optimal knowledge transfer and frees  production teams from time-consuming manual tasks, allowing them to become faster and more adaptable.

    It’s common that organizations claim that Agile and SCRUM simply don’t work for them. However, this is mainly because they haven’t achieved advanced enough automation levels or outsourced enough tasks to be fully flexible. But with automation, they can embrace it more effectively–as DevOps aims for the simplification of the entire process, its focus on automation establishes solid foundations for efficient Agile. This means organizations cut down the time to market and are more equipped to face the challenges of high season.

    High Flexibility for a Better Product

    Seasonal peaks such as Black Friday and Cyber Monday are always bound to agility. The revenue potential of an organization depends largely on the flexibility of the team and the processes and systems that, if maintained effectively, can add significant value.

    Agile teams usually work in two-week grinds, and each of these sprints should result in certain features. However, in order to meet the business goals, these features must be usable. Without DevOps, it would be difficult to deliver something that brings incremental value. This is, for example, because agile teams often disregard automatic testing. But incorporating this DevOps principle ensures that the process is always moving in the right direction.

    Likewise, if agile teams pay close attention to collaboration with customers, product management, developers and other parties, as DevOps suggests, they can develop a better product that has undergone review from various parties. That’s because DevOps can extend agile principles beyond the code to the entire delivered service.

    The two approaches fit great together. Agile enables teams to produce minimum viable software and its features in a swift cycle, while DevOps ensures that the delivered product is of a higher quality. Such a complementary relationship wouldn’t be possible with the waterfall scenario where cycles last months or even years. While DevOps encourages integrational behavior, Agile relies on quick evaluation and incentives, so they come together in flexible solutions. This is fundamental throughout the high season, where priorities often shift fast and the needs of customers become more urgent than ever.

    Close Oversight to Avoid Failure

    As teams face increased pressure during Black Friday and Cyber Monday, errors become more likely. Deploying Agile together with DevOps guarantees there is a robust architecture that’s able to prevent failures.

    DevOps practices allow agile teams to respond faster–not only to business requirements but also to any other problems that arise, too. When combined with the best practices of modularity and microservices, failures can often be isolated to a single system. Such a system can be fixed, replaced or completely removed with ease.

    Organizations are increasingly aware of the importance of preventative analytics. For example, Slack integrations throughout the DevOps cycle can go a long way for early diagnostics. Agile teams incorporating DevOps can leverage continuous testing and monitoring that bring both constant progress and quality assurance.

    Companies such as Etsy, a global handcraft and vintage marketplace, saw a lower occurrence of errors and disruptions after exchanging the waterfall model for a more agile approach. The organization relied on DevOps structures to deploy a fully automated pipeline and assured that continuous delivery practices secured a higher number of daily deployments–with fewer errors at the same time.

    High Levels of Synergy

    As mentioned earlier, DevOps has many convergence points with Agile, which is why they form such powerful partners with little chance of clashing. One of the fruits of their synergy has crystallized in the form of Infrastructure as Code (IaC).

    IaC in a modular system, as promoted by DevOps, brings organizations remarkable benefits, as significant changes can be rolled out across the entire network typology with a single code change in minutes. In the context of feature requirement (or a failure), such velocity is instrumental to leveraging seasonality. This way, developers can alter entire environment configurations, regardless of the initial stage, and thus avoid tedious rebuilding.

    Teams looking to optimize their performance shouldn’t be deciding between Agile or DevOps. In fact, organizations will find that their powerful combination can equip them much better for the seasonal peaks and help them deliver greater results than ever.

    — Ain TohVri

  • 5 Best Practices for Feature Flagging

    5 Best Practices for Feature Flagging

    Feature flags, also known as the feature toggles, are important concepts in the field of DevOps. They have been around for a long time and have been used continually by developers for their convenience. As software development services providers are striving for higher agility, practices such as feature flagging are becoming increasingly relevant. If you are not familiar with the concept of a feature flag, let’s start with understanding what feature flagging is.

    What is Feature Flagging?

    Feature flags are used for the process of branching your code, which means to turn certain code paths on and off to separate the release of new features from the actual deployment of our code. Why do DevOps developers want to branch their code through feature flags? Well, feature flagging is a great software development practice for two reasons. Firstly, for features that are a work-in-progress, developers can use a feature flag to hold their release until they are ready. This ensures the release of other features that are ready is not delayed just because one feature is not. Secondly, features that are being worked on can be kept off using the feature flag and only once it is ready it can be turned on. This allows developers to work directly on the feature without changing the main code, also known as the trunk or master branch. When all is working and ready to be released, turning on the feature toggle releases the feature, just like a switch.

    To understand feature flags in the simplest term, feature flags are “if/else” statements to develop a conditional branch or pathway into your code. For example, you can use a feature flag to state that “if so and so feature is enabled, run new algorithm; or else, run old algorithm.” This allows developers to control the functionality—if the feature flag is enabled only then the new algorithm is executed; else skipped. This is the most basic form of feature flags, however, they could be more complex.

    Feature flags are a practical way for software teams to maintain the delivery and release of features, and there are many advantages of using them. However, too many feature flags can make your code more complex and difficult to manage.

    Here are the best practices of feature flagging that can save your project from becoming a feature flag hell.

    More Feature Flags Equals Complex Code

    Feature flagging is a good practice that can help you maintain the delivery and release of different features, however, using too many feature flags can quickly turn your code into a feature flag hell. The more feature flags you create, the more code paths are going to be created, this increases the complexity of your code and makes the testing process harder. A higher number of code paths can cause a mess and risk the functionality of your code. Therefore, it is best to only create feature flags when it makes sense and always check if a previously created feature flag can be re-used instead.

    Name Your Feature Flags

    For efficient management of feature flags, it is crucial to have a proper flag naming system. Without it, it is possible to confuse the flags, forget why they were created for, or create flags with the same name—all of which can cause disruptions and confusion. When working with a team of developers, it is important to ensure that everyone follows the same labeling method so they can get the information about the flag just by looking at the name. Otherwise, it is likely that a team member might mistakenly trigger the flags, upsetting your code.

    Log and Clean Up Old Flags

    Over time, it can become difficult to keep track of the changes that are made to a feature flag. Therefore, it is important to track changes through logging, especially when working in teams. Using logging, you can keep a track of who created the feature flag, who made a change and at what time. This can save you from wasting time figuring out who was responsible for what change. A log will set the record straight about every change introduced for every feature flag.

    This is also great to know which flag and code paths are obsolete and redundant now. When you know which flags are unused you can clean them from your code to avoid any conflict or confusion.

    Use a Feature Flag Management Tool

    For better management, control and visibility of your feature flags, you can use available feature flag management tools. Like every other management tool, feature flag management tools are enterprise-scale software to manage feature flags. Some of the most popular tools include LaunchDarkly, Rollout and Bullet Train. Tools like these can make the management of feature flags easier for you and increase your development speed.

    Use Feature Flags to Collect Data

    When it comes to scaling your product, feature flags can be of great help to collect data from users. For instance, if you want to introduce a new feature, you can roll out the change slowly to a few users first. Using feature flags, you can make the new feature or update available to a smaller set of users only and collect data about how well they respond to the release. You can then use their feedback to improve or change your features to deliver an improved user experience. Feature flags are used in this manner by companies such as Facebook and Netflix to test their new features before releasing them for the entire user base.

    — Emily Williamson

  • The 20 Best DevOps Podcasts

    The 20 Best DevOps Podcasts

    Skip the whole “starting a podcast is easy, all you need is a computer and a mic” pitch. The truth is podcasting is hard and it takes considerable work to create something special. The low barrier of entry intrigue is a double-edged sword, making it harder for good podcasts to get noticed in the incessant wave of new shows coming to market every day.

    Fortunately, there are podcast fanatics out there who can help curate the signal from the noise to ultimately recommend the best podcasts. I’ve worked in the DevOps space for awhile now, most notably at AppDynamics and Sumo Logic, and also run Discover Pods–a site about podcasts.

    Without further adieu, here are my picks for best DevOps podcasts.

    Arrested DevOps

    Hosted by Joe Laha, Bridget Kromhout, Matt Stratton, Trever Hess and Jessica Kerr, Arrested DevOps is probably the most complete podcast on this list. With over 130 episodes produced, they’ve found a nice balance between giving actionable advice around key DevOps strategies, team management, tool selection and integration along with having truly interesting guests from all over the world on their podcast.

    Episodes typically come out about once a month and hover around 45 minutes.

    Listen: Apple Podcasts | Stitcher

    Software Defined Talk

    Software Defined Talk just crossed 200 episodes, making it one of the most accomplished podcasts on this list. Hosted by the always-amusing Cote, Brandon Whichard and Matt Ray, SDT is likely the most entertaining DevOps podcast you’ll listen to as well. They focus on what’s trendy and what has staying power in the greater DevOps landscape, plus go into news and current events.

    New episodes are dropped weekly on Fridays and are usually and hour to an hour and a half long. On an unrelated meta note, I appreciate how they share their technical podcast experience with their audience including changing their podcast hosting service and frustrations with some of the major players.

    Listen: Apple Podcasts | Stitcher

    Masters of Data

    Hosted by Ben Newton, Masters of Data is almost entirely composed of interviews with industry professionals, authors and more, around the role data plays in their lives. Though the majority of discussions are focused on data within the technology landscape, what separates MoD from others on this list is the diversity. For example, Sarah Guo from the venture capital firm Greylock Partners, talks about how she uses data to make investing decisions. John Visneski from Pokemon talks about how they use data to better ensure privacy. David Shor worked on Obama’s reelection campaign and talks about how data gave them the upper hand.

    Episodes are published about once per month.

    Listen: Apple Podcasts | Stitcher

    DevOps Chat

    In full disclosure, this podcast is produced by the fine folks at DevOps.com, however, it most definitely still warrants placement on the list here. By leveraging their standing within the community–and most importantly as an unbiased third-party–DevOps Chat is able to bring guests on from every vendor to provide a glimpse into how they’re tackling major DevOps issues at their company.

    Episodes are released frequently, sometimes multiple times per week. They’re usually relatively short and make for a quick listen on a commute.

    Listen: Apple Podcasts | Stitcher

    The Changelog

    Billed as “Conversations with the hackers, leaders and innovators of software development,” and I guess I can’t describe it any more succinctly than that. Hosts Adam Stacoviak and Jerod Santo have a good podcast rapport and produce a weekly show (typically 45 minutes to an hour long). Usually, they’re joined by guest in the greater developer community.

    Interview-based podcasts in particular seem to run the gamut of quality. Most outsiders likely think they’re the easiest since you simply need to book a guest and speak into a microphone. However, they require more of the intangibles like interview rapport, research and personality to create a successful show.

    Listen: Apple Podcasts | Stitcher

    Puppet Podcast (Pulling the Strings)

    I typically hesitate to recommended branded podcasts of any kind, but Puppet has earned the respect and doesn’t shill in their show. The podcast mostly comprises of industry discussions in automation and config management. Their latest episode discusses the most recent State of DevOps Report and the larger trends they’re seeing in the market exemplified from the survey.

    Listen: Apple Podcasts

    Azure DevOps Podcast

    Hosted by Microsoft MVP, Jeffrey Palermo, the Azure DevOps Podcast is an interview-based shows with both practitioners and execs specializing in .NET and Windows-focused DevOps roles. Though I don’t quite agree with the podcast description using the words “hard-hitting interviews,” Palermo is able to conduct an interesting conversation with his guests as well as provide listeners with actionable strategies on pushing out new features and monitoring them in production.

    Listen: Apple Podcasts | Stitcher

    Honorable Mentions

    Some of these are no longer in circulation but many of their episodes are usually around evergreen topics and are–at the very least–entertaining listens on DevOps topics:

    • ThoughtWorks
    • Continuous Discussions (c9d9)
    • DevOps Radio
    • Software Defined Interviews
    • DevOps Mastery
    • Software Engineering Daily
    • All Things DevOps
    • Real World DevOps
    • Nice Work
    • The Hanselminutes Podcast
    • Greater Than Code
    • This Developer’s Life
    • DevSecOps Community

    There you have it, my favorite DevOps podcasts. They’ll interview your favorite thought leaders, discuss container strategies, CI/CD and more.

    — Kevin Goldberg