devopsy.kinsta.cloud

Tag: scalability

  • Funding Failure by Intent

    Funding Failure by Intent

    Why is it that when we purchase a lotto ticket, every part of our brain tells us we won’t win, yet some portion of it is able to inspire hope where there shouldn’t be any? Could it be the same part of our brain that inspires hope in the face of nearly certain failure has been nurtured by the software development projects we have been exposed to over the years?

    It would appear that the ability to prognosticate has been the foundation of nearly every waterfall project that had a timeline of three years or longer. It is this myth of “certainty” that agile methods are designed to eliminate. In our new approach, we simply break down and digest the scope of work until it is small enough for our risk tolerance, and easy enough to implement in our abbreviated time horizons. We try, we evaluate and we try again.

    But no matter how low we believe the risk to be (and generally the reward we expect from it), risk does remain. And now instead of a three-year project plan well-equipped with milestones, corrective actions and projected costs, agile gives us only the current sprint, based loosely on our historical velocity and a literal “addiction” to continue this process going forward without any real regard for a predictable end-point or total project cost. Continuous delivery infers continuous innovation (emphasis on the word “continuous”). And if done right, why would a business ever choose to end innovation that works?

    Imagine the analogy of Mozart sitting down to compose a quick advertising ditty that your company could play as background music on its website. Mozart begins with a few bars, and notices a theme that is worthy of far more exploration … so he does. He continues exploring this until an epic symphony emerges. This was not the plan and does not meet the original requirements. This could be considered scope-creep in the waterfall world, and thus a failure, as no one stares blankly at a website for the 30 minutes it takes to hear an entire Mozart epic … or do they?

    Every audience grasps the concept of background music we pay little attention to but subconsciously understand it improves our experience. But nearly every audience also realizes the beauty of a Mozart epic, and would gladly buy CDs, MP3s or concert tickets to hear one of those over and over again, relishing every note. The question this analogy begs is simply this: Would Mozart’s employer—more specifically, his development manager—have been free to fund this “failure” or not? What happens when a project changes direction significantly away from the first vision we had of it? Does DevOps enable us to pursue this radical change in direction, or are our legacy financial systems and methods binding us to a waterfall delivery and reporting structure—and, in so doing, limiting what we will ever be able to accomplish through radical innovation?

    The beauty of agile construction and DevOps delivery is that we may deliver “unexpected” innovation. Nearly every time a user “sees” what we have delivered to meet their need, they immediately think of “x” more things we could do to make it even better. This may result in taking a project in a radically different direction within the confines of a single financial year. To further enable our ability to do this, our legacy financial funding and reporting systems must evolve to match this demand. Our new risk profile must include the ability to fund failure on purpose, understanding that failure itself is part of the maturation process of innovation, not the death-knell of the company.

    A more honest approach to funding agile/DevOps efforts would be to determine the annual investment the organization wants to make in certain capabilities, and innovations it expects to see in each thematic area. Overarching themes are funded on an annualized basis; however, great autonomy is granted and expected within the themes to ensure delivery is achieved. Pushing financial responsibility down in the organization to as low a level as possible has the effect of pairing the team leaders with the costs of what they do. Spending too much time on delivery of any given feature or function may cause the team to re-evaluate the approach and propose a better solution. This is more unlikely when budget responsibility is managed several levels above the team doing the work—in that instance, the budget resembles more of an infinite resource they need not be concerned with, until it is too late.

    When proponents of DevOps claim it is changing the culture of a company, they are not limiting this view only to IT. Tertiary services in a company, such as the supporting financial budgeting systems, hiring systems and even contract management systems, may require significant change to match the pace of delivery and innovation DevOps delivers. As an enterprise looks to implement DevOps, it should look broadly at what else may be required to fully enable the pace of innovation it is capable of.

    To continue the conversation, feel free to contact me.

  • Measuring the Cost of Service Creation & Adoption

    Measuring the Cost of Service Creation & Adoption

    Adopting and implementing DevOps isn’t an overnight project. It takes time and effort to determine how it best fits into your organization—and what benefits your organization will realize as a result.

    Creating an enterprise-class DevOps service is typically done in one of three ways, depending on the organization, its leadership and its employees. Each has its pros and cons, and each has a number of factors to consider.

    A Fully Centralized Model

    In this model, tooling decisions are done in a central service provider group for the entire organization. Tools are managed centrally and standards are created centrally. DevOps automation creation (at least the templates or frameworks) is done centrally. This model is highly advantageous for organizations with a wide variety of technology platforms (think large population of legacy apps) and use many technology components for their apps, such as multiple types of databases, middleware and operating platforms, for example.

    Not many large organizations are ready to scrap what they have in the application portfolio and start over using the cloud and containers and develop in a singular methodology. Using a centralized model, companies can continue to improve upon their years of investment in legacy applications and adopt a DevOps continuum without starting over from scratch.

    This also is the lowest-cost model, as tooling is standardized and administered from a central point and licensing can be negotiated to maximum effect. In addition, labor (generally the highest ongoing cost of any service creation and implementation) is minimized by centralizing it. Training costs also can be minimized by establishing a centralized training program for all impacted audiences. Companies can change their incentive programs to achieve hiring objectives and business results as priorities change—something that isn’t practical in a decentralized or federated model.

    A Fully Decentralized Model

    In a fully decentralized model, tooling decisions are determined and managed by each development team. This has the advantage for shops with small numbers of applications or singular methods of development—fewer applications are better candidates for starting over with cloud or container integrations and can take advantage of the newest technologies from the starting point, if needed.

    Shops such as Etsy or Facebook seem massive, but from a user point of view, they really only present a single application that needs to be maintained and improved over time. Instances like this may favor a decentralized model because the actual number of application candidates may be highly limited—sometimes only one.

    This model typically represents the highest costs, as repeatability and standardization can only be measured at the development team level. As such, this is practical for a very limited number of applications, or technologies; if those numbers grow, the strategy quickly becomes untenable. In addition, this strategy has an uncanny history of pulling development resources to do the DevOps automation engineering that needs to be created and maintained. Typically, the highest skilled developers are assigned these types of duties, and thinking in terms of templates and frameworks is rarely their first approach. Developers tend to be expedient, and believe they will always be there to make sure it works. This also can reduce the potential for repeatability and predictability (raising costs).

    A Federated Model

    In a federated model, tooling decisions and standards are done centrally, but management and operations of the tools are performed by a limited number of subservient organizations (often by line of business). Centralized templates may or may not be viable depending on the variability of the business units they serve. The chief advantage of this approach is the segregation of tooling and reporting by business unit (typically) without having to build that kind of segregation in to tools that were not designed that capability in mind.

    This model also has the advantage of increasing the speed of adoption, as several parallel efforts can take place at the same time and spreading engineer resource demand across multiple organizations. It typically makes the most sense in larger organizations with multiple business units whose application portfolios that would benefit from an onboarding to DevOps.

    What’s more, a federated model may be the most cost-efficient. Labor costs will be inherently higher as functions are replicated from business unit to business unit; however, that replication will result in increased adoption rates and project completions (you get what you pay for). If business units are able to leverage centralized templates for each technology type, adoption rates become an order of magnitude faster (or have that potential). Tooling costs still can be negotiated centrally, even though management will occur in the business units. Again, there is hardware replication from business unit to business unit, but the ability to segregate users, permissions and operations down to the business-unit level can save tremendous labor costs if development was the only other option to create that capability in tools not designed for it.

    Adoption Costs

    Measuring the cost of adoption requires focus, no matter which method you employ to create the DevOps service. Applications moving to DevOps are said to be onboarded (where the original build, deploy, test and release automation is created). To successfully onboard an application requires education of the sponsoring executives, training of the impacted developers and testers, and coordination with the change and release managers.

    Collection of meta data about the behavior of application components is no small task, requiring frequent follow-ups and action items to reach completion. The effort to onboard applications to DevOps becomes the first cost for estimation and tracking as the application portfolio is attacked.

    Even after onboarding of a given application is complete, costs continue. The terms “continuous delivery” and “continuous integration” should also infer that application development methods “continuously” evolve and may adopt new technology components. Significant changes may occur that require revisiting the DevOps automation to adapt to these ongoing revisions and updates.

    While this work is rarely as extensive as the initial onboarding, it is measurable. Work of this nature is usually referred to as BAU (business as usual) and represents the second significant cost of DevOps services that must be measured and tracked. If no thought is given to this work effort, inevitably the team assigned to do onboarding will be distracted by the BAU requirements until they reach the point where no new applications can be completed.

    Staffing cost estimations can be presented at the component level for both onboarding and BAU efforts. For example, if one DevOps engineer can onboard 20 components per week (assuming familiar technologies within a given portfolio), and if the general ratio of components per application is 4:1, then a given DevOps engineer can successfully onboard five applications per week. If the engineer is never distracted with BAU, this can continue until the entire portfolio is onboarded. However, my experience has shown that for every 20 applications completed with onboarding, the BAU work effort can absorb nearly a full DevOps engineer. So if you start with a team of 10 people, you will onboard only 200 applications total before the entire team is required to focus on BAU work effort and has no more bandwidth to continue onboarding new apps. Understanding this demand, and learning how efficient your particular engineering team is, will be key to managing costs while completing the onboarding efforts of large application portfolios.

  • DevOps and the Near-Decapitation of Variability

    DevOps and the Near-Decapitation of Variability

    Not all automation scales. One of the chief impediments of delivering an enterprise-class DevOps service is the decentralization of tooling choices—or worse, the methodologies within the tool for delivering CI/CD. If the methods used to deliver automation are as varied as the last engineer who created them, the spaghetti that ensues will not scale much farther than previous methods did.

    The trick to scalability is not so much tooling-dependent as it is approach-dependent. Templates or frameworks (or any word you would use) to describe a relatively preformatted approach to automating a particular kind of technology component are the key to achieving scale.

    Ideally, your highest-skilled engineers create the basic template or framework that other, lesser-skilled engineers will use to create the actual automation for each system as they move forward. Using a single way to automate a particular kind of technology enables you to manage large numbers of systems utilizing that technology from a single point of administration. For example, a standardized template used to create the automation to deploy a MySQL-type database could be altered over time to include new testing requirements detected from results found later in the QA process.

    Once the new testing is incorporated into the standardized template, the resulting automation is fixed once for every instance of that kind of database in every system that uses it. Otherwise, you’ll need to remember to update every script for every instance of a system that uses this kind of database. Those updates likely will be implemented differently across the applications by different engineers, which may negatively impact testing results (thus defeating the purpose of testing consistency, and actually generating more unnecessary work to locate the variations).

    Templates or frameworks (or pick the synonym that suits you) can reduce variability in “how” the automation continuously delivers and integrates. Once variability is killed (or nearly killed), predictability can emerge. Predictability throughout the entire SDLC lowers overall costs, as finding and fixing bugs becomes more systemic. What’s more, there are fewer bugs, thanks to battle-tested processes that produce consistent behavior. If the tooling is sophisticated enough to offer visibility, the costs of quality are reduced even further, and the speed to deliver is radically increased.

    DevOps promises that combination of reduced failure rates with increased speed of delivery. However, embracing unstructured variability in creating the automation can kill it. The Ops side of the DevOps equation is all about doing things systemically and predictably, as that is the only way to maintain stability while changes whiz through their world. Getting Ops to buy into DevOps is much easier when they have complete visibility into a predictable, repeatable, process for delivering change. Avoiding that 3 a.m. phone call is something everyone has in common, and reducing variability by focusing automation through the lens of templating or frameworks is an excellent way to achieve it.

  • 3 Tips to Avoid Painting Your DevOps into a Scalability Corner

    3 Tips to Avoid Painting Your DevOps into a Scalability Corner

    When everything is firing on all cylinders, DevOps can be awesome. Teams work together seamlessly. Routine tasks are automated. Continuous delivery is continuously integrated, continuously tested, continuously deployed and continuously monitored. All is well, and right with the world—unless your DevOps needs begin exceeding the capacity of the platforms and tools you depend on. Scalability is a crucial element of DevOps success.

    Wikipedia defines “scalability” as the capability of a system, network or process to handle a growing amount of work, or its potential to be enlarged to accommodate that growth.

    The need for scalability is one of the primary reasons that DevOps and cloud computing are so integrally linked. While it’s possible to do DevOps on some level using physical hardware or virtual servers running on hardware in a local data center, it can quickly become impractical—if not impossible. Cloud computing using platforms such as AWS, Rackspace or Microsoft Azure enables an organization to increase server capacity exponentially at the push of a button.

    “Many of the issues and challenges experienced before the advent of cloud are no longer issues: Engineers now working on cloud implementations remember working at companies that feared the Slashdot effect—a massive influx of traffic that would cause servers to fail,” explains Jason McKay, VP of Engineering for Logicworks, in a blog post.

    Tackling Scalability

    There are a number of factors impacting scalability in your environment, which could impact the success of your DevOps. Here are three things to keep in mind to ensure your environment is ready to scale:

    1. Make sure your tools and apps are scalable

    The ability to scale capacity on demand—automatically—is one of the chief advantages of cloud computing, and one why cloud computing is so important to DevOps success. There is more to scalability, however, than just server and network capacity. You must also ensure that the tools you rely on can seamlessly scale with the infrastructure, and that the apps you’re delivering through your DevOps environment are designed to take full advantage of the increased capacity as well.

    2. Continuous optimization

    Some tools and apps will continue to run as the underlying infrastructure scales up or down, but may not run optimally. Make sure that the tools you use and the apps you deploy are able to continuously adjust to changes in server, bandwidth and storage capacity. The tools and apps should be able to take full advantage of the resources available to ensure the best performance possible at all times.

    3. Don’t forget about storage

    Most apps have a data component and that means there is data storage that must be scaled as well—either as a function of the broader network infrastructure scaling or just in response to increased demand for data storage.

    Dan Leary, VP of Products, Solutions and Alliances at Nimble Storage, believes that scalability is emerging as a differentiating factor between storage solutions. “Due to the high performance offered by many vendors of flash-optimized storage solutions, performance will no longer be the primary differentiating factor when considering one vendor over another. Instead, a much greater focus will be placed on the array’s reliability, data integrity, scalability, manageability and in-built data-protection capabilities. We are now getting to the point where performance in storage has been democratized and will change the way people purchase storage.”

    There is one more thing to remember when considering scalability: cost. While it may be easy to automate scalability on cloud platforms, it’s not free, and the DevOps tools you depend on may include licensing fees that could be impacted as your usage scales, so keep that in mind.

  • The Third “S” To A Successful App: Street Smarts

    The Third “S” To A Successful App: Street Smarts

    We have discussed how scalability and simplicity are leading factors to an app’s success, but in addition to having a strong coding skill set, developers need street smarts, or knowledge of the state of the market and of themselves. Why? Because an app is unlikely to reach its full potential and achieve success without the developer knowing their strengths and what markets to build for. That is the third “S” to successful app.

    Know Your Strengths

    A key characteristic developers need to know is their strengths. People are always coming up with new ways of doing things, and it is easy to get caught up in the latest programming language or new database technology. While new tools can be helpful and support your development initiatives, it can also be time consuming to learn how to use them. When you have a great idea, stick with your strengths and build a product you know will work, without losing time.

    Identifying strengths also applies to choosing which platform to develop for. The language or platform you are strongest at or have a specific preference for should be used for at least the first version because it will allow you to make iterations more quickly.

    With strengths come weaknesses, and it is important to be able to identify those as well. For example, sometimes a developer is unable to go out into the field to gather user feedback. When this is the case, it may be wise to look into bringing in a market expert. This is an individual who is an industry expert in a certain marketplace; ideally for a market that you have an app idea for. By bringing in a market expert, you’re able to marry your coding skills to their market knowledge to gain deep industry knowledge of the marketplace while allowing the developer to focus on building something great.

    Market Validation

    Whether a specific market expert is brought in or not, a good app developer should realize the importance of user feedback immediately. More often than not, a developer will have an idea for an app and go into a code trance, where they start hacking and building code but does not take the time to develop a long-term plan for their vision. Building an app without market validation can be very risky, as developers want to get into the code but fail to take a step back to identify what they can use to validate the need for their app.

    AppSumo, an app that offers bundles of software for discounts, started out with a very lean model where the developers would manually input website traffic data into a spreadsheet and then analyze the purchasing behavior of their target audience. AppSumo had no code written at all, but this manual data input allowed them to validate the need and demand for their product. Because of this initial research, when they started writing their code, the team knew the app would be valuable.

    Ship Imperfect Products

    Once you have market validation, you’re ready to build and eventually ship your product. It’s important to know where to cut corners in the beginning so you can ship product as soon as possible, and also knowing where you can’t to ensure you’re shipping a quality product. This is ultimately the hardest facet of management: knowing how much tech debt to accrue and when to pay it off.

    In the case of a brand new app, it may be best to accrue tech debt and solve it at another time because shipping a perfect product that no one uses means you’ve essentially wasted precious time. Willingness to failure is critical to success. It is rare to release an app that has perfect product fit, that perfection comes from rounds of testing and versions. Therefore, a key component of developer street smarts is a willingness to ship imperfect product.

    Final Thoughts

    Gaining effective developer street smarts means interacting with other developers to learn their perspectives and gain their knowledge. Sharing stories, asking questions and challenging yourself are all a part of becoming a better developer. This means taking some time off-screen and going to meetups, listening to talks and elevating your skills by learning new things. You’re also growing your professional network which increases opportunities for your business and your personal brand. These opportunities begin through the online channels with research, but getting out in the world and making face-to-face connections is invaluable for becoming a better developer and creating the best apps.