devopsy.kinsta.cloud

Tag: developers

  • SaaS-Powered FlightPartner Fueled by DevOps

    SaaS-Powered FlightPartner Fueled by DevOps

    Private jet charter company FlightPartner is reaping the benefits of DevOps to build a powerful software-as-a-service (SaaS) offering to keep its customers flying high with no delays.

    “FlightPartner does not own any aircraft but rather facilitates the travel needs of clients by offering air charter brokers (agents to the client) access to a global network of private jet charter availability,” says Douglas Schmohl, CEO and founder of FlightPartner. By acting as a global distribution network (GDN), FlightPartner enables access to real-time availability of private jets.

    What TravelPort is to airline seat availability, FlightPartner is to private jet availability, Schmohl says.

    FlightPartner’s cloud-based SaaS manages data exchanges and transactions through a web browser as a web-based service, requiring no client software installs for the brokers or operators. The service uses Angular.js, a “structural framework for dynamic Web apps” to pass data from the browser to a PHP back end. A database and integrated systems back end builds real-time pricing quotes for available private jets, while the payment mechanism supports search, booking and payment, completing the transaction, Schmohl says.

    FlightPartner uses a Laravel open-source PHP framework, which was created by Taylor Otwell and was originally targeted to developing Web applications following the model-view-controller (MVC) architectural pattern. “This framework allows for rapid development, testing and deployment of our SaaS solution,” Schmohl notes.

    SaaS Technical Challenges

    Schmohl and company endured some technical issues as they built FlightPartner software and worked to raise it to the skies where it would eventually soar. Schmohl was committed to following the lean startup practices of Eric Reis, author of “The Lean Startup,” which redefined the roadmap for a startup tech company, Schmohl says. According to the book, one of the first steps in approaching a new business as a lean startup is to try to float a “minimum viable product,” or just enough product or service to test the market demand for it. “This was nearly impossible for us due to the complexity of the air charter marketplace,” he says, so Schmohl had to invest in a more technically developed product rather than a simple solution.

    In the private jet charter industry, with so much change, so many variables, so many players needing access and so many systems that FlightPartner would need to access, a minimum viable product still necessitated numerous external integrations and APIs. “Developmental collaboration to achieve this was difficult at best and developers often pushed code to production only to create more issues with each deployment,” Schmohl says.

    A second technical misstep—and in the absence of formal development structure—Schmohl’s team wrote code to solve specific functional problems one by one. However, the team found that a method for developing one function would conflict with a different method that was used to create another function, and so on. “The constant patching of the front and back end consisted of whatever worked. This code became challenging to manage and scale,” he says.

    Because of these two technical challenges, FlightPartner’s attempts at a minimum viable product and at following a lean startup approach soon were tangled in a mess of spaghetti code and unmanageable methods, Schmohl says.

    FlightPartner Development Pipeline

    Today, development at FlightPartner is much improved. The SaaS and its components start with a local build that the system pushes to a development server in AWS. FlightPartner developers use SourceTree, a Git client and the Git versioning system to manage software versions. FlightPartner uses quality assurance to manage development and to test code, and developers fix errant code and deploy it to AWS using Elasticbeanstalk. “This enables rapid builds and continuous deployment,” Schmohl says.

    Using a DevOps approach to development brings structure to an otherwise-highly dynamic marketplace. “Offering a SaaS solution as a marketplace presents challenges unlike a typical marketplace. Our inventory (aircraft) is fluid in that it moves often and the price changes depending upon location. This operational challenge translates to numerous and constant development with changes in software functionality,” Schmohl notes.

    These are the operational needs that drive software requirements; the developers’ goals increasingly are to meet those requirements so the software seamlessly supports operations. Development uses a priority status to identify and escalate the builds that are most critical to continued, smooth operational activity.

    Results

    FlightPartner’s development process has created a workflow process that minimizes the resources necessary to achieve operational results. Thanks in large part to DevOps and the cloud-based SaaS system, FlightPartner’s GDN has aircraft available for charter in six of the seven continents, including 35 percent of available aircraft located outside the United States, says Schmohl.

  • The Two-Month Hello World

    The Two-Month Hello World

    How Project Onboarding Can Undermine DevOps Efforts at Large Organizations

    A “hello world” application is the most basic program a developer, typically a beginner, will write to learn a new language. It’s a sort of “everything works”—a stack-checker app to kick off a project quickly and see its results.

    The resulting hello world program is typically a message printed to your screen: “Hello world.” That’s it. No microservices, REST API or class factories, but very satisfying, nonetheless. The stack works; you built it and ran it from scratch.

    The average time for a programmer to write a hello world application code is probably a few minutes, or even seconds. The average time to set up the developer’s computer or workstation and servers to be able to compile/build and run a hello world app may take a while longer, maybe a few hours to set up. And I’m thinking cloud server or container. Publishing a hello world app to Google Play, including setup, build and deployment, takes around a few hours for the first-timer. That’s not too bad. But here’s the deal: It’s a few minutes for your first program to run vs. a few hours for your first deployment.

    That’s a 60:1 ratio.

    Let’s raise the bar a little: Take a large corporation with an IT department of 1,200-plus employees, for example. In that environment the average time to publish a brand new hello world application increases by at least an order of magnitude—even more if the developer is not just onboarding an application but also herself or her team into the organization.

    Cutting through IT red tape is a great distraction as epics, user stories and customer requirements pile up. These tasks can be simple to complex, such as setting up your code repository and development stack; identity, security, disk, network and other stack provisions; boilerplate artifacts and following testing and release policies. That’s a considerable chain of requests and wait periods—enough to make that hello world program loop around for months before it reaches production.

    DevOps for the Hello World

    The ability of a business to respond to market changes start by a quick new-project ramp-up. And in a microservices world that’s often the case. But that’s mostly at the architecture level. How easy is it to provision and manage the external environments beyond microservices? Or, how easy is it to provision and manage the full Dev to Ops stack at your organization, period?

    A good way to tackle this is to offer a menu of automated tasks through a help desk-like setup, a configuration catalog if you may. A configuration catalog holds tasks à la carte for every need in the DevOps cycle. Call it the IT user interface for teams requesting or delivering changes to the organization.

    This is nothing new. ITIL talks about it everywhere. It’s called a service desk. In a DevOps culture, however, traditional service catalogs do not cope well with continuous deployment and infrastructure-as-code approaches. And vice versa. The reason is that there are plenty of needs that tend to fall outside the common service desk and configuration management automation realms, so they end up in an issue tracker, service ticket or, worst case, plain emails, chat and phone call requests.

    From Zero-to-Deploy

    A catalog of configuration services can be application or project-oriented and should keep state, knowing what has been requested and delivered in the past to enable future interactions. Every task offered by the catalog has one or more rules associated with it. These rules, or recipes, do the dirty work and automate provisioning, emails, boilerplate setup and other tasks that antecede or surround most of the core DevOps delivery automation.

    Think of it as your classic service desk catalog and configuration management tool set combined.

    Building a configuration management front end can greatly mitigate and simplify the effort of onboarding applications. Here are some of the features and strategies to consider:

    • Baseline Templates: Offer teams a set of baseline configurations, tasks and services that can be provisioned around project typologies, such as “Mobile App,” “Python-Django-Postgres” or “Java-Spring-AngularJS-Oracle DB.” That means that on Day 1 the developer only has to select the preapproved application type to get started. Push-button onboarding should include straightforward guidelines that guarantee that the project starts on the right foot, including committing boilerplate source code to the repository and directory verification.
    • Incremental Services: Make the catalog offering incremental, providing development teams only the options they should have at a given time for each project at its given stage. A good configuration system will keep track of what every application has installed at a given point in each environment, so progressive options will be offered.
    • Broaden Your Scope: Automate all provisioning tasks in sight. Just because your application is containerized or configuration management is automated it doesn’t mean it’s a push button from start to finish. Reasonably sized applications have many dependencies, approvals and setup steps that are don’t have a place in typical scripted recipes.
    • Manual Tasks: Make the work flow by bringing every manual process into the onboarding pipeline and avoid the need for emails. Make sure to cater to non-standard requests also, as that will help keep everything in one place.
    • Draft and Versioned Provisioning: Allow provisioning plans to be built along with the code, be it infrastructure-as-code or catalog requests. As new requirements come into play, revise provision drafts accordingly.
    • Scale to Environment: For every new environment in the onboarding chain just reapply the previous requests and configurations, but adjust them to that particular stage. For example, a preproduction environment template will probably need to scale CPU, network and memory to closely mimic production.
    • Just-in-time: Make sure provisioning happens before code is delivered. If before means just-in-time, great. Maybe it’s just a matter of putting provisioning before deployment in the automation pipeline. Or maybe it’s a non-standard request that needs human intervention and holds up a release for a short time. The closer provisioning looks like continuous deployment, the better.

    The next generation of configuration management tools must tackle many different aspects of onboarding applications and services to become an effective DevOps front end. This is not just giving developers a “Request Cloud VM” button but the full how, why and where chain that assists teams with writing boilerplate application and infrastructure code, passing the first build or scaling environments correctly. And moving forward, ChatOps and machine learning are replacing the classic request workflow, adding needed insight and foresight to manage project deliveries from zero-to-deploy.

    DevOps practices should start early on in the cycle and offer a great user experience all the way through. A good delivery chain needs to tackle the newcomer workflow, just like the application interfaces of today tend to target first-time users. A friendly and capable DevOps front end is what separates mature organizations from the rest, helping them smoke the from-hello-world-to-deploy test suite gracefully.

    About the Author/Rodrigo González

    roddyRodrigo Gonzalez is the CEO and co-founder of Clarive. Besides running the day-to-day operations, Rodrigo oversees the overall strategy for the company. Rodrigo, originally from Brazil, holds a B.S. in Computer Science from Texas Tech University and has nearly 20 years experience working at tech companies in the United States, Brazil and across Europe, including Lucent Technologies and CA Inc.

  • Developers Cite Startling Lack of Digital Leadership

    Developers Cite Startling Lack of Digital Leadership

    Despite app backlogs, developers say their efforts are underfunded and undertrained, and lack a priority for security

    We’ve long written about the challenges enterprises face when developing the apps they need, digging out of app backlogs and technical debt, and the overall importance of running a productive development pipeline. And, how to keep up, enterprise’s are seeing more and more citizen developers rising to fill in try to close the lag between what software apps they need built.

    Which is why it was disheartening to read in a recently published report, “Investment in Enterprise Developers is the Best Risk Management System,” based on a survey of more than 100 developers by the Application Developers Alliance, that 40 percent of respondents have experienced failed development efforts from lack of funding, and a third are concerned that current projects could be pulled for budgetary reasons. Also, despite survey after survey of executives citing security as a “top priority,” only 45 percent of enterprise developers surveyed believe that their organization sets a “high” priority for security.

    Not surprisingly, according to the survey, four out of five enterprise developers surveyed work on Windows and/or OS X, and 24 percent work on Linux. Yet only 20 percent work on mobile platforms exclusively. And, interestingly, two-thirds of enterprise developers develop their apps exclusively HTML5, hybrid or native, with native being the most common. And, not surprisingly today, cloud is the most common way to deploy apps during development, test or production.

    Unfortunately, many developers report underfunding in technology, processes and training. A third report outdated technology and processes in their organizations—more than half (51 percent) of those in organizations with more than 100 employees.

    Leadership Falling Short in Many Areas

    When it comes to cultivating processes and technology guidance, many enterprises also fall short. Only 37 percent of those surveyed said their organization has a Center of Excellence. Not surprisingly, the larger enterprises were more likely to have a Center of Excellence; however the lack of such centers is more common in North America and Europe, Middle East and Africa. Most every organization surveyed in Asia, however, has a dedicated Center of Excellence. Additionally, a surprisingly low 29 percent of of developers surveyed reported that they are required to attend training, certification programs or other continuing education.

    That’s stunning, considering that increasingly businesses are driven by their digital efforts. There’s no shortage of surveys citing this. In a recent study commissioned by CA Technologies, “The Battle for Competitive Advantage in the App Economy,” 43 percent of survey respondents said that becoming software-driven is central to being competitive today, and nearly 80 percent said that would be true in three years. Broadly, to be competitive in the years ahead, enterprises are going to have to be exceptional software companies in addition to their core current products and services. Yet survey after survey finds that enterprises are falling behind in their app development goals and are having a challenging time finding the talent they need.

    Additionally, the Application Developers Alliance survey found that only 50 percent of enterprises discuss the capabilities and skills they need to succeed, while only 26 percent integrate such competencies in their evaluation process. And, just as when it came to establishing Centers of Excellence, Asian companies stood out, with 78 percent of Asian developers saying necessary skill development is discussed.

    Centers of Excellence, the Application Developers Alliance survey found, also are excellent indicators of whether an enterprise is committed to building secure software. While 64 percent of developers believe they’re “adequately” trained to build secure software, more than 70 percent at organizations with a Center of Excellence contend that security is a high priority where they work. If they work in an organization without a Center of Excellence, that number drops to 28 percent, and 17 percent rank security as a low priority in such enterprises.

    Clearly, if enterprises expect to compete globally in the years ahead they’re going to need to step up their priorities when it comes to budgeting app development, cultivating the skills they need and building the internal organization necessary to succeed as a digital enterprise.

  • 3 Skills Developers Should Master in 2016

    3 Skills Developers Should Master in 2016

    The era of mobile and web is upon us, which has caused nearly every business decision-maker to ask the question, Is now the time to hire a developer to build our app or website? In 2016, businesses are going to be looking for the best developers to join their team and help scale their business. We’ve already started to see this trend, as a recent Harvey Nash Technology Survey 2016 found that developers are the most sought-after of all technology personnel.

    To stand out from the pack, developers must master the critical skills businesses are looking for. The most valuable developers will be those who can adapt to the changing technology landscape and further hone their craft to help companies scale and solve specific business challenges.

    To stay competitive in today’s market, developers need to invest in these three skills to be relevant in 2016 and beyond.

    Master the Skills to Scale a Product or Service

    Startups increasingly are evolving from early stage concepts to building a growth model to take their business to the next level. To be an asset today, developers need to master and specialize in scaling technologies that are key for businesses looking to expand their customer base and focus on technological innovation.

    There is a demand for developers that have not only a breadth of experience in bringing an organization up to scale but also depth in their technical skill set. Having the ability to work on major systems, support large amounts of traffic and customers and maintain the system for several years are critical competencies that companies will be looking for.

    Also, having a data model or database experience is very important—NoSQL solutions such as Cassandra, CouchDB or Riak, as well as traditional data sources such as MySQL, are in high demand. Middleware languages such as Java and Golang also are an important part of a developer skill set when building services and APIs.

    Understand the Customer Perspective

    When building a technology solution, an engineer must be able to look at the product from a customer’s perspective to understand what problem the product is fixing. While technical skills in developing a product and user experience (UX) are very important, the ability for a developer to understand and create a product that solves a customer challenge far outweighs the focus on building the most elegant software.

    Engineers who are able to demonstrate project management skills and have experience around scaling infrastructure and ensuring the platform is robust will be in high demand. Businesses increasingly are seeing a need to support growing customer bases and traffic with utmost end-user reliability.

    Become a Subject Matter Expert

    There is a gradual shift in the developer community, in which engineers are focusing on specializations and becoming subject matter experts. There is an opportunity for developers to create career highlights by merging their talent and passion with opportunities.

    Developers are encouraged to identify an area they are passionate about and focus on refining those skills. By being able to find these opportunities that build on their passion, these developers can demonstrate they are able to solve a challenging problem for a large number of users—a competitive skill in high demand across the industry.

    The overarching strategy of a successful developer career is to invest in learning how to scale a product or service, understanding the customer perspective and becoming a subject matter expert. Investing in your skills as a developer in 2016 can lead to a longer, more fulfilling career.

    About the Author/Sha Ma

    Sha Ma_SendGridSha Ma is the senior director, Software Engineering at SendGrid. She led development teams at both startups and Fortune 500 companies prior to joining SendGrid. In her current role, she drives strategic planning, development and management of large scale projects. Ma is a graduate of MIT where she earned both her bachelor’s and master’s degrees.
  • What If We Had Git for DevOps?

    What If We Had Git for DevOps?

    I used to hate Git. Compared to my previous source control systems (SVN, SourceForge and floppies), it was cryptic, opaque and hard to learn. It was comfortable to create a set of files and folder, then snapshot them as a unit. It was simple to understand and improved my individual productivity.

    But that snapshot created a huge tangled mess when I tried to collaborate with other developers.

    Once I got over the learning curve, Git revolutionized how I wrote code. Small changes made a lot more sense with frequent integrations. I could easily review the impacts of collaborators’ changes. Even more critically, I could keep my work around while I tooled their incremental fixes and improvements. Development—especially open source development—is a team sport and requires tools that facilitate collaboration at the most fundamental level.

    But it was not just learning Git; I also had to adapt how I structured projects and collaborated.

    In that context, DevOps automation has been missing the equivalent of Git. We need composability!

    Can a Git Model Work in DevOps?

    We’ve been building great configuration automation tools (Chef, Puppet, Ansible and Saltstack) over the last few years and now there’s a new wave of cloud templates (AWS CloudFormation, OpenStack Heat, Terraform) entering the fray. These tools are like my original source control systems: they create a vertically integrated stack of configuration that works for a specific case and platform. There are ways to merge them into new stacks, but that ends up feeling like creating a whole new stacks.

    Why does merging scripts create new stacks? Changes in other components often break each other.

    When you reuse someone’s script, their changes can quickly break your carefully built automation. Instead of collaborating, you’re just forking their work. It feels like we’re running around data centers with automation on floppies. When we fix something in one module, it’s very hard to push that back into the community—we have not built clear, functional boundaries between modules that isolate changes in a way that reduces the risk of using other authors’ scripts and makes it easier to contribute improvements.

    Isolating changes does not mean we don’t break integration! It just makes it easier to collaborate upstream.

    The answer in our experience is to think of ops scripts like Git: Even inside of a larger set, we need to make the parts independent and decomposable. How do we translate Git into ops terms? Repos are like full systems—directories map to components, files to roles. Like Git, when we make incremental changes to any item, the whole system is impacted. However, we can still isolate that change down to a single line so we can manage collaborative work. This works when the impacts of changes are isolated (like a text file) and fails when changes are part of a tightly integrated unit (like an image file).

    We already have git with ops scripts. How is this different?

    We are suggesting an even more composable approach. Each ops role should be designed as a functional role with clear inputs and minimal side effects. To achieve that isolation, we’ve been building an orchestration platform, Digital Rebar, to feed the right inputs into existing opts scripts. That allows collaboration within a module without unexpected external side effects. It also allows reuse of modules because changes are well isolated.

    When changes are contained, incremental and managed, the whole system is more tolerant to multiple authors. In that way, we may build system automation that’s more Git-like.

  • Force Multiplication

    Force Multiplication

    Recently, I have found myself reflecting on the statement, “Be a force multiplier.” This usually comes to mind when faced with some sort of burnout: hearing it indirectly from a colleague or friend, or experiencing it firsthand. The intent is good and aligns well with some core tenants of DevOps. Force multiplication fits into the DevOps ethos by encouraging the need to cross-train and collaborate. By working together and sharing knowledge about respective domains (development or operations), team members gain empathy for each other. This in turn has a downstream effect of now enabling better collaboration around the repair/growth/operation/expansion of the delivery pipeline.

    The only challenge I have seen over and over again is that one or more major bottlenecks exist by the time this idea of “force multiplication” is needed in a team. These bottlenecks are often real and result from an imbalance of resources: servers arriving faster than there are resources to rack/stack/configure/turn up. Database administrators cannot handle the number of schema changes needed to be reviewed/tested/deployed. Developers must wait on a certain person or people who have access to publish a package before it can “go live.” You probably have your own bottleneck that you see very clearly in your own mind.

    The next question: Have you (or someone you know) attempted to be a “force multiplier”? The answer always is complex, but a there is always a common theme in noble attempts at team cleanup: lack of training. Think about it: Have you been taught how to be a force multiplier? How to build well-formed teams? How to effectively pass on knowledge? To compound issues even further, our current incentive structure actually celebrates the “heroes” and “unicorns” of the world, calling out these as examples of success. Our economy rewards those first to market, and team-building and normalization often are not high priorities for budding startups today. Unless improvement efforts have a direct impact on the bottom line, often it isn’t a priority. The folks that are brave enough to still even take on the challenge of being a “force multiplier” in these conditions often find little success because knowledge transfer must happen in synchronous fashions: code-pairing, over-the-shoulder demonstrations, etc. These are expensive activities in both time and energy. Opportunity cost is now lost because one resource is cross-training another. Eventually the benefits will be realized, but not today …

    The amount of work is too large. As a result, the can is kicked down the road until such a time that the pain is unbearable and something must be done. At this point, everyone wants to burn down the offending code and start again. But next time, we’ll learn. We won’t make the same mistakes we did last time.

    Bite-sized changes

    The problem is that when looking at the road in front of you, you’re constantly optimizing for the next day or week as opposed to months and years ahead. In the world of quarterly returns, this is the incentive that drives us. Thus, by the time problems are real problems, it is easy to immediately begin the bikeshed conversation and begin designing v2. The next step is to re-adjust the optimization factor. Growing the meat-cloud and getting additional resources often takes too long (three-to-six month onboard time, at least), the time cost of cross-training can be too much, and overworking resources can only go so far. What is the answer?

    Level the playing field

    One of the ideas that I share around ChatOps and Event Driven Automation is to expose small tasks to users. By doing this, you allow team members and others in your company to “consume” actions that you curate. As the curator, you can expose safe actions to users and allow them to do things once relegated to a single individual or team. I’d like to share a real-world story with you and show you how we solved it.

    Bottlenecks at StackStorm

    Recently, our team was bottlenecked in releasing updates to our Puppet module for StackStorm. The team could make changes all day long in the test environment, but getting changes out to the world depended on a single person releasing code to the Puppet Forge. Obviously, this was not ideal.

    Let’s break it down. To release a module to the forge, a user must know:

    • Know how to make a change to the puppet module
    • Update module itself
    • Update metadata
    • Update git repository (tags)
    • Know how properly package up the module for release to the forge
    • Know the credentials for the Puppet Forge

    At face value, this seems minor—just throw these commands into a README, and let others just copy/paste commands. But, my team is not full of Puppet experts. They are developers, eager to help solve interesting problems. But because of this barrier to release, the natural reaction was to shy away from making changes—not because of capability or willingness, but to avoid the pain of friction involved with releasing the code. This meant that any change to the codebase usually required at least two people (change and release). Likewise, any changes that were small or simple naturally got relegated to Puppet people because they knew how to manage the debt and navigate the waters.

    Instead, let’s sprinkle some magic ChatOps dust.

    The Setup

    First, let’s set up our user interface: ChatOps. User experience is a topic that will come up over and over again with ChatOps, so the best place to begin is to decide how to expose commands to colleagues and users. The first thing to set up is the ChatOps Alias.

    Screen Shot 2016-01-27 at 10.56.23 AM

    Source: https://github.com/stackstorm-packs/st2-publish_puppet_forge/blob/master/aliases/publish-puppet-st2.yaml

    I want an all-encompasing command that a user should have to input a minimal amount of information toward. The goal is to have a big red button that can be mashed, and magically a Puppet module is released. This should now happen anytime a user types in !puppet publish puppet-st2 … StackStorm will run the stackstorm.publish-puppet-st2 action with the branch attribute set to whatever the user types in. We choose to reuse the puppet namespace since it is already populated with similar actions.

    The next step is to set up some accompanying Action Metadata for the new stackstorm.publish-puppet-st2 action. We’ll want this to run a new workflow that will take care of all the actions we discussed above. The only variable that I need to get from the user is the branch parameter, so writing the metadata should be a snap. We’ll only take a look at the workflow today.

    Screen Shot 2016-01-27 at 10.59.04 AM
    Screen Shot 2016-01-27 at 11.00.26 AM
    Screen Shot 2016-01-27 at 11.01.04 AM
    Source: https://github.com/stackstorm-packs/st2-publish_puppet_forge/blob/master/actions/workflows/publish-puppet-st2.yaml

    At the top of the file includes our vars section. The puppet-blacksmith application was not built to be run with multiple users, so our workflow needs to be able to adapt. As such, we need to know that the gem will expect a credential file (forge_file) with a username (forge_username) and password (forge_password). We also need to know where our staging directory is (repo_dir), where the upstream target is (repo_url). Simple enough. The next few steps, clone-puppet-module-from-git, bootstrap-puppet-module, cleanup-vendored-gems and set-puppetforge-credentials ensure the build host is setup properly. These actions serially download the repository from Upstream, runs all bootstrap commands, attempts to ensure a pristine directory, and then sets up the puppet-blacksmith forge file. At this point, nothing has actually been done, just some preparation.

    It’s important to note the third step here (cleanup-vendored-gems): This was a small step that often was forgotten and ended up creating archives in the size of megabytes as opposed to kilobytes, which is more reasonable and expected.

    The final steps—tag-puppet-module, build-puppet-module, upload-module-to-forge and push-git-tags—are where the bulk of work occurs. The next steps are focused on running several rake tasks, as well as then tagging the git repository and republishing the package upstream. Most of these actions were enabled by the puppet-blacksmith gem, but now are encapsulated in a nice workflow that is easily consumable.

    And sure enough, if you build it, they will come …

    slack slack_1

    Techniques like this directly enable force multiplication within your team. Essentially, your role shifts from a person “responsible” for delivering to the person for “enabling” delivery. This is a very small example, but powerful. Enabling others on your team to be a force multiplier enables them to keep working, enables you to focus on more important things and also creates a new degree of transparency that may not have existed before. Everyone knows when software is released; anyone has the ability to do this. Add up enough of these small building blocks, and suddenly team members are more apt to take risk and help drive change knowing there is a safety net behind them.

    Worth consideration

    Two very interesting points were presented to me that are worth discussion here:

    Agile Manifesto, rule #1: Individuals and interactions over processes and tools. Tools that try to enforce a workflow of a human? GTFO!

    How far does this paradigm extend? For sure, “process” is the worst. When invoked, it usually means a large amount of paperwork or rules that must be strictly adhered to. The downside is that many of said process are manual in nature and, as a result, error-prone. Likewise, in companies that implement a tool before understanding what they want to achieve or, at the very least, defining a set of principles, the tools end up shaping how the company works rather than the other way around. The point here is that humans are in charge.

    Agreed. What is unreasonable here is the underlying expectation that exists here and across the IT industry: technologists must know an immense amount about an immense number of things. Even in my own domains of expertise, I can think of nothing more demotivating than trying to remember how to deploy code that I haven’t touched in three-plus months. Eventually I’ll remember, but it’s a drag and introduces unnecessary friction. Now, play that out in a larger team and multiply.

    ‘I will become redundant’: I have yet to see this happen. What instead happens is collaboration. Developers can focus on shipping and keeping their code up to date without bugging operations, and operations can care about aggregate problems (capacity, power, storage) as opposed to transactional tasks such as “push this code” or “reset this password.” Most devs I know are not afraid to get dirty, but it’s not where they would rather spend time. Devs want to develop. If you can enable them with better tools to do exactly that, everyone wins. The glucose you were once burning on smaller tasks can and often do become self-service, and you get to go and create even more cool interfaces for users to consume.

    Wrap Up

    Bottlenecks exist. They can and will continue to pop up as companies grow and expand. Sometimes you may find yourself in a situation where bottlenecks accumulate over time as a result of technical debt, while other times you join a new team and that debt may already be gaining interest on amounts owed. Either way, the situation exists where something needs to be done. Instead of going directly for the RPG, burning down the house and starting over, consider smaller approaches to enable others to do the things you do.

    Some thoughts to take with you today:

    • Tend your technology garden: curate, don’t operate
    • Make small, transparent changes for lasting effect
    • Grow it together: don’t control the change

    Until next time!

    About the Author/James Fryman

    jamesNow a senior DevOps engineer at StackStorm, James most recently worked at GitHub assisting in the development and curation of systems scaling within the Operations group. James is also a frequent speaker on the topic of automation at conferences throughout the world.

    Twitter: @jfryman
    email: james@fryman.io / james@stackstorm.com
  • IT Governance at DevOps Speed: Time to Shift Gears

    IT Governance at DevOps Speed: Time to Shift Gears

    Technology plays an incredibly important role in the idea economy, where business innovation is dependent on the software that powers the business. Speed and agility in IT are key to a competitive advantage in this modern economy, in which small startups with disruptive ideas are competing with well-established enterprises and industries. Consider how Uber and Airbnb are challenging taxis, rental car companies and hotels, all based on the power of an idea and software. Yet, traditional enterprise IT struggles to deliver at the speed business demands.

    Have you ever heard a business leader say, “Our IT teams are extremely responsive and very fast”? The typical response is quite the opposite. IT is commonly viewed as a bottleneck to business innovation and, in many organizations, the business simply avoids IT and instead creates some form of shadow IT.

    IT Governance, the Old Way

    The principles and practices of DevOps creates the potential for faster delivery and higher quality. However, one of the largest sources of friction is the traditional focus on IT projects and project governance. Consider this definition of a project from the Project Management Institute:

    “A project is temporary in that it has a defined beginning and end in time, and therefore defined scope and resources. And a project is unique in that it is not a routine operation, but a specific set of operations designed to accomplish a singular goal. So a project team often includes people who don’t usually work together – sometimes from different organizations and across multiple geographies.”

    Given that projects can be expensive investments, and because of limited resources, enterprises justifiably carefully prioritize and plan which ideas and initiatives should be approved.

    Faced with limited resources, enterprises expend significant effort to prioritize, optimize and fund only the most important projects. But no matter how well-intended, this governance approach also struggles to respond to the speed and velocity required in the idea economy. Indeed, the typical portfolio-planning and project-management approach is often an underlying cause of dysfunctional IT behaviors such as excessive requirements, throw-it-over-the-wall deployments and questionable quality—none of which are pleasant.

    Consider how the business approaches user stories and requirements management. In a traditional, project-driven world in which the project is a one-time opportunity, businesses often are given an exhaustive list of ‘must have’ features in their user stories. This behavior is understandable—given the ‘feast or famine’ nature project, funding businesses need to make sure their future needs are included, even if the need isn’t immediate.

    Sadly, this introduces both technical debt and excess complexity from the very beginning. Especially, when you consider that only a fraction of those ‘must have’ requirements will ever be actually used. In fact, a recent IEEE paper noted that 28 percent to 45 percent of software features often go unused.

    Consider how “project success” is typically measured. Project teams typically are evaluated on their ability to deliver on time and on budget, along with other various quality and satisfaction measures. The emphasis on schedule and cost can drive sub-optimal decisions, as deadlines and cost pressures loom. As a result, testing is often abbreviated, as the schedule compresses. Sometimes the scope is reduced to conform to the deadline. This routinely leads to challenged releases in which the service desk is left holding the bag, which ultimately degrades trust and teamwork.

    After the project is completed, the project team often redeploys to other projects, depending on demand for specific skills. The accumulated knowledge and wisdom of the team is lost and there is no further incentive to invest in automation or process improvements because the team has disbanded.

    The DevOps Shift

    DevOps is a very different approach that requires rethinking how work is funded and governed. Fundamentally, a DevOps team is charged with delivering small, high-quality releases, very quickly. The team embraces fast feedback to remove bottlenecks and waste and utilizes relentless automation to make tasks repeatable and efficient. Given that the team will deliver dozens or more releases per year, it is incentivized very differently from the traditional “project” team. The DevOps team often “owns” the application in Development, Test and Production, and has a shared accountability for the value it delivers. The team reduces handoffs and delays and implement small, frequent changes in response to business demand, customer feedback and system performance.

    The friction of traditional governance needs to be addressed using DevOps to enable the enterprise to thrive. One of the ways to remove the governance friction is a paradigm shift from projects as described above to products. It is vital to focus on value and business results along with employee and customer satisfaction. The key difference is that a product is funded for the long haul:

    • A product team delivers ongoing value and the business can count on improvements to meet market demand.
    • A product team does not form and disband depending on funding.
    • A product team is focused on delivering value through the product and will naturally enhance and improve their processes. CIOs and senior leaders need to consider using the Product model as an alternative way to fund and govern their initiatives.

    I’ve talked with several IT leaders who are exploring how to evolve the concept of the traditional project toward a longer-term product orientation. Some organizations explicitly make the change to products, and others are taking a more evolutionary approach. Regardless of the path, the need to evolve the funding model is clear.

    Consider how product governance is different from projects. The product team, in partnership with the business, is funded for a sufficient period of time (year, half-year, perhaps quarter) depending on the business needs. The product team then delivers fast releases of business-driven innovation and rapidly updates the system based on feedback from continuously assessing the product. As the team remains intact, it is able to learn and improve while also accelerating delivery. Additionally, because the product team is able to deliver quickly, the incentive to overload releases with “nice to have” features is greatly reduced.

    Here’s how the mindset is different:

    Project                                                                         Product

    Short time frame                                                   Long term commitment

    Limited funding                                                     Predictable funding over time

    Team disbands                                                      Team owns the product and results

    No incentive to automate                                   Automation improves quality and speed

    Incentives based on schedule/budget            Incentives based business value

    This does not mean there won’t be projects in the future, but there will be a combination of products and projects in most large enterprises. Most likely, we will see projects will be the way new products are proposed, evaluated and conceived. However, once the product is launched, the product team then will operate under a product model and mindset.

     

    About the Author/John Jeremiah

    John Jeremiah_HeadshotJohn Jeremiah leads the Digital Research Team with the HP Software Applications Delivery Management team. He is a software professional with over 20 years of experience including a variety of leadership roles with the U.S. Navy, IT consulting and Fortune 500 IT organizations. His previous roles span from application developer, project and program manager, Director of Testing Services, and Delivery Process and Methodology Director. Prior to joining HP, he was director of Testing Services and Project Delivery Services for Thrivent Financial for Lutherans, where he led the adoption of CMMI Maturity Level 3 process framework.

  • To Be Continuous Podcast: The Clash in Open Source

    To Be Continuous Podcast: The Clash in Open Source

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this episode, Edith and Paul talk about the clash in OSS between developers looking to get paid and software companies looking for things for free.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/244279037?secret_token=s-oeBCP” params=”auto_play=false&hide_related=true&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Edith: So Paul, what do you like about open source?

    Paul: So I guess that depends on whether we’re talking about writing open source or using open sources as part of a company. Writing open source, I like just having all my code out there.

    And I’ll do that as often as possible. Anytime I start a little bit of code, a little library, I’m looking to open-source it and have a readme and have it out there for other people to use. I had a very long discussion, which I eventually won, about the …

    Edith: I like the way you think of all discussions as win or loss.

    Paul: Well, yeah, yeah. It was more on the argumentative side of discussion. But with the other co-authors of the PHP compiler I wrote, which was originally GPL. And I argued very strongly that I actually don’t care if people give back, I just want people using my software. So to me, open source is a way of having people using my software.

    Edith: That’s really interesting. So, I liked what you said at the beginning about, like, you liked open source ’cause it meant getting stuff out sooner. ‘Cause I’m a big fan of that.

    Like, I always say you should just publish stuff. Like, my co-founder John said something really good at our team meeting last week. He’s like, if it’s 90 percent done and you don’t publish it, it’s actually 0 percent done.

    Paul: It’s 0 percent done, yeah.

    Edith: Yeah, and that’s a hard lesson, ’cause you always try for perfection lately.

    Paul: Right, right, right. Well, I think there’s also some really hard downsides about that. So when you publish something before it’s ready, before you really —

    Edith: Well, ready.

    Paul: Well, there’s ready ready, but let’s say ready is, let’s say you publish it before you clarify what the software is to do, or the values or the mission of the software.

    And then you get contributors who are saying, ‘Oh, this is great, but I really want it to do this other thing,’ and you’re like, ‘Well, that’s not exactly what I was going for here.’

    Edith: Well, I’d say most open source projects are so lucky as to get contributors at all.

    Paul: True, true, true. So I’ll give you an example. I wrote the V8 bindings for Closure. So a binding that allows you to write JavaScript and run it in Closure. And I did that for a purely selfish thing, for another, for a nascent pipeline that I was building called Stefan.

    And then people came along and they wanted to do other things with it. They’re like, ‘Okay, let’s add a binding for this other thing.’ I was like—

    Edith: Well, that’s kind of like a pivot.

    Paul: It is, it is, but I really was not interested in it.

    I’d rather someone took it over, took over the project and actually ran with it where they wanted to go, and I can stick on the old version, or I can see if the new version solves it for me, but I wasn’t interested in becoming a maintainer in this thing, I was interested in solving my itch.

    And I just happened to open-source it, because I felt it might be useful for other people.

    Edith: So Paul, I hate to say this, but I agree with you.

    Paul: No, not again.

    Edith: Not again.  No, I mean, I think what you said is exactly right, that you said, I did this for this reason, and I want somebody else to take it over now.

    I think where open source runs into issues where people are like, it’s only this purpose and you can’t do anything with it besides my original thing.

    Paul: So you think that’s a good thing, or you think that’s a problem? Or it causes problems in the community?

    Edith: I think open source is so misused for so many different purposes that it’s hard to even say. Like, so, in Europe, people say open source is good because it means free software.

    Paul: Libre or gratuit?

    Edith: I’m sorry?

    Paul: Free as in beer or free as in, what’s the other side of that?

    Edith: There’s no such thing as a free lunch, is that what you’re talking about?

    Paul: No no no, like people say open source software, it’s like free-as-in-beer software, as in software that has no cost vs. software that has no price tag vs. software that is, they use the French word libre for—

    Edith: Oh yeah, that’s what I was talking about. So I remember I went to Europe in, I think it was like 2004 or 2005, and they just had this big push for open source, ’cause they thought it meant free.

    Paul: Oh, this is like the German government and that sort of thing.

    Edith:  So literally like, so I went to this customer and they’re like, ‘So it doesn’t cost you any more to give us an extra copy.’ So why shouldn’t just everything be free?

    Paul: Oh, right, yeah, well.

    Edith: And so I think open source has been misused by many businesses who think of it as just like, hey, we get all these developers working for free.

    Paul: So I think that the main advantage to business of open source is not that it’s free, but it’s that it’s low-risk.

    Edith: Low-risk?

    Paul: Lower risk. So it removes—

    Edith: Ah, I would argue—

    Paul: Well, so, what I mean is that it removes a certain element of risk from it. So there’s, Joss Belsky wrote a blog post about how you shouldn’t use, I think it was a database that you didn’t have the source code to.

    Edith: Well, so I’ll argue the reverse, and I think open source is sometimes more risky. ‘Cause I talk to people now who are like, nobody’s maintained this open source project that I depend on.

    Paul: Right, right, right. But that vs. a closed-source version.

    Edith: Well, so the opposite risk is that you end up that you are responsible for this open source project, and you’re like, well, actually, I just wanted to use this thing. I didn’t want to do the maintaining.

    Paul: So absolutely, you don’t wanna end up the only person that, well, it’s actually fine to end up the only person which is using this thing.

    Edith: Well, no, it’s not—

    Paul: It’s not ideal.

    Edith: It’s that many people are using it, but nobody is paying money, and then you’re stuck.

    Paul: Because no one is maintaining it.

    Edith: No one is maintaining it, and you depend on it.

    Paul: Right, so, but imagine that no one maintained it and all that you could get was like a binary block. That’s the real problem. The company goes out of business. You have random database version seven, the binary, that only runs on 32-bit, that you’re maintaining, like, companies are literally maintaining COBOL—

    Fortran things from 30 years ago. It sits in a machine, you can’t touch it, you can’t fix it, you can’t apply security updates to it.

    Edith: Yeah, that’s fair. So Tim Chou, who is, he’s an awesome guy, he was early Oracle, he said that people don’t buy, when they buy software, they’re not buying it what it is today.

    Paul: Right.

    Edith: ‘Cause basically, anything that you’re buying today, you could just hire 20 engineers that could build it. What they’re buying—

    Paul: Yeah, you could just copy it.

    Edith: Yeah, what they’re buying is the promise of stability, that it will continue to be maintained. That there is a vision. And that’s actually, I went over to a big customer who won’t let us use their name yet, but they said they love LaunchDarkly because we thought every day about feature flags.

    Paul: Right, right, exactly. This is why people buy SAS, because, because people think all day about CI, about QA as a service, about queues, or whatever it is from various Heavybit companies that I just shilled for, they’re, you can probably figure out yourself which ones they are by looking at the homepage.

    Edith: Well, and Paul, by the way, thank you very much for wearing your LaunchDarkly t-shirt today. It looks very handsome.

    Paul: The, I definitely agree that the benefit of the thing is that you use it every day. But you can do that with open source as well. You can have open source software that, so, for example, our front end is open-sourced.

    Edith: Oh, I’m absolutely a fan of open source. Like, we use open source every day. Our client libraries are open source. I’m not saying that it’s, I don’t think it’s a religious thing vs. open source vs. not, I just think that many times people assume open source is something that it’s not.

    Paul: Yeah, fair enough, fair enough.

    Edith: I mean that if you have open source, that just means that developers spring out of the ground willing and happy to update your software every day.

    Paul: A mixed blessing, definitely. If it even happens.

    Edith: Like, I remember, so I remember when Sun, so I used to work on a, I used to be at Epicentric, and we had a portal server, and we were killing Sun. Absolutely killing Sun. And this was in 2003, so what they said is, ‘Oh, we’re gonna make our portal server open source.’

    Paul: Right.

    Edith: And so somehow this whole thing, we’re like, everything will be open source, and like, developers will spring out of the ground, and it’s like, well, no, developers, they don’t wake up every day like, ‘I’m gonna work on your thing.’

    Paul: Right, right. They will if it’s sexy or exciting. There’s lots of developers who are showing up at Mozilla just to contribute. And in fact, they’ve fought through incredibly large hurdles to contribute to Mozilla’s software.

    Edith: Well, that’s fascinating, ’cause I see so often the reverse, where people, so what do you think about Mozilla made people want to contribute?

    Paul: I think the fact that it was an open-source-first company, and that it was a nonprofit, and it was all about the free and the open web and that sort of thing. Also, people had had very good experiences with Firefox for, like, a decade at this point by the time I was there. Well, maybe seven years, but, you know.

    Edith: It’s fascinating, ’cause I went to dinner last night with the CEO of MySQL, and I don’t know if I should say this, but he said—

    Paul: This is Marten Mickos?

    Edith: Yep. So he gave a talk—

    Paul: He has a new startup?

    Edith: Called HackerOne?

    Paul: Oh, he joined HackerOne.

    Edith: Yeah, he’s the new CEO. So the dinner was kind of like a debutante ball for him.

    Paul: Nice, nice.

    Edith: So what he said was that somebody had asked him flat out how much of MySQL was actually from contributors. He said 1 percent.

    Paul: Seems about right.

    Edith: He said the open source was more just a branding thing. Like it wasn’t actually like a productivity thing. It was more just a branding thing.

    Paul: So you have this in Mozilla as well, that any time that there’s a really avid contributor, that Mozilla would hire them.

    Edith: Yeah, that’s what he said.

    Paul: Right, and if Mozilla didn’t hire them, very often they stopped being a contributor. Like, it was a real sort of a double-edged sword.

    So someone who did it for six months, they were doing it for the love of the game. They weren’t doing it to be hired at, at least I think for the most part, they weren’t doing it to be hired at Mozilla. And then six months in, someone’s like, well, you know, you’re doing a really great job, wouldn’t you love to work on this full-time? They’re like, yeah, that’s so exciting. They’re flown out for an interview. And then it’s like, you know, actually, I don’t think so.

    Edith: Wait, they didn’t wanna work for Mozilla, or Mozilla didn’t wanna hire them?

    Paul: Mozilla didn’t wanna hire them. I mean, this isn’t a particular case. But you know, some subset of people, Mozilla didn’t wanna hire. And then what happens? Well, you kind of don’t wanna keep contributing to the company that just told you to f*ck off.

    Edith: Yeah, I mean, it’s so true. I mean, I think there’s a very human thing about volunteering.

    Paul: Right, right.

    Edith: I volunteer, I was the moderator for a long time for the Lean Startup Circle, and somebody new came on, and they’re like, yeah, I’d love to raise some money so that we can get paid, and I’m like, they cannot afford to pay me.

    Like, I don’t do this to get paid, ’cause—

    Paul: Yeah, it’s like, it’s an altruism that people are trying to do.

    Edith: Yeah, I was contributing like 10 to 15 hours a week, and at my going rate, it’s not, it’s like, I’m not doing this to get paid, I’m doing it ’cause I like giving back.

    Paul: So there’s this thing that we were talking about earlier, about the current discussion around people getting paid in open source.

    Edith: Yeah, and I think a lot of what has fueled open source is the whole 20 percent time. Like, that Google will give their engineers 20 percent for open source, and I know that other big companies, like Twitter and so on, do that too. So these people are, in essence, getting paid.

    Paul: Right, right, there’s a, in most companies, in most startups at least, there’s an unspoken rule that you’re expected to contribute to the libraries that are in your application, and that sort of thing. If you’re spending time upstreaming paths that you’ve deployed, that’s work.

    Edith: Yeah, so I think we’re agreeing, which, I thought we agreed that we were not gonna agree. It’s funny, ’cause I’m Twitter friends with a guy I’ve never met, who’s, he started Jango, and right now what he tweets a lot about is—

    Paul: This is Jacob Kaplan-Moss?

    Edith: Yeah.

    Paul: Okay.

    Edith: That people are working their nights and weekends, and contributing to open source, and then just, people are kind of, for lack of a better word, jerks. People are like—

    Paul: People are jerks.

    Edith: Yeah, like—

    Paul: Even when you’re paid, people are jerks.

    Edith: But it’s more aggravating.

    Paul: Right, right, because you’re doing it for free, or whatever.

    Edith: Yeah, and so people will, like, they’ll be like, ‘I have this pull request and it’s been open for two hours. Why haven’t you reviewed it?’

    Paul: Right, right. I mean, we get the exact same thing. We get customers who email us, or more often it’s the people who get, who are using the free version, who send us a support request and then, 30 minutes later they’re on Twitter going, I can’t believe Circle hasn’t solved my problem yet. It’s been 30 minutes.

    Edith: What do you think drives that?

    Paul: There’s a pattern across the population in general, that people are jerks.

    Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of Circle CI, and Edith Harvaugh of LaunchDarkly.

    You can find the full episode of this podcast here. For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.

  • Don’t Make Me Code Podcast: A New Heavybit Series on Developer Experience

    Don’t Make Me Code Podcast: A New Heavybit Series on Developer Experience

    Welcome to Don’t Make Me Code, a new podcast that discusses developer experience design and the unique challenges you face when building developer facing products. Don’t Make Me Code is hosted by Steve Boak, co-founder and CPO at Opsee, and David Dollar, co-founder and CEO at Convox.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    In this first episode, Steve and David discuss the origins of the term “DX” (developer experience), they share their thoughts on what makes a great developer experience, and they talk about some of the challenges they face when designing developer-facing products.

    [soundcloud url=”https://api.soundcloud.com/tracks/242787724?secret_token=s-09pht” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    David: Hi, I’m David Dollar. I’m CEO and co-founder of Convox, a Heavybit company. I’ve been working in developer experience for about the last three or four years now.

    I’ve founded the DX team at Heroku, worked on a whole bunch of open source tools including Foreman, and now working on an open source platform as a service.

    Steve: I’m Steve Boak, the co-founder and head of product at Opsee. We are a monitoring company, and I’ve been working at several monitoring companies over the past six years, helping to help developers improve the experience of their products as well as find bugs in their code and improve the quality of their products.

    And though I didn’t hear the term DX for the first time until a few months ago, it’s been a passion of mine for several years, and it’s nice to finally attach a name to it. And I’m excited about this first episode because we get to unpack the term a little bit and figure out what it means.

    I’ve thought one of the most interesting parts of your background, David, was that you were part of this founding team at Heroku, that sort of created DX, and I’m interested to hear more about that.

    David: Yeah, so I guess this was maybe late 2012, maybe early 2013, right as at Heroku we started to expand, out of being just a Ruby host into being a host for really any language. That meant we also started to see more, you know our standard user before that had been sort of like, and I was ten user building rails and was very, there was sort of one happy path for that.

    As we expanded languages we had Windows users and Linux users and sort of, everybody coming together to try and build really any thing. And that started to get out of hand really fast. So, really, we formed the DX team, brought together basically the people that were in charge of the various languages at Heroku. Somebody in charge of the Ruby, and somebody in charge of Python and Node. Brought us all together into one team and really decided to try and make a consistent experience, sort of the Heroku way, across all of these languages and …

    Steve: And was it called the Developer Experience team from the beginning?

    David: It was, yeah.

    So the first time I heard the term was from Adam Wiggins. He sort of like came to me and said, “Hey, I think we should build this DX team. What do you think?” And yeah, so I mean it sort of exactly described what it is we were trying to do, which was make this nice experience for people building things on Heroku.

    Steve: And when was that? What year was that?

    David: I think it was the end of 2012, but it’s kinda fuzzy at this point. But yeah, it was a while ago.

    Steve: And it does seem to continue to be an important thing for Heroku. I, just in researching the term of it, saw that there’s a Heroku.com/dx, this is a big part of how they market themselves.

    And clearly it’s been important to Heroku for a long time. Like I said, it’s interesting to now talk about this with a name because a lot of us are doing this and have been doing it for a while but, giving it a name gives it so much more weight and it kind of marks how important it is for all of us. Mmm-hmm.

    David: It’s interest … you think of DX and what does that actually even mean? And its, there’s a certain type of user that you’re talking about now, right? When you’re talking about designing an experience for this user that likes to kind of get under the hood and break things apart and do things their own way and sort of knows what’s going on and maybe you could even consider a power user.

    Yeah, you want to make sure that they can sort of do whatever it is they’re trying to do but that the options aren’t overwhelming and that it’s not complete, just utter chaos.

    Steve: Yeah, it’s almost like this partnership that we as the makers of tools that developers then use to build tools for their customers, if they don’t like the product that they’re using that may show in the product that they’re ultimately trying to build. And so it’s this partnership where we all try to write good code and create great products together.

    And I think that’s part of what makes this different from user experience design as a whole, and why it’s interesting as a topic all unto itself.

    David: I definitely agree with that.

    The most rewarding thing about working, I guess in developer experience that I found is when people do things that you never expected them to do.

    You put all the pieces in place, you thought like, “Oh, maybe somebody will use these pieces to build X,” and it turns out that somebody comes and builds something you never even thought of. I think that that’s just really rewarding to see that.

    Steve: And it’s interesting that, I think a lot of product builders in general would probably share that sentiment in terms of the unexpected ways that people find to use their products, and the hacks that they create, and I’ve heard, even with consumer products you know, people being surprised by how the users are actually using the product in ways they never expected, but with developers that really is to another level because they may try to break your APIs.

    They’re much better at exploring the depths of your code and then creating tools that you never expected with it, which is both surprising and wonderful for us.

    David: It’s still largely about aesthetics, right? Like, you’re a good user experience, a good developer experience.

    At the end of the day is this thing easy to use? Is it pleasant to use? Are you able to figure out what’s going on? Does it make me happy to use the thing at the end of the day? So yeah, it takes a different set of things when somebody’s trying to come in and actually build something on top of your product.

    Steve: I remember in the early days of the Google Maps API just how many experiments and apps were being written with that.

    I think, partly, because of how easy and accessible they made it for developers. And I heard a talk from someone on that team talking about that the product that we all knew and loved then was not the first iteration of it and they actually had a much harder to use initial version. Do you know more about the history of that?

    David: I don’t really know.

    Steve: Yeah, but that was one of my first exposures to using another company’s API and it was a really rewarding experience. And they had all this boilerplate code that you could very easily lay down to plot pins on a map and see that.

    Especially for me as a designer it was so rewarding to get that gratifying output. The actual map on the screen with the pins that I had laid out. I think that is a really great way to measure how effective and how elegant your developer experience design is, in how quickly a developer can make something themselves.

    David: One of the barometers I use for the things that I’m building is you look for the blog-posts people are writing around your tools.

    Are they writing a blog post of like this thing was so insane I couldn’t figure out how to use it? Like here’s my giant blog post teaching you actually how to use it because I had to figure all this out. Or are they talking about how easy it made their life? Like, look how awesome this is, you should try it, too.

    Steve: Yeah, and that makes me think of the JavaScript world today and just how many frameworks are being launched, and the competing arguments for this or that. And in some ways this has really gotten good for the development community as a whole because there are so many choices now, and so many good frameworks to choose from. Almost too many, actually.

    And I think we’re starting to see some of that in the JavaScript world. Just competing ideas of how things should be done. And that almost raises a counter-argument to the value of developer experience design that when we open up too many possibilities and don’t create a set of tools, I think you described it in your Heavybit talks as like, sharply …

    David: I like to compose sharp tools. I mean it’s sort of like the Unix philosophy, right? Like, you want to make tiny tools that do one thing really well. You generally want to have some good examples or defaults, like, this is how to use these tools in combinations to make interesting things. But having those tools be sort of separate and composable is where you start to get really interesting things happening that you didn’t expect.

    Steve:

    Yeah, that each piece does its thing really well but also that when combined they can form something greater and that the whole is greater than the sum of the parts.

    David: Right. Like, a good example of that, it’s not really developer experience, but I have TTT, it’s sort of like an infinite variation of all these little components that you can put together, but they also have some default pipelines that come out of the box, you say, like, “Okay I really just want to make my email go to Google drive or something like, just like these nice defaults.”

    It’s really important, I think, when you have all of these choices to really also have the guides through them.

    Steve: Yeah. We use the Amazon APIs a lot now, and they’re an ecosystem all unto themselves. For every product Amazon offers inside of AWS there’s a unique API and many of those APIs seem to have differing views of the world and so, they in some sense they’re prolific in the amount of tooling they create for developers and they clearly have won a huge share of the market with just how much they offer, but it still feels to me like they’re a counter example.

    They seem to have APIs that a lot of us use but not APIs and tools that a lot of developers enjoy using.

    David: Even despite that, like pretty much everybody is still using their Sphinx, right?

    Steve: Yeah.

    David: So there’s something to that.

    Steve: You know, Facebook’s API or Twitter’s API I can’t speak to how good or bad those are but man, a lot of people use them. And there’s something to that, too, that the product as a whole has so much value and impact on the world that you are sort of forced into a place where you need to use it.

    And it doesn’t absolve you from the responsibility to make your tools great for your developers and your partners in this, but yeah, it’s sort of mind blowing to see how much can be done when you are hitting on a big share of the market.

    David: The AWS database is another great example of that. There’s so many options, so many ways to do things that you can, like, it’s great. You can really do anything. Like the sky’s the limit. But there is, like, finding the happy path can often times be, it’s just a lot of trial and error at the beginning.

    You end up just having to build up some expertise around that stuff and eventually you can, okay now I can understand how to design things using these tiny permanents, but yeah there’s not a really great, like I just have some code and I want to put it on the Internet.

    Steve: Yeah, and part of Convox’s business is really helping developers do this, right? That there are in fact so many possibilities to create an infrastructure in AWS, that giving a little guidance is probably a good way of helping developers through that process.

    David: Yeah it’s basically just trying to create a happy path. I think there’s a lot of choices that you don’t necessarily have to make.

    Steve: Yeah, we too, at Opsee, use AWS heavily. It’s a monitoring product designed just for Amazon web hosting. And when we were creating this product we were assuming we would see a pretty high level of consistency in how our customers are configuring their environments, but that was shot down almost immediately.

    There have been no two environments that are the same and so now we’re dealing with a wide variety of customer environments and it’s an example of just how much people can do with these tools, but how differently they’re choosing to do them.

    That though does get to this other interesting question of how we measure the quality of developer experience and how we know that we’re doing well. And we talked already a bit about time to value and how quickly developers can make tools with our products, but then I think it’s also interesting to talk about how that becomes part of how we price these tools. Like, we try to look at how much development time we can save our customers and then try to price the product based on that. Do you try to do something similar?

    David: We’re trying to think about this as a peer developer tool, so rather than, you know a lot of people try to charge for resources, things like that, we’re doing more of like charging per developer that’s using the thing. So we actually, actually creating value for developers I think is sort of the core of what we’re trying to do, so we’re approaching it from that angle.

    Steve: Yeah, and monitoring being a little bit different, we’re mostly trying to measure how much code our developers don’t have to write when they’re using this tool, and what they would have to do in a competing world, but

    I think the one thing that all Heavybit companies, certainly in all developer tools companies share is that at it’s core we’re trying to measure our value that way: How much code can we save developers from writing, and how much time can we save them?

    And that feels in many ways different from normal user experience design and consumer user experience design. That we have to look at that, the value that we’re creating for our users in order to assess our own value. So as designers of developer experience products I think it’s interesting to talk about the inspirations and the other companies that are doing this well.

    The first time I had ever really thought about developer experience as something to be measured was looking at Stripe and how … You said earlier how blog posts and recognition within the community can mean a lot. And I remember hearing lots of developers talk about the quality of Stripe’s documentation, to the point that it became synonymous with a great developer product.

    Like do you have Stripe-level docs? And so they were always a barometer for, you know, are we doing well and who do we measure ourselves against?

    And I feel like they’ve done a great job not just in creating great documentation and great APIs but in great products. I mean even their web design. They’ve taken all aspects of their design to a level that I think was not common in our world when they were doing it.

    David: Yeah definitely. I mean, we use Stripe and I just remember when I was first clicking through their web I was just like, “Wow, this is really nice.” You just have that thought to yourself while you’re using it, so yeah it’s very pleasant to use.

    Steve: And I don’t know, you know in the early days of Heroku I don’t know if you saw much of this but certainly five, six years ago if we designed a UI that was polished and really nice looking there was almost a bit of hesitation among our developer customers that we didn’t have the street cred like, “No, you can’t be good, you can’t be real because no one in our world would do this, would make a product like that.”

    And I think that’s changed a lot over the last few years, but we were almost fighting this uphill battle to try to create an elegant and beautiful user experience because people didn’t expect that.

    David: I certainly think that sentiment has changed. I think having a good design aesthetic, a sense of that, is actually like seen as a good thing in most developer circles now, as opposed to being like, maybe you don’t know what you’re talking about if you have a sense of style or something.

    Yeah, it seems to be important. People really look for it when they’re looking for tools or products and that’s also something that we tend to see celebrated.

    Steve: That’s about all we have time for today.

    If you’re interested in being a guest on the show, or if you have a DX topic you’d like us to dive into you can reach us at dmmc@heavybit.com or on Twitter @dontmakemecode. To learn more about Heavybit visit heavybit.com. And while you’re there, check out the library. It’s packed with amazing talks from founders of developer tools companies and other industry leaders.

  • Successful technology leaders will embrace the citizen developer

    Successful technology leaders will embrace the citizen developer

    There’s just no slowing down the consumerization of IT. The Bring Your Own Device (BYOD) trend itself has created an enterprise market that will hit $266 billion in 2016, according to a recent report by Markets and Markets. Goldman Sachs released a study earlier this year that found the global software-as-a-service market will reach $106 billion in 2016. And a good portion of those software services are being bought outside of the traditional IT department.

    But the consumerization of IT isn’t stopping with BYOD and cloud services. It’s now extending to application development, more specifically, it’s extending to so-called citizen developers – who are any user (and especially not traditional developers) that are building new applications for use in their organization. They are generally using high-level, newer generation language platforms and cloud services to create their apps. And we’re not talking about creating apps for individuals or a small group of workstations in Excel or Microsoft Access. No, users have been creating for those years. We are talking about full-blown enterprise apps.

    All of this is part of the broad Shadow IT trend, which we have written about extensively. Have a look at our post The Growth in Data Center Uses for Shadow IT for a good overview. The research firm Gartner expects, by the end of this year, that roughly 35 percent of IT expenditures will be managed outside traditional IT, within this shadow IT “channel.”

    What does this mean for CIOs and IT leaders? The emergence of the citizen developer has both positive and negative aspects to it, but in the long run there is likely more value to be found effectively managing and encouraging these citizen developers, than there is in fighting them. For instance, the citizen developer is a front line asset when it comes to clearing enterprise application backlogs.

    We’ve covered it many times how much of a challenge enterprises have today when it comes to meeting the demand for applications, especially mobile apps. As covered in Clearing through the app backlog clutter, 85% of enterprises have an app backlog, with about half of those that do find themselves lagging behind 10 to 20 apps.

    When workers, business managers and executives find themselves unable to do their job as effectively as they need or would like and that IT can’t deliver the apps they request, then these citizen developers are more likely to try to build the apps they need on their own. And the numbers of these citizen developers are increasing.

    According to Intuit QuickBase’s State of Citizen Development Report (Intuit QuickBase is a cloud-based workspace that enables the creation of apps), and based on results from attendees (citizen developers or IT pro who worked with citizen developers) at its EMPOWER 2015 user conference 97% of respondents had traditional word processing and spreadsheet skills, 36% had front end web development skills in HTML, CSS, Javascript, and 8% had traditional coding skills in Java, .NET, Ruby, PHP, C++

    Interesting, 75% of respondents in IT said that they develop more than three-fourths of the app and leave the “last mile” to the citizen developer. And, not surprisingly it is the apps they built to get work done (65%), 42% built apps to run the business, while 27% said they build apps targeting customers.

    These citizen developers can go a long way to help clear the application backlog. Per the report, (29%) have seen a 2X or more increase in application development speeds. Over three-quarters or (76%) of respondents cited operational efficiencies as the primary reason for using the platform. And a whopping (62%) of Citizen Developers responded that it takes less than 2 weeks for them to complete an application, on average,” Intuit QuickBase’s John Carlone wrote in this blog.

    However, all isn’t rosy. IT needs to always ensure that all apps are created and used to policy, that regulated data isn’t being used out of an allowable scope for example, and that appropriate security controls – such as access to that data – be enforced.

    So it’s important that IT is always on the lookout for these apps, catalogue and manage them, and when they are not up to policy, enterprise developers and other teams can then work to get them there.