devopsy.kinsta.cloud

Tag: product development

How Systems Engineering Can Help Avoid Product Development Pitfalls

Product development programs all start with great optimism regarding their potential to provide an excellent product to customers, along with excellent profits to the company. But research shows that a majority of product developments end in failure. The reasons for each failure are unique, but the overarching reason often can be described as the lack of systems engineering.

I managed a program several years ago that is a great example of how programs that have an excellent idea, sound technical basis and talented people working on them can still fall into this trap, and how systems engineering and a systems mindset can rescue these programs from the brink of failure and propel them to a successful conclusion.

Here’s the scenario: Scientists working for a Department of Defense electronics supplier devised and patented a method for doubling the frequency range of a radar amplifier in the same physical footprint. Additionally, advances in technology allowed two of the most sensitive components to be replaced by components that were substantially more reliable, increasing the amplifier’s projected life by more than 500 percent. Further, these new components were not destroyed by the absence of a vacuum, allowing easier, faster and cheaper repair of the parts. Funding was secured, and the product development process commenced. The project was not plagued by requirements issues (though many product developments are), as the performance requirements were identical to the product it was replacing, aside from the increase in bandwidth and reliability.

Unfortunately, prototypes were failing to meet the requirements in either bandwidth, power or both. As the program neared the end of its allocated development funding and the pressure on the engineering team increased, each subsequent amplifier prototype was billed as being “full spec,” with multiple changes made in each iteration to meet this promise. Each prototype failed to live up to this billing, and the program was eventually deemed a failure and scheduled for termination.

How did this program devolve to this point? The scientists and engineers involved were brilliant individuals with a history of successful projects and a long list of patents to their names. The requirements were well-understood, and the program had avoided the all-too-common pitfall of neglecting to consider how the piece would fit into the overall system and its effects on upstream and downstream components, overall system power and cooling budgets, etc., which tends to happen in projects run by component designers that focus on the design and capabilities of their particular widget. However, as the program budget and schedule for development was steadily consumed, those involved felt obliged to make each next prototype do everything, rather than taking a deep breath, seeing how far they’d come and how far they had left to go and then creating a road map to get there.

When I was brought in to the program and we had secured some additional development funding, we got down to the business of creating the road map. In addition to the electronics company’s scientists and engineers, we brought in experts from both academia and a federally funded research and development center to review the initial concept and progress to date, determine whether the program had a path forward and, if so, delineate a series of experiments that would culminate in a successful prototype.

Termination of programs is always an option and should not necessarily be viewed as a bad outcome for a program in dire trouble. In evaluating whether it the right move, look beyond the siren call of spent budget, take a stark look at the chances for success and determine if the program or project can still deliver its anticipated benefits. Many programs in trouble attempt to recover by “dumbing down” the requirements; while that may offer an “out” for a development that is in trouble and allow the program to move to the next phase, ask whether continued investment in the system worth the lesser benefits of a product that meets the revised, easier requirements.

After careful review, we determined that the program had a path to success. A series of six prototypes were planned: three experiments to evaluate the performance of individual design changes, a fourth to combine successes from the three experiments, a fifth planned to be production representative/fully specification-compliant and a sixth to show repeatability.

With this new plan in place, the project was able to proceed in a systematic fashion without pressure to produce instant success. Previously, multiple improvement ideas were incorporated into each new prototype with the hope that this would be the full-spec version, and the program could then proceed into production (and the company could reap its financial reward). Experiments were defined up front, and their results were evaluated without the uncertainty of multiple changes masking the successes (or failures) of individual changes.

Success was achieved on the fourth prototype, and the program eventually was successful in creating a significant capability and reliability enhancement for this weapon system. Much of the credit belonged to the hard work and dedication of the engineering leader for the electronics firm (who received its Product Development of the Year award). But even more so, the credit for this dramatic (and very unusual) recovery was earned by the team that had the wherewithal to take a step back and apply basic systems engineering principals to lay out a plan to recover the project from its death spiral.

All too often, companies or program offices bring in a “Cleaner,” but many of these are just tyrants who bully everyone into working long hours, hoping that if you try everything under the sun, something will work. A systems engineer, however, has the composure to take a step back and review, despite the immense schedule pressure. Rather than rushing into the next setback, a systems engineer will look at what the program or project is supposed to accomplish (high-level requirements), review the efforts underway to support those requirements and set a plan with discrete steps and measurable progress points. By not trying to do everything all at once, wherein successes and failures are indistinguishable and possibly masking each other, having a systematic (there’s that word “system” again) approach allows the system engineer to move toward a path that will result in meeting the original requirements.

About the Author/ Jay Schatz

Jay Schatz is senior program manager at Base2 Solutions. He is a high-performing technical project management and program management professional with the breadth of experience that comes from over 22 years of experience managing programs for the aerospace industry and in the US Air Force. Jay is currently providing program management and systems engineering support to the Boeing 787 program, providing expert advice to clients on program management, supplier relations and engineering solutions. Connect with him on LinkedIn.

December 8, 2017
In Product Development, Transparency is the New Agility

The rise of DevOps has shown that agility and speed are key drivers for turning a business into a high-growth company. But as product managers, that’s not all we can take away from the DevOps culture. Those same methodologies that enable DevOps to function, such as having cross-functional teams and embracing a culture that embraces transparency, can also help product teams adopt experimentation.

The product cycle should be an ongoing opportunity to try new ideas, gather data, make changes and celebrate failure. Teams with an experimentation mindset are better-suited to accept these goals, knowing that they lead to learning and succeeding quickly. Yet, experimentation and developing innovative products will succeed only when there’s a foundation of transparency—and not just about financial data and company successes, though that certainly helps. Everyone on the team needs the freedom, insight and tooling to make big bets, as well as the accountability to see them through.

This is where DevOps comes in. Leaders at the C-suite and VP level need to apply DevOps systems to product development to stay agile. Here are three ways to do so:

Make Development a Collaborative Process

DevOps brings IT and engineers together to meet customer and business needs. Product development teams should adopt a similar cross-functional team culture to remove silos. When product managers, sales managers, success managers and developers have regular and open conversations about industry trends, customer requests and big successes, they’ll have more insights in the customer experience and product road map that can help each team be more effective. For example, sales managers interact with customers daily and bring a broad perspective about customer problems, while product managers can gauge whether the ideas align with the business goals and developers can assess whether ideas are feasible to implement.

At Optimizely, I encourage my team to brainstorm several big bet ideas on a quarterly basis and discuss whether these ideas are new offerings we need to build or buy or whether we can build on into our current offerings. This collaborative process gives the product, sales and developer teams insight into what other teams are thinking about and encourages all of the teams to put their heads together to experiment with new ideas. Recurring brainstorm sessions that incorporate recent trends will ensure the team stays agile.

We also have a collaborative product feedback process that allows go-to-market teams to suggest features to the product team at any time and have a conversation around the business benefits of feature delivery. This is accomplished with pods of product areas comprised of product, design, engineering, product marketing, success and sales. The members of the pod meet weekly at standups and biweekly at product reviews to ensure alignment between what is being built and what the go-to-market teams are hearing from customers and prospects.

Shop Around Your Proof of Concept and Be Ready to Course-Correct

Product development isn’t a one-and-done process. It requires building minimum viable product, collecting feedback and revising. At Optimizely, we try new ideas and first get feedback from our internal teams and around five trusted customers to start. We implement their feedback and updates before rolling the feature or product out with a slightly larger subset of beta customers, around 20 or 30. With even more feedback, we continue to improve and build on the product, rolling it out to larger and larger customer groups under beta before making it generally available.

As feedback is collected, it’s important to bring our cross-functional teams together to analyze the comments and bring their different perspectives, whether it be customer-centric or developer-focused. We have to be willing to accept the feedback we get and make changes based on it. A recent example is when we rolled out Optimizely X Experimentation, which was built with reusability in mind to overcome some of the issues of Optimizely Classic. While this was a worthwhile goal and customers wanted reusability, we ended up overcorrecting and having to build new workflows after the fact. These ended up looking more like Classic and were more user-friendly but less reusable across experiments.

Hire Product Managers who Prioritize Trust and Transparency

People underestimate the importance of getting along with your colleagues and building trust with them. That doesn’t mean you can’t disagree or have different approaches. It simply means you’ve established trust and open communication—and you’re open to hearing and considering these differing opinions. These two elements are key for staying agile and quickly adapting to industry trends. You often heard the phrase “strong opinion, weakly held” at Optimizely, which encourages lively debate.

At Optimizely, we evaluate trust and transparency by having engineers and marketers interview prospective product managers to make sure the candidate can build trust with different teams and see pain points and priorities from various perspectives. During the interview process, it’s equally important that people acknowledge their mistakes and are able to learn from them. If a candidate can’t think of a mistake or explain what they learned from one, they likely aren’t taking enough risks or reflecting on their failures. Opening up about failures creates trust that leads to better teamwork and ultimately better business decisions.

Ultimately, we can all take a page from the DevOps playbook to make trust and transparency a priority on our teams. Those who do will see new business ideas flourish and team members work in a more cohesive way.

About the Author / William Gradin

William Gradin is the VP of product management at Optimizely and is responsible for building the Experimentation Platform to help businesses boldly test their assumptions and respond more quickly to ever-changing market dynamics and customer expectations. Prior to joining Optimizely in 2015, Will worked in Product Management at Salesforce, where he owned the Platform teams for the Salesforce Communities product. Connect with him on LinkedIn.

May 26, 2017
Benefits for the Product Team in Working Support Cases

An increasingly popular practice that’s being adopted by DevOps groups are developers working regular shifts in front line support. The thinking behind this exercise is pretty sound. When exposed to the real world problems and frustrations customers face when using their software, a developer’s work becomes more about supporting that customer experience and increasing customer satisfaction and less about “Get the tests passing and ship it fast!” The practice also increases an individual developer’s understanding of the larger solution their company provides. If a particular developer is master over a small part of the total application, working support will give them valuable insight into the bigger picture of the problems they’re solving and how well their corner of the world is supporting the larger solution.

We practice this at my company, Datical, but on a wider scale. In addition to the app dev engineers every member of the product team, myself included, regularly works support cases in direct contact with the customer. It has been a pretty eye opening experience for all of us and we are most certainly realizing the promised benefits of this practice.

Our product team operates with a single guiding principle: Well defined, customer driven requirements yield excellent solutions. When you schedule calls or take customers to lunch, you get a lot of great feedback on where your product should go and net new features you should consider implementing to improve the overall solution. The conversations we’ve had with customers in this way are generally very positive and supportive with a focus on future direction and growth.

Working a support case is an entirely different and much more crucial customer conversation. The conversation that takes place during a support interaction is centered on a pressing need. The customer may need clarification on usage that isn’t as intuitive or well documented as it should be. They may be dealing with a gap in previous requirements or a coding defect that is keeping them from completing a goal. Whatever prompted the call, it’s occurring because the customer is not delighted and we are getting a firsthand, real time account of what is problematic and what we must do better. These needs translate directly into our product plans and requirements.

But there’s more to glean from these interactions than the matter at hand. When the customer is amenable we like to use GoToMeeting to remotely look over their shoulders as they demonstrate for us the issue they are having. During their demonstration, we get additional insight into the mechanics of our solutions. How do customers interface with our products? Are there minor inefficiencies our users have to contend with that they aren’t reporting? A few of the most well received features we have delivered since adopting this practice have been very small stories that were pretty easily addressed, which removed a minor annoyance that was encountered frequently in our workflows. We’ll release these fixes with a minimum of fanfare, but they are almost always noticed, commented upon and appreciated by the customer.

The last class of lessons learned from our foray into front line support pertains to my new favorite non-word, troubleshootability. When something goes wrong, how easy is it to diagnose the problem and how long does it take? How detailed and centralized is your logging? Are users being updated on the progress of long running operations? If an error is encountered due to customer misuse, is the customer appropriately guided to correct the error? Anything that we can do to shorten the cycle and cut through the murk when something does go wrong improves the overall experience with the product and increases customer trust.

Working support in product management and engineering has been a really great exercise. It’s given us a deeper understanding of our solutions, our users, our strengths and our weaknesses. This understanding has led to requirements that are tightly focused on our customers’ needs, holistic development practices concentrated around comprehensive customer workflows, and a solution that clearly communicates what’s going on as it happens for fast diagnosis and remediation of issues in the field. I highly recommend it.

About the Author/Peter Pickerill

Pete Pickerill is Vice President of Products and Co-founder of Datical. Pete is a software industry veteran who has built his career in Austin’s technology sector. Prior to co-founding Datical, he was employee number one at Phurnace Software and helped lead the company to a high profile acquisition by BMC Software, Inc. His ability to understand product demands from a customer’s perspective and translate those demands into actionable product and development plans has led to expanded duties at every company for which he’s worked. In his time at Symantec, Pete worked directly with large financial services companies and online retailers to implement and improve online fraud prevention solutions that were used daily by millions of people worldwide. In addition to managing a development team and product schedule at BMC, Pete worked directly with large financial services institutions to provide the features they needed to achieve greater efficiency and cost savings in their application deployment processes.

Connect with Peter on twitter: @PetePickerill

August 5, 2015
A Critical First Step in Bringing Dev and Ops Together

We all want to deliver excellent software to our customers. We now have a greater awareness than ever before of the interconnectedness of all the activities required to do so. The agile movement started this but it was not until the advent of DevOps that Ops was considered part of the full picture. In this article I will explain an effective first step that agile organizations can take to complete the picture and bring ops more completely into the product development process.

Delivering excellent software starts with product definition and flows from those initial stories and designs, into dev, verification and testing, into production and then back again in the form of customer feedback and bugs. In reality the feedback paths represent a transitive closure of all activities just mentioned as you can see in the diagram below. Agile recognized the importance of feedback loops in good product development. Agile also realized that for the right information to flow, walls had to be broken down. The motto “Done means tested” arose and indicated that product owners were not done with a story, could not consider it closed, until it had been tested and they had verified it. This represented the integration of PM, development and QA functions. A huge step, but one step too short. Operations was still not viewed as a key part of building excellent software and there was not much insight into the fact that the feedback loop did not stop at testing. DevOps has recognized this shortfall and taken it to its logical conclusion – “Done means in production.”

Product development is a cycle, a cycle that operations is a critical part of. In describing how all parts of the cycle can be integrated lets start at the most natural beginning; product definition. Product definition activities produce stories, designs, functional acceptance criteria (A.C.) and prioritization. In this phase we don’t produce all we need for the finished product but just enough to get us going. With popular agile methodologies like Scrum this is ideally not done solely by a product owner but instead in conjunction with dev and test at story grooming meetings, design reviews and sprint planning. In these meetings dev, QA and product communicate and negotiate stories, comment on designs, and flesh out acceptance criteria. In this way the stories, designs and acceptance criteria are improved and everyone gains an understanding of what is being built; they also become invested in it. A story may look something like this (abbreviated):

Story: Blog administrator can change user roles SO THAT users can be managed effectively

A.C: Given blog administrator is logged in
And on a user profile
When he selects a permission scheme from a permissions drop-down menu
Then the users permissions are updated and stored in the DB

This is a great start but it is not enough. If we are to be excellent we must modernize this just a bit and add in some DevOps spice.

A first step towards bringing ops effectively into the cycle is to include someone responsible for actually operating the software in the story and acceptance criteria grooming sessions, design reviews, and sprint planning meetings. The critical aspect is that when the team negotiates the acceptance criteria associated with a story it will be ops responsibility to ensure we for each story or feature the team adds success and failure metrics, initial alarm values and any operational functionality such as a kill switch that will be needed in order to operate the functionality effectively. So a story will now contain things like the following in the A.C. (acceptance criteria).

Given operation X fails
Then the metric operation-x:failed will be sent through the metrics feed

We will also alter our definition of done to make sure that a story will not be closed until we meet the ops driven acceptance criteria. For example: we can see the metrics we have defined in our A.C. show up on an ops dashboard. There are so many benefits to this such as folks that are unlucky enough to be on on-call will have participated in this story grooming and will know exactly what these metrics mean when they encounter an issue with them for the first time in production. Furthermore, they will know what operational levers they can pull or request to be pulled.

This first step represents a connection from product definition to dev, to QA, to operations. This connection is perfectly natural and intuitive when you think about it. Being a true DevOps organization and developing operationally excellent products requires more but having connected product definition to operations as we have provides a nice toe hold from which to flesh things out further. Best of luck. If you have any questions leave them as comments and I will respond as best I can.

April 2, 2014