devopsy.kinsta.cloud

Category: Doin’ DevOps

How Continuous Delivery is Changing Software Development

When you start to implement a new methodology, it’s very easy to get bogged down by the specifics and lose sight of the overall goal. With Continuous Delivery, we see endless debate about whether to use Puppet, Chef, or Salt for deployment, or discussion about how to build a CD pipeline with containers. These are just technical challenges to overcome. They’re implementation details that have little to do with why CD is such a powerful choice for businesses.

What’s the most important metric for your software? It’s not the speed of delivery, the number of features, the performance under load, or the defect count. It’s end user satisfaction. The question you need to ask and keep asking is – How well does your software cater for the people using it?

The CD mindset keeps you laser-focused on that metric in two very important ways.

Real representation for the end user

Too many organizations are still just paying lip service to this concept. The developers and testers working on a piece of software often have no contact with end users, and that can mean that they have very little idea about their needs. If you really want to create a great piece of software for someone, then you have to understand how they work. What are their requirements and constraints?

Traditionally, software development has been siloed. Individual departments have focused on their own tasks first. Developers may be delighted with their clean repository layout. QA may be satisfied with how complete the test coverage is. But neither of these things tells you about the end user experience. The system may be completely unusable.

Personal contact is a great idea. If your team actually talks directly to end users, asks questions, and learns to empathize, then you’ll see the benefit in the product.

CD also encourages greater visibility. Not just oversight for management, but for the whole team. Everyone, from development, through QA and operations, to the actual end users should get a look at how the overall service is being put together and delivered. Have channels in place so that anyone can contribute to service improvements, regardless of their area or expertise. Greater visibility over the process, and an ability to influence it, makes for a really cohesive, motivated team.

Collecting, analyzing, and acting on the right data

You can empower everyone on the team to make suggestions and you can say that end user satisfaction is paramount, but without collecting the right data you aren’t going to be able to realize the benefits of this approach. You need help understanding where the opportunities for improvement are, and it’s vital to measure the impact of what you’re doing.

In the big data age, it’s common for companies to collect all sorts of statistics. It’s easy to automate the capture of technical operational details such as CPU load or memory usage, but how much do they really tell you? Though it may be more challenging to find out whether your end users are satisfied, it’s also a lot more important.

Eliminate the risks of educated guesses and gut feelings. Apply a solid methodology to improve your service based on user feedback, just as you hone technical performance based on technical data. You don’t need to spend hours brainstorming new features. You can analyze end user interaction, find a pain point, create a hypothesis about the cause, tweak the system to address it, deploy the change, and measure how it really performs.

Adopting this process and repeating it over and over will improve your service. Because you’re rolling out small changes, it can be done quickly and efficiently, with easy rollback if required. As long as you’re careful to measure your hypothesis and the effectiveness of your solution each time, there’s no downside.

The ideas here are nothing new. Your marketing department most likely already conducts A/B testing and measures user interaction. But how many companies are really taking the holistic approach, applying these concepts also to software development and delivery?

To make the end user the focus, and really leverage the potential of Continuous Delivery, the whole team has to pull in the same direction. Agile and CD methodologies help you cut down barriers, set shared goals, and keep the focus firmly on the customer to revolutionize the way software is delivered for the people who need it.

About the Author/Andrew Phillips

Andrew Phillips heads up product management at XebiaLabs, a provider of continuous delivery automation. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

July 13, 2015
An Open Letter to DevOps “enthusiast”

Dear DevOps “enthusiast”,

You are confused. But I am confused as well. For approximately two years now this DevOps thing has been both exciting and frustrating. And in ways that are totally different for each of us. At the intersection of all the disparate confusion is the “perfect storm”. What that means for now is that we are all hunkered down in the basement, reading books on “DevOps” to pass the time, and better understand the thing that is attempting to destroy us.

(more…)

July 9, 2015
Appster Must Produce Mobile Apps Using Nimble DevOps Practices (It Has No Choice)!

In order to meet the many challenges of producing loads of different app/server projects large and small in extremely rapid succession, Appster collaborates on, automates, and dashboards everything it can. “Unlike traditional product companies (i.e. one product with lots of developers) or corporate IT shops (i.e. silos of products and services that rarely change), Appster has to handle a wide spectrum of software projects across a myriad of industries and vastly different client profiles,” says Martin Halford, CTO, Appster. Many of the projects Appster works on are mobile apps.

To do this, the DevOps tools and processes must tackle each new and unique development project in a continuous project stream with the same fervor that any large development project requires. “Our DevOps focuses on automating testing, builds, and deployments for highly custom, highly complex app/server software development projects,” says Halford.

The tools that help Appster to accomplish this include the XCode IDE, the Eclipse IDE, Atlassian’s BitBucket, SourceTree (for source control), and Jira (for Agile project and issue tracking), Jenkins for build automation, Amazon Web Services, and Klipfolio for dash-boarding, lists Halford. “And due to the short turn time and short duration of app/server projects—between 6 to 16 weeks—Appster does not at present practice Continuous Integration,” says Halford.

Appster has three main objectives: to deploy only from BitBucket, to protect the non-development servers from the developers, and to display the build statuses via dashboard to all. Deploying only from BitBucket holds the source code repository accountable as the “single source of truth”, says Halford. In order to protect the non-development servers, developers do not have root level access to the testing, staging, or production servers, only to the development server in this four server stack, Halford explains. Further, the QA Team has access to the development and test servers as well as limited access to the staging server, but no access to the production server. Only the Dev Ops Team / Linux Admins have access to the production server; they also have responsibility for the maintenance and administration of the entire server stack, Halford says.

App Dev At Appster Prior To DevOps

Prior to moving to DevOps approaches, Appster developed apps using manual steps and documented processes that were prone to human error, says Halford. “The process was also prone to human shortcuts and attempts to hot fix builds by circumventing source control and documented protocols such as those that demanded that no one make direct updates to the production server,” Halford says. These issues were not immediately uncovered.

A Typical Trip Through Today’s Appster Development Sprint

CTO Martin Halford shared his take on a round trip through Appster’s development process today. Appster will first generate development servers from saved generic Linux server images, which the Dev Ops Admins will also secure. Servers will ultimately include development, testing, staging, and production servers (or DEV, TST, STG, PROD). Appster will then generate source repositories in BitBucket, and Jira Projects that it will link to BitBucket. Appster will also create a product backlog using an Agile Scrum Software Methodology.

According to Halford, the development team will begin to develop software using Sprints, and each Sprint will follow this process:

A. The developers will check out their code and associate tasks with User Stories and / or Bugs in Jira.
B. They check code in as Changesets that are attached to tasks, which are in turn attached to User Stories or Bugs.
C. Developers automate and deploy builds as needed to the development server.
D. At the completion of a Sprint, the QA Manager selects completed tasks (User Stories/Bug Fixes) for deployment to the test server.
E. BuildMaster initiates the build, which pulls Changesets from the source repository, generates the build, and deploys it to the target server for testing.
F. After a successful Smoke Test, the QA team performs more involved tests to verify the build.
G. BuildMaster follows a similar process to deploy the build to the staging server for User Acceptance Testing.
H. The next Sprint begins and the cycle repeats itself.

Concluding Thoughts

“DevOps can heighten the effectiveness of mobile development by forcing and enforcing discipline on all parties, preventing human error, speeding QA testing cycles, effectively stopping defects, and highlighting inefficiencies in the existing development process such as could be alleviated through additional developer training and mentoring,” says Halford. Further, having a dashboard that is visible to all parties providing transparency into server issues so that Dev Ops Admins can respond immediately during build or production is critical to the DevOps process.

July 8, 2015
The Wonder Pill for Mobile Application Success

Dr. James Brown stunned the British Science Festival by announcing a wonder pill that could reduce your weight, lower the risk of high blood pressure, cancer, diabetes, depression, and anxiety. With everyone in attendance eagerly awaiting what it was, he plainly exclaimed that the “wonder pill” was daily exercise.

The most effective solution was the simple one. The same applies towards developing a successful mobile application.

There is a sure fire way to generate greater sales, leap frog the majority of your competition, promote wider brand recognition, and win loyal customers. The wonder pill: make sure your mobile application works.

“The majority of mobile applications get a one star rating from users due to poor quality. Either the apps don’t work properly or they perform too slowly. The best way to guarantee the financial success of a mobile application is to simply test it properly so the user has an ideal experience.” says Tal Barmeir, CEO of Experitest. If a mobile application has a function that isn’t working, 80% users will simply delete it. In some cases they will tell their social media followers about how lousy the company is. They won’t pan the app mind you – they will put down your entire company. Performance is no different. A slow performing application, even if it works, might as well be broken.

If a daily walk is the simple way to quality health, mobile application testing is the simple way to a quality app. If you can guarantee that your application works properly, your mobile application just leapfrogged the majority of its competition that cannot make that claim.

Is Mobile Testing as Simple as a Walk in the Park?

To guarantee top quality of your app you have to test it over all types of devices, and for each device you have to test over several mobile operating systems. For each permutation of device and operating system you need to make sure your application works fast. But be warned! Just because it works fast over a cutting edge local network in Singapore doesn’t mean it will perform as well in rural Indonesia.

Add in smartwatches, other wearables, and apps managing the internet of things and the matrix of test scenarios you need to cover can seem endless.

The best testing tools make it look easy. You can set up an on premise mobile device lab in your company, and using the cloud attach as many devices as you want to it. Each device can have a different operating system, creating for yourself a large a matrix of devices and operating systems, expanding your test area to meet your needs. Your quality assurance testers can access these devices by remote using automation testing tools from wherever they are in the company. A manager can decide to assign devices to different testing teams working in different time zones to create a 24 hour nonstop testing operation. By including a Network Virtualization tool, testers can simulate any local network condition on earth. They can verify that their application works as smoothly in Nairobi as well as it does in New York City. The best testing tools also constantly update to enable you to test for new technologies like wearables and the IoT.

That’s why mobile testing is the wonder pill of mobile application success. It makes your mobile application works. It makes sure your mobile application performs fast. It makes sure your customers and prospects know that your mobile app is the best there is – and they tell their friends about it.

About the Author/David Fink
David Fink is a product evangelist for Experitest, a leading provider of mobile test automation, manual testing and monitoring tools for mobile applications. David worked on Wall Street during the internet boom of the 1990s and 2000s. Watching the same transformation taking place in mobile, he writes about what we can learn this time around to make this new phase in digital growth usher in smoother to benefit everyone involved. Reach him on Linkedin

July 7, 2015
Top 5 Anti-ESB Arguments for DevOps Teams.

Unix and Mac. Developers are always in combat. My last verbal combat was a “discussions” around ESB and Microservices. They are fun because both supporters tend to have very strong opinions about what they think is best and both want to fight each other. I recently got involved in one of those discussion about ESB vs. Microservices and the ESB guy throw lots of arguments at me. “ESB is magic” got almost burnt into my mind until I started to investigate a bit: “How to fight ESB discussions in combat?”. Here are my top 5 findings that may help in your next verbal ESB combat situation.

(more…)

July 6, 2015
Three Ways Dev and Ops Teams Can Manage the Integration Lifecycle
Many enterprise organizations did not invest enough money into integration as they built their cloud or hybrid infrastructures. As if integrating on-premise systems and applications wasn’t tough enough, in today’s cloud era with the rise of hybrid environments, connectivity has only gotten more complicated – especially when combined with the increased speed of deployment. Puppet Labs found that high-performing organizations deploy code 30 times more frequently than their lower-performing counterparts, meaning dev and ops teams are struggling to break down data silos while simultaneously managing constant technology upgrades for line-of-business users.

There’s no doubt that, whether public or private or hybrid, the cloud is now part of most companies’ infrastructures, and its share will only increase. Goldman Sachs estimates that spending on cloud computing infrastructure and platforms is expected to grow at a 30 percent CAGR from 2013 through 2018 compared with 5 percent growth for overall enterprise IT, while Forrester predicts that 25 percent of all software applications purchased will be cloud-based by 2020.

With the speed and agility the cloud imposes on businesses, dev and ops need to come together to look at data integration throughout its entire lifecycle, from developing new apps or services on top of existing apps, all the way to the ongoing service, maintenance and upgrades of those applications. The two teams must join forces to manage this process as one comprehensive integration lifecycle, where dev and ops play equal roles solving for new data integration challenges brought about by hybrid deployments and siloes created by line of business users’ cloud purchasing habits.

Today’s dev and ops teams have to look at data integration as a service, not just a one-off implementation they can hand off. Here are three strategies to help bring the two teams together to jointly manage the data integration lifecycle.
1. Assess Current Integration Challenges Collaboratively
  Without alignment between dev and ops teams, integration lifecycle can’t be managed successfully. Organizations need to align on the people and process portion first, asking questions like: what systems does each business unit need data from? How likely is it that their needs will change, and how can you plan for a flexible integration as a service? What capabilities will ops need after dev implements the integration?
After reaching agreement, teams will need to jointly assess and select a data integration as a service technology or platform that allows for joint requirement setting, collaboration, testing and ongoing adjustments throughout the lifecycle of the applications. Selecting the right integration platform as a service at the onset is critical to minimize cost, time and wasted dev and ops cycles down the line as business and technology needs change. The most agile integration option typically consists of a plug-and-play, flexible platform that makes use of APIs and also allows for deeper customization down the road.

2.Understand the Four Key Stages of Integration Lifecycle Management
Dev and ops teams are now part of the same cycle dealing with frequent updates and agile product development. It’s no longer about developing, testing and then having ops maintaining, but about a seamless and transparent handoff from dev, to ops and back throughout the entire application and data integration lifecycle.

To capture the most value from disparate organizational data and to minimize ongoing data and application synchronization costs, dev and ops teams need to adopt a joint integration management strategy that accounts for the four distinct phases of data and application connectivity. These include:

3. Design phase. After understanding business use cases and selecting a platform, dev and ops must collaborate on priorities – what near-term functionality can they design for based on the plug-and-play layout, and how can they tackle that low-hanging fruit while maintaining the flexibility to meet future requirements?

By starting with the right technology and a common integration framework, dev and ops teams can design connected core systems and data early on with plug-and-play API connectivity for solutions like CRM and ERP, while maintaining the ability to easily expand functionality with custom/deeper integration capabilities with marketing automation, business intelligence and other apps.

In addition to functional capabilities, the design phase should address performance and scalability by taking into account throughput and latency. This process involves planning to meet performance expectations, even with the customization and extensions users will likely want down the road. Dev and ops should ask themselves questions like: which company systems will be on premise, in the cloud or hybrid? What are the performance, scalability and security tradeoffs of each approach?
1. Develop phase. Dev and ops need to work together in the development phase to carefully document key aspects of the integration while building connectivity for specific use cases. This documentation will be critical to the seamless handoff between the two departments, so they both need to be involved in the process. Strategic collaboration and thorough documentation can improve the efficacy of maintenance and debugging activities, help companies adjust to staffing changes and more easily transfer integrations to and from the two teams during the ongoing development and roll-out process.
2. Deploy phase. Given that you’ll likely deploy in a hybrid environment, make sure you test both on-premise and cloud integrations to ensure that data is flowing freely. To help test integrations and move them into a production environment, look into staging and production environment tools that streamline the process. Infrastructure costs should also be considered, as they may increase unexpectedly if you don’t watch them throughout the deploy phase.
3. Run phase. Continuously monitor to confirm that the data is flowing as designed. If dev and ops selected the right technology platform, the teams won’t have to worry about ongoing API management, because the platform takes care of it.
The right platform will also lower downtime and maintenance costs with a multi-tenant management environment that allows the expert for any given situation to dive into the integration backend or runtime. The platform should also provide ongoing diagnostic reports and proactive alerting to allow ops to home in on potential performance issues.
1. Build a Responsive IT model
  With dev and ops coming together to take advantage of the speed and agility offered by the cloud, the resulting just-in-time services for businesses are turning IT from a cost center into an innovation center. The data integration lifecycle is another result of the new cloud-shaped world, helping dev and ops teams create a more responsive IT model and allowing them to better meet business needs while driving innovation. While shared environments bring with them an additional element of risk, a responsive IT model is an important step towards both short-term and long-term success in the modern connected enterprise.
These three steps allow dev and ops teams to offer the value, speed, efficiency and opportunity that today’s enterprises require. By adding collaboration and agility at every stage of the integration process to plan, implement, track, debug and update continually, dev and ops teams can scale to respond to the data-driven needs of the business more precisely.

How have you overcome cloud and hybrid application and data connectivity challenges in your organization? Connect with me in the comments below, or on Twitter at @scribesoft.

About the Author/John Joseph

John Joseph, VP of marketing for Scribe Software, earned a master’s degree in business administration from the MIT Sloan School of Management, a master’s degree in electrical engineering from the University of Southern California and a bachelor’s degree in electrical engineering from MIT.
June 23, 2015
DevOps Leadership Series: Gov Does DevOps

This past week, I had the opportunity to catch up with some more industry thought leaders at the DevOpsDays DC event in our nation’s capital. This was the first major DevOps Days event to feature a large audience of government participants. It was an awesome event and is certainly going to be on my must-attend list for next year.

First off for the series, I had the chance to chat with Nathen Harvey, the Community Director at Chef. Nathen also did a great job leading the organizing committee for DevOps Days DC.

In this episode of the DevOps Leadership Series, Nathen illustrates some recurrent topics he noticed at DevOpsDays DC. Nathen ensures us that government is ready for DevOps, enterprises are ready for DevOps, and small businesses and web innovators are ready for DevOps. Then he highlights how we need to create high velocity organizations that are safe, scalable, and humane for our teams.

https://youtu.be/hdUl8-d1za4

Next, I had the pleasure of catching up with Greg Elin, the Executive Director at GovReady and former Chief Data Officer at the FCC. What really stood out to Greg at the event was how many people already have their DevOps “sea legs.” He notes the number of different organizations that are comfortable with DevOps, and are familiar with its language and concepts.

https://youtu.be/XoZ4PGIAZx4

Finally, I caught up with Dave Johnson, Senior Technical Director at Ansible. Dave shares that “the market really is not dominated yet by any particular player in this (DevOps) space.” Dave explains when he first entered the industry he thought the IT world was dominated by open source and commercial technologies. However, when talking to the industry at large now, he found most the enterprises aren’t yet doing anything yet for automation, and most enterprises are still tackling their problems with humans.

https://youtu.be/ky51j1-fNsU

Next up in the series, you’ll hear from John Willis (Co-Author of the upcoming DevOps Cookbook) and Leon Fayer (Vice President at OmniTI).

If you have missed any of the other videos from this series, you can find them here. (We’re up to 15 so far).

June 22, 2015
DevOps Leadership Series: Gov Does DevOps (Part 2)

During my second day at DevOpsDays DC, I had the opportunity to catch up with a couple more industry thought leaders. First up, John Willis, (@botchagalupe) DevOps Days core organizer and co-author the upcoming “DevOps Cookbook”.

In this episode of DevOps Leadership Series, John highlights his first introduction to DevOps and the journey it has taken him on, and what about the current state of DevOps makes him excited. He says DevOps is really happening everywhere. Today large organizations and CIO’s of government agencies are now experimenting with DevOps at all stages. He references legacy companies that do not necessarily have “aggressive IT”, and how they now want to share their DevOps experiences.

https://www.youtube.com/watch?v=jU1O2bCU_hc

After that, I had the chance to chat with Leon Fayer, (@papa_fire) Vice President at OmniTI, following his DevOps Days DC presentation. Leon illustrates the things he found most interesting from the DevOpsDays Conference. He noted that a lot of people have started to pay attention to the monetary strategies, and how to monitor from top-to-bottom in a DevOps fashion. Leon was happy to see that people are now more generally interested in how to generate and collect data more accurately to solve their problems.

https://www.youtube.com/watch?v=kx4Y3P3Njro

June 17, 2015
Continuous Delivery and The Proof of Quality Assurance

Quality Assurance is critical in Continuous Delivery. Every change we commit to the common source repository is a release candidate. That’s why we test our code before we release it. We find out whether objectives has been met or not. Unfortunately, testing is a discipline that many try to avoid and it’s one of the main reasons for having specialized testers in the teams who know how to break the code best. But shouldn’t be the developers of the code know best how to break it?

(more…)

June 16, 2015
2015 State of the Software Supply Chain Report
In April of this year, I embarked on a six-week journey diving deep into an analysis of the world’s software supply chains. I evaluated the practices of 106,000 organizations, the 100,000+ suppliers they relied on, and the billions of software components that fueled their agile, continuous delivery and DevOps practices.

The facts I discovered and share in the 2015 State of the Software Supply Chain Report: Hidden Speed Bumps on the Road to Continuous, (pre-register for the full release: Tuesday, June 16th) fundamentally changed the way I thought about software (and about DevOps).

The volume and velocity of consumption, the variety of parts and suppliers, and the impact on innovation and quality astounded me. Early reviewers of the report including Gene Kim (co-author of the Phoenix project), Gareth Rushgrove (Puppet Labs and DevOps Weekly newsletter), Nick Galbreath (Signal Sciences), and Nigel Simpson (Fortune 100 Entertainment and Media company) agreed.

My aim for this research is not simply to present facts about the global software ecosystem. I’m aiming to point a spotlight on software supply chain best practices within across a variety of industries that could be used as new benchmarks for software supply chain automation. Similar to manufacturing of auto, pharmaceutical, healthcare, or defense systems, the effective management of supply chains will create winners and losers. I’ll share evidence that the best, high-performance software development organizations are benefiting from:
- Working with fewer and better suppliers
- Relying on the highest quality supplies from those suppliers
- Maintaining traceability and visibility throughout the software supply chain for prompt and agile recall
Key Points from the Study

In the best organizations, the research revealed developer net productivity increasing by up to 40%. Just imagine applying that time to more innovation, rather than to rework and maintenance efforts.

At the same time, the report touches on inefficiencies and complexities that are creating a huge drag on the velocity software development teams are aiming to achieve. A lack of discipline, focus, and visibility around the software supply chain has resulted in mountains of technical debt, unnecessarily context switching, and outdated sourcing methods that wasted over 3.3 million build days last year alone.

Hightlight: Automation Across the Software Supply Chain

The other key insight from this research is the clear need for further automation across software supply chains. With individual organizations consuming hundreds of thousands and sometimes millions of software components annually, it became obvious that waterfall-centric approaches to identifying the most functional parts, checking quality, validating appropriate licenses, or evaluating security vulnerabilities could not keep pace. Sourcing practices that regularly go unchecked have also resulted in the use of severely outdated software components, even numerous versions of the same component part.

Download the Full Report

Over the next few weeks, I will publish excerpts of findings and best practices identified in the 2015 State of the Software Supply Chain Report and invite you to read along. Or you can access the full report now through this link.
June 12, 2015