devopsy.kinsta.cloud

Category: Features

Deliver faster, but don’t sacrifice quality

Developing applications, and deploying servers faster are great goals. Being agile and streamlined can give an organization a clear competitive advantage. However, faster does not equal better, and being speedy for the sake of being speedy can lead to avoidable mistakes. Faster is only better if you can achieve greater speed without sacrificing quality.

At the same time companies are racing to be faster than their rivals, they’re also dealing with an increasingly complex IT infrastructure. IT environments are becoming highly heterogeneous—merging physical, virtual, and cloud platforms, and combining legacy systems with cutting edge technologies. The challenges of managing such a diverse IT infrastructure simply add to the potential issues that can arise as organizations try to work faster.

That is where DevOps plays an important role. DevOps tools and technologies establish a process chain between development and deployment. It enables organizations and IT admins to reliably track and understand the components in use, and any changes or updates that occur. In order to achieve speed without sacrificing quality, it’s important to maintain as consistent an environment as possible, and to know exactly what has been changed.

“For hypervisor based virtual images, this depends on the strict process around handling the images, but for container based virtual images depending on a cascading snapshot mount namespace, where there are layers of images built one on top of the other, strict handling becomes unnecessary because everyone can see transparently (via the image cascade) what changes at each stage of the process,” explained James Bottomley, CTO of server virtualization for Parallels. “This gives everyone in the DevOps chain the ability to verify each step and see the changes for themselves, leading to much higher assurance that the resulting deployment is correctly tested and released.”

Ken Cheney, VP of business development for Chef, describes another way that DevOps helps boost speed without affecting quality. “Automation turns infrastructure into code. For example, Chef recipes automate all facets of management and orchestration—from provisioning bare-metal and cloud servers to automated configuration and application deployment. You can replace manual, slow, error-prone procedures with code that is versionable, testable and repeatable.”

“When infrastructure is code you can test it just as you would applications—using automated tests that find problems early, before you release to production,” added Cheney.

In its 2014 State of DevOps Report, Puppet Labs drew an interesting correlation between speed and quality. Essentially, Puppet Labs found that speed and quality become a self-feeding circle.

“There’s a virtuous circle at work here: As stability improves, IT performance improves. This improved performance helps to create a better-functioning business that can pay attention to the communications and processes that enhance and improve stability.”

The converse is also true. Poor quality results in issues, and necessitates allocating resources to troubleshoot and resolve problems, which impedes the speed at which an organization can function.

DevOps tools and technologies enable organizations to streamline development, and work more efficiently without sacrificing quality. Businesses that attempt to work faster, without also working smarter by using DevOps are putting themselves at a disadvantage.

January 7, 2015
DevOps Transforms Copyright Clearance Center

Copyright Clearance Center (CCC, www.copyright.com) works primarily with business, academic, and medical institutions, enabling them to access the vital content that supports their work. CCC manages fee payments to the license holders and content creators who own the content. DevOps approaches have transformed CCC, a global leader in content rights-licensing solutions, resulting in expanded service to an increasing market.

“In production, CCC is already processing millions of dollars in transactions. This came to life within a year of the inception date for the DevOps project. For a startup that would be fast. For a medium-sized enterprise, I think that is outstanding,” says Babis Marmanis, CTO & VP of Engineering, Copyright Clearance Center.

The DevOps transformation that took place in the internal structure of CCC’s engineering department continues today, allowing the organization to respond to the massive disruption in the publishing industry, which is due in part to digital content and the Internet. Using DevOps, Marmanis and CCC have accelerated product development, sped time to market, and provided a stronger consumer experience with the product.

Accelerated Product Development

“The Copyright Clearance Center vision is to create the most comprehensive portfolio of global licensing and content solutions (products) that make copyright work for everyone. My product platform strategy stems from that,” says Marmanis.

CCC has multiple software-based products, each performing the same or similar functions and acting on the same or similar data. “There are certain fundamental things that we must abstract appropriately and encapsulate into common components to make development work optimally,” says Marmanis.

But when Marmanis came to CCC, there were no common components. And what is more the internal systems lacked interoperability, skills were different across internal teams, and there was no cohesiveness to the development approach. These factors slowed software product development.

A DevOps mindset was crucial to changing the status quo, to treating infrastructure as code, and to bringing in automation, says Marmanis. A DevOps approach would help to speed the delivery of software projects into the marketplace.

“Now we use Jenkins (www.cloudbees.com) to glue together elements of the continuous integration process. We have Jenkins build jobs that retrieve the source code from GitHub (https://github.com). We publish software artifacts to Artifactory (www.jfrog.com),” says Marmanis.

For each release, Marmanis’s teams use Jenkins to execute Chef (www.chef.io) recipes. They deploy artifacts from Artifactory onto a pre-specified execution environment. “In addition, and this is key to the overall scheme, we execute the schema updates with Liquibase (www.liquibase.org), and everything is coordinated with Jenkins,” says Marmanis.

Further, CCC built its cornerstone platform from the ground up using DevOps-related tools, build detection software, and software libraries. The infrastructure that CCC added is based on best-of-breed open source libraries.

“Using programming languages such as Java (www.java.com) and Groovy (http://groovy.codehaus.org), Gradle (www.gradle.org) scripts, postgreSQL as the relational database, Liquibase for database objects, Tomcat (http://tomcat.apache.org) as a servlet container, Apache Camel (http://camel.apache.org) for messaging and enterprise integration, and countless other tools including Apache Directory Studio (http://directory.apache.org/studio) and Bootstrap (http://getbootstrap.com/)—the list is very long—we were able to create a harmonious ecosystem,” says Marmanis.

Stronger Consumer Product Experience

CCC offers its consumers a better product experience through DevOps approaches and through a homogeneity of standards and technologies, which is available when using open source tools.

“When we’re building the product platform, these initiatives allow us to embody timeless design principles such as crisp abstractions, but also allow us to scale in a number of dimensions including the number of works that we can represent, the number of publishers we can admit and manage, the number of customers that we have, the number of applications that we can build and deploy, and the number of market verticals that we can occupy,” says Marmanis. This is crucial to customer satisfaction.

One product that is a prime example of how CCC keeps customers happy is RightsLink for Open Access. “Using RightsLink for Open Access, we facilitate fee collection for articles published under an Open Access license, mediating between the publisher and the author to collect those fees,” says Marmanis. Then CCC aggregates those transactions and presents them to the publisher.

“Obviously, we give the money to the publisher, but we also create reporting for the authors, publishers, and institutions as well as the founding agencies that gave the money to the author or the institution respectively for funding that research,” says Marmanis. The RightsLink for Open Access product keeps transactions simple, integrating with publisher workflows to expedite publication.

January 6, 2015
Equalizing IT infrastructure with DevOps

IT is getting more complex. Managing on-premise servers, cloud servers, and hybrid environments that span various operating systems and standards creates challenges for IT admins. DevOps technologies and platforms, however, can be an equalizing force among the cloud chaos—making it easier for organizations to deploy servers and applications quickly and consistently regardless of platform or environment.

Once upon a time most organizations were fairly standardized. Servers were located on-premise in some sort of closet or data center, and IT admins knew exactly what operating system they were working with. Now, those servers may be physical or virtual, on-premise or in the cloud, Windows or Linux, Azure or Amazon, and in many cases IT admins are dealing with an infrastructure that amounts to an all-of-the-above approach.

As the name implies, the core of the DevOps movement is to remove the barriers between developers and operations. Many of the tools and technologies that comprise the DevOps movement, however, also have the added benefit of providing a consistent, modular approach to developing and deploying applications and infrastructure. Those technologies also reduce friction between on-premise and cloud environments, and allow organizations to streamline operations across different platforms and operating systems.

“One of the biggest reasons for the increased interest in DevOps tools and practices is because businesses have to move and iterate faster today than at any other time in history,” declared Tim Zonca, director of product marketing for Puppet Labs. “So IT teams are pursuing DevOps flows, where development and operations practices, systems, and tools work in lockstep.”

As IT infrastructures become more complex, DevOps becomes more crucial. Expansion of IT across physical, virtual, and cloud environments make even minor inconsistencies into critical issues as organizations grow, and brings additional burden to address those problems.

Many DevOps technologies revolve around using virtualization to deploy applications inside pre-built environments, or modular blocks that can be deployed and re-used across various platforms and environments. These tools remove most of the chaos, and enable organizations to simplify deployment across virtually any infrastructure.

Docker is a great example of the impact DevOps is having. Docker containers enable developers to package an application, and transport it anywhere Docker containers are supported without modification to the virtual environment. As Docker continues to spread influence across Azure, Parallels, and other cloud and virtual platforms, it is asserting itself as an equalizing force that eliminates the need to even care what the underlying infrastructure or operating systems are.

Puppet Labs is another illustration of the equalizing force of DevOps. Zonca explained, “Puppet makes it easy to standardize and add consistency to your infrastructure so you can make fast, repeatable changes across physical and virtual machines, on premises or in the cloud. Because these tools create a predictable environment, you don’t have to worry about a simple update taking down an entire system your business relies on.”

James Bottomley, CTO of server virtualization for Parallels, suggests that even DevOps has its limitations, though, and that it could eventually get in the way of advancing productivity. “However, one cannot help thinking that DevOps, while a leveler, is also a separator because it is only accessible to developers with the Ops skill, not to all developers. Additionally, the utility of DevOps depends upon the availability of Infrastructure and, while IaaS is clearly the dominant Cloud technology today; it won’t be forever. At that point, one might imagine that the DevOps technique might impede rather than assist the next paradigm shift in the cloud.

For now at least, DevOps technologies enable developers and IT admins to take a modular, platform-agnostic approach to many aspects of applications and infrastructure to simplify and streamline regardless of the underlying environments or operating systems.

January 5, 2015
Top 5 DevOps and Operations Challenges in 2015

DevOps and operations teams are under increasing pressure from tech-savvy, app-centric business users to collaboratively solve complex business problems with IT. Adding to the battle, the pursuit for the perfect synchrony between software development and IT operations is still ongoing, and striking the balance won’t happen any time soon. New tools, technologies and processes change and grow at a problematic pace – leaving us with no option, but to accept and address all upcoming challenges.

Teams must keep up the pace to succeed, but operations teams cannot handle it solo and must work closely with DevOps. The operations group is responsible to drive flawless, organization-wide execution – this was a straightforward request when the systems were configured and maintained by operations. However, recently, development became more collaborative, with involved users demanding more, using the newest tools and making the process harder to manage. Additionally, the product environment of business users has rapidly moved to portable devices, which are self-contained and hungry for enterprise-wide cloud services. This shift has created five principle challenges. These are my top 5 DevOps and Operations challenges for 2015:

Challenge 1: Building the DevOps Culture

Culture is an abstract group of ideas and behaviors, which makes it hard to identify a standard collection of requirements. It is much easier to decide which tools to use and how to use them in the most efficient way. Tools are material objects, but the people who use these tools are much more complex and difficult to change. As DevOps is all about continuous change – including reconfiguring a company’s core culture – there is a steep hill to climb.

Replacing traditional development and operations silos with cooperative teams of people from both camps can only get us so far. While DevOps is supposed to intentionally foster collaboration and results-driven effort, it needs the support of operations. Until operations can identify how to embrace the new mentality of culture-fostering activities into its daily regimen, there is no way to help this aspect of DevOps grow.

Challenge 2: Meeting the Quick-Fix Tool/App Mentality

While most end users now rely on a wide palette of tools to get their job done, they often circumvent the IT department to get the tools they prefer. As many are open source or offer initial trial versions, unsanctioned tools can be adopted without any internal oversight – leading to major issues down the road.

Making matters worse, business users believe everything can be solved with an app, but that is often untrue. While it is the overwhelming consensus that every tool should have an easy-to-use app, making the app a useful, secure one is much more complex. Even if there is an “app for that,” it must be properly integrated to back-end systems as part of the cohesive enterprise infrastructure. DevOps must balance tool delegation and management early on.

Challenge 3: From Legacy to Cloud

Technology-savvy business users may want to use the latest tools, but that is a tall order for operations. When it comes to DevOps tooling and configuration management, there is still a lot of confusion as to which tools do what, and how they play with each other – if it all. Tools need to be compatible with existing solutions and technologies used by the company, and that is becoming increasingly complex with the rise of hybrid environments.

New hybrid environments include on-premise tools, IaaS tools like Amazon, Google or Azure, PaaS compute services, and SaaS business applications. Somehow, DevOps must ensure that they all work well together and deliver a seamless experience for the business user.

Challenge 4: Networking

Network engineers – if they are applying automation at all – are either doing ad-hoc scripting or tap a dedicated software engineering team to build homegrown systems for their use. Inflexible networks run rampant in today’s IT environments.

Looking for an example? Even vLANs have a spider web of interdependent IPs, ports, routers, routing tables and perimeter networks. Changing, breaking, moving or replicating any part of that intricate web often seems impossible.

Challenge 5: Marginalized and Changing Budget

Taking another perspective: Will customers pay more for network automation tools from the equipment vendor to make the networking user experience better? If a networking customer is spending a lot of money on the physical equipment, they often do not want additional software tools from the vendor, even if the network engineers do. The gap between the network engineering teams that need the tools and the decision makers who pay for them is widening.

While capital expenses may be separate from operational ones, CAPEX spends are now often mirrored by OPEX ones. Trying to quantify spend on an on-premise project management tool with a SaaS application is a major headache that can no longer be left solely in the hands of the CFO. This shift impacts the structure of IT budgets – it could even take away IT’s control if OPEX spend becomes more distributed.

What can be done to overcome these growing problems? Taking the following five steps should help you start off 2015 in the right direction.

Collaborate Wisely:

It might seem to get a little tense at times, but open forum discussion is well worth it. Collaboration allows teams to learn from each other and suggest possible solutions. It also encourages critical thinking and creativity. When you have different people working together on a project, you get more expansive creative input. To manage the potential leadership glut (i.e., the excess of people trying to lead the group and not that many willing to slow down and focus on a specific task), break the traditional decision-making process chain and turn it into a continuous flow of fresh thinking and ideas.

Empower Users:

Let your business users pick their own tools, but make sure that they report them to you. It’s a good idea to consider a library of vetted tools available for them to pick from. Encourage business users to suggest new tools so they have a sense of decision power. However, keep the tools current and make sure they actually address user needs.

Speak Accounting:

Try not to let OPEX issues become obstacles in tool adoption. If IT can help finance understand what is going on in the world of technology licensing vs. cloud service, it can fast track the inevitable shift, and ease budget reporting and bottlenecks.

Break the Immutable Network Infrastructure:

With immutable network infrastructure, you run the risk of being locked into vLAN and physical devices. What is the solution then? Leverage advanced virtualization tools (i.e. converting legacy systems to virtual machines) and take Software-Defined Networking (SDN) on board. With a centralized, programmable network that can automatically and dynamically address changing requirements, you can reduce both CAPEX and OPEX by enabling algorithm control of the network and use ASIC-based networking hardware supporting pay-as-you-go models. Deliver agility and flexibility by helping organizations rapidly deploy new applications.

Prioritize Analytics:

Analytics should make IT’s life easier. Build analytics and monitoring into everything from the start and think about it as “making sense” of the data or logs being collected. This information helps IT determine the true business impact of a performance issue to better prioritize where and what to troubleshoot. A strong analytics platform, where all application and tools data can be stored, gives operations the systems visibility it requires. Become a facilitator while maintaining all needed operations practices.

There will always be challenges ahead as we work to get smarter/ faster and develop better systems and applications. However, the right analytics tools and collaborative processes designed for cloud-enabled, distributed environments can make 2015 much easier than years past.

About the Author

Trevor Parsons, as chief scientist and co-founder at Logentries, Trevor Parsons is responsible for product strategy and direction. He works closely with customers and partners to continuously understand what they need, and to validate product market fit. Trevor also leads the product management and UX teams and assures the best possible user experience. Trevor enjoys speaking at local dev ops meet-ups and events, and is always looking for how log data and analytics can be applied in more and more powerful use cases. Trevor was a post doctoral researcher and member of the Performance Engineering Lab at the School of Computer Science and Informatics in University College Dublin, Ireland. He received a PhD from University College Dublin for his thesis titled “Automatic Detection of Performance Design and Deployment Antipatterns in Component Based Enterprise Systems.” You can follow Trevor on Linkedin and Twitter at https://www.linkedin.com/in/trevparsons and https://twitter.com/trevparsons.

December 19, 2014
How DevOps Eases The Holiday Shopping Experience

“I have been building retail software for 15-years,” says Harper Reed, CTO, Modest, CTO emeritus, Threadless, and former CTO, Obama for America. “With the advent of DevOps, development is now about how we can make the online experience better for the customer,” says Reed. It used to be about taking care of the systems.

Today, the infrastructure is in software with Amazon and the Cloud, with what DigitalOcean is doing with Docker, and with what CoreOS is doing with server deployments. Before cloud-based infrastructure, the enterprise used the data center, which was a fortress that operations had to protect. It wasn’t as easy to think about the customer with those kinds of concerns to look out for. Now everyone has time to think about the customer.

“With the cloud, you can start to have the same lifecycle with infrastructure that you had with the software that actually ran your ecommerce,” says Reed. Teams can leverage a uniformity of tools for measuring performance. Because everyone is using the same tools, it becomes easier, more uniform, to test the software. “We can all be thinking about the user,” says Reed.

With DevOps, infrastructure in software is making the online Holiday shopping experience easier by scaling up in the cloud using very simple scaling rules. “In 2007, 2008, I was CTO of Threadless and we had very good software, but we had to plan months ahead in order to scale things up during the Holidays,” says Reed; “today we don’t have to think, what is our capacity going to be, how am I going to add 15 more servers? We just turn up the budget on Amazon and away it goes.” Of course, scaling keeps shoppers from experiencing server time-outs and slothful transactions.

Because scaling is already addressed, teams can focus on development efforts on the front end, on making the user experience better. The time teams would have spent scaling the back end they now spend ridding bugs and adding and improving services and features, and doing it all much more quickly thanks to continuous deployment.

As a result of a stronger commitment on the front end, efforts become possible such as you see today where stores like Everlane , an online clothing retailer has partnered with Postmates, an on-demand delivery service to drastically cut order delivery times, including same day deliveries http://blog.postmates.com/post/104856354257/powering-on-demand-logistics using Postmates API.

“I visited a pretty well-known ecommerce startup in the Valley a couple of weeks ago. I can’t name the company, but their technology team had zero ops people because they are able to manage all of the ops that they need through software,” says Reed. This retailer hires developers who handle the ops and they don’t have to split their team up, which means more people are building better features.

They’re not spending their time managing hardware sitting in a data center somewhere. They don’t have to manage their bandwidth consumption as much. “They can do it through a control panel in software rather than by making calls to a data center,” says Reed.

And the faster the ecommerce site is, the better the conversions will be. “If your site is very quick, which goes hand in hand with scaling and how you handle capacity, the better the experience will be for everyone from the consumer who is purchasing the products to the retailer selling them to the shipper shipping them out,” says Reed. This all adds up to increased customer satisfaction.

Through continuous integration / deployment, DevOps is able put customer feedback into action very quickly, creating and testing fixes to customer experience issues, which teams can then immediately deploy. “Since we started using DevOps concepts in development, the product development cycle has decreased the time it takes to get things done and into the marketplace,” says Reed.

Before if you were a user and you reported a bug, the enterprise might well have said, “well, okay, we’ll get to it after the Holiday season because we don’t want to change anything at this critical time for fear of introducing more issues”. “Now,” says reed, “we’re able to do some testing and make sure it works and that it’s the right path for us. We’re able to get there very quickly and then we can make a more educated decision on whether now is the right time to make the change.”

Further, the online retailer has the opportunity to consider whether making the change immediately might save a significant amount of sales that would have been lost.

December 18, 2014
Clearing through the app backlog clutter

Development and operations teams today are under more pressure than ever to develop apps the business needs to make its workforce productive, its partners connected to the resources they need, and to deliver the apps to the marketplace that will make it competitive.

The number of production hours for development is at a precious premium as demand for apps soar and resources remain tight. According to Gartner, the demand for enterprise mobile apps alone will be a $40 billion market by 2016.

Of course, the inevitable happens and the list of designed applications gets backlogged. They get caught-up in budget requests that drag on, and then they get caught in the long list of requested apps in the development queue. A recent survey commissioned by PaaS provider OutSystems found that 85% of enterprises surveyed had an app backlog, with about half having backlogs of 10 to 20 apps.

App Backlog, agile and DevOps

Sure, development and operations teams move as quickly as they can to meet the enterprise demand for new applications, but when there is any deep app backlog, there is the potential for business costs when it comes to lost opportunity, untimely product development, and future missed sales. In a time when agile IT means competitive advantage, any amount of latency that can be cleared from the application backlog is crucial.

Agile development approaches and agile IT departments tried to remedy some of these issues, but didn’t go far enough. Fortunately, DevOps picks up where agile left off in the evolution of enterprise development. And in the face of agile IT, continuous integration, continuous deployment, and the rise of DevOps – operations, development, and infrastructure bottlenecks become minimized – and infrastructure and applications move more quickly, and existing applications are iterated more swiftly.

Of course, DevOps itself, can help to accelerate app deployment, in addition to continuous integration and continuous deployment efforts. And all of the associated deployment automation tools and deployment processes are essential to eliminating manual waste so teams can focus on pressing business demands. This waste can include time burnt on manual process, manual testing, and the configuration of the software development environment.

DevOps-driven organizations have shorter app backlogs. A recent CA Technologies study in fact found that organizations embracing DevOps report a 17 to 23 percent improvement in certain business outcomes, such as increased revenue, and faster time-to-market.

Cloud architectures and easier to master development languages also help, not just infrastructure services, but also platforms that help to free up operations teams and developers to focus on the applications that need to get out yesterday.

Even with DevOps practices in place and automation running full throttle, the business development teams still find themselves having to make difficult choices about which apps to build and deploy. Not all of these apps will survive, in fact, many apps will go unused and dormant. This means that a lot of time can be lost developing apps people don’t want to use, or are too cumbersome to bother to use.

Turning to the Minimum Viable Product

In addition to DevOps, development teams are also turning to approaches such as the Minimum Viable Product (MVP) as a way to help identify apps that have real user demand and those that do not. Essentially, the MVP app is an app that consists of only the essential. The MVP provides the essence of the app and what users need to get done and no more. The MVP is a potent way to provide the essential attributes of an app to see if users embrace it before building the app out further. If the app is used, new features can be added over time.

Not everyone is convinced that the gains made against app backlog aren’t necessarily illusion. David Mortman, distinguished engineer, says “DevOps and MVP both have the ability to create the impression that you are solving backlog issues. This is because in both cases you are enabling the development organization to get that first release as early as possible and then to keep iterating it and adding features as the dev group goes,” he says. “This however may not get you to a particular defined endpoint any faster since regardless of the organization there are always resource limitations.”

Still, “One advantage of DevOps and Agile is that they let you interweave development cycles so multiple apps can be worked on faster and each can be incremented more quickly. They also make it easier to adjust priorities between apps more quickly and more quickly identify apps that aren’t meeting the business needs and either deprioritize or eliminate the app before overinvesting in its development,” Mortman adds.

There’s no doubt, based on my discussions with enterprises, app backlog repeatedly comes up as a serious challenge to competitiveness. But by embracing continuous integration, deployment, and MVP development, organizations can go a long way to help clear through, at least the initial stages of their app backlog. What is your organization doing to help clear app backlog? We’d like to hear your experiences and lessons learned.

December 16, 2014
Four strategies for supporting DevOps collaboration

As we’ve discussed before, the ever elusive DevOps culture doesn’t form magically. Rather, it is fostered by building a collaborative environment. While there are many ways to skin the cat in terms of what kind of tools organizations use to start collaborating more effectively, the tool set and the process they enable should revolve around some key principles. The following strategies can help guide organizations to choose the right infrastructure to foster a transformative DevOps culture and support DevOps collaboration.

Centralize And Unlock Knowledge

One big mistake some organizations make as they embrace DevOps is they consider their automation tools as knowledge repositories. But things like recipes, manifests and YAML files are great for automation, not collaboration, explains Alan Sharp-Paul, CEO of ScriptRock.

“They are an awful way of storing desired state,” he says. “Desired state should be visible. It should be easy to understand and simple to modify by any stakeholder. If the systems that define your desired state are not open and accessible then collaboration will suffer. When knowledge is locked up, collaboration breaks down.”

Sure, there may not be a super platform for everything, but centralize documentation with pointers out to the right details, says Chris Clarke, senior vice president of product management for CollabNet, who suggests that this place should show how work is spawned and what is being asked of whom, where work is being outputted, where to find all of the tools being used across the development and deployment lifecycle and visualizations, diagrams and step-by-step details about important processes.

“It’s almost like a mall map. ‘You are here,’” Clarke says.

Enable Timely Communication

Of course, knowledge doesn’t stand still and there are always day-to-day snags that come up, so constant communication is fundamental to the collaborative process. Many DevOps and Agile leaders find the success they see in speeding up iterations stems around the experimentation they’ve done in developing a communication channel that work for their teams.

This usually starts with using a chat platform like HipChat, FlowDock, or Slack, and frequently continues by pairing that with task managers like Trello, Asana or Basecamp. Many organizations are also integrating IT-specific tools like Git, PagerDuty and Raygun into this mix.

And, increasingly, DevOps teams have been up-leveling that by implementing GitHub’s open source side project, Hubot, a chat room automation bot that helps teams share scripts, automate tasks and enable simple deployment tasks and keep everyone else apprised of deployment.

The sky is the limit for creative combinations—the key is that the overall stack facilitate timely and meaningful communication across the team.

Find Ways To Visualize Technical And Process-Based Interdependence

So, what about SharePoint to accomplish these goals? The problem with SharePoint, says Matt Selheimer, chief technical evangelist for ITinvolve, is that it really isn’t designed around the workflow and the dependencies of IT-specific work. As he explains, most organizations never get beyond the stage of using SharePoint as a glorified file share.

“That’s not going to help you understand risks and relationships and dependencies and nobody has time to go and read through all those documents and try and parse them anyway,” he says, explaining that a collaboration platform should be able to help various employees track activity around applications, policies, servers and what have you that they have a stake in. When someone else is touching those specific objects, they should be alerted and connected with the resources necessary to look into potential issues, resolve any problems that arise and potentially suggestion improvements on current progress.

Componentize Knowledge

Centralization is important, but that doesn’t mean that knowledge should necessarily be consolidated.

“You need to componentized it correctly—in other words, you don’t try to write a 40-page document, you use a wiki or something that should change pretty frequently,” Clarke says. “I’m a huge fan of continuous improvement in everything, including this.”

Selheimer agrees.

“We don’t think that trying to capture everything in one giant, massive document and have it be reviewed by a committee is effective because it’s never going to get published It’s going to be outdated,” he says. “Whenever I ask people about what diagrams do they have to use to manage their environment, they always say, ‘Oh, we got Vizios.’ But they’re all six months out of date, right?”

December 15, 2014
Organic engineering team a must for long term DevOps success

Last month I was out in Las Vegas at CA World. While there I had a chance for a one-on-one meeting with CA CEO, Michael Gregoire. In our discussion Gregoire told me that one of the things he is most proud of in his time at CA is the fact that he has instilled a culture of an “organic engineering team”. According to Michael you can’t truly be a successful software company (and just about every company is a software company today) or a DevOps empowered organization without this organic engineering team.

I don’t think any CEO would be against having a high functioning engineering team. But what does Gregoire mean by organic? My understanding is that he is talking about the team being internal. Not outsourced or acquired, but built, grown and nurtured within the organization.

So what does that mean for outsourcing? When software is eating the world, can you have a successful business when you can’t develop and operate your own software? If DevOps is a key ingredient in a high functioning IT organization, can you outsource DevOpst?

In the case of DevOps, with cultural change being such an important ingredient in the recipe, it would seem that you can’t outsource culture, it has to be organic. But if you can’t outsource cultural change and DevOps, can you import it? Can you graft a culture of DevOps into your organization?

This was exactly the conversation I had with another person at CA World and with several people thereafter. At CA World I had a chance to speak with Rohit Antao, a Principal at PwC who heads up many of the DevOps engagements that PwC is involved in. Rohit shared with me that in many of the IT advisory and transformation engagements that the firm undertakes instilling a DevOps friendly culture and process is just about obligatory.

I asked Antao how they go about grafting a DevOps branch onto the main trunk of a company’s IT tree. Like similar paths I have heard from IBM and others, he explained that their blueprint is usually to pick a few small projects to build success upon. From these small projects where DevOps is injected they seek to expand this DevOps way deeper and deeper into the organization. The idea is that after some months the DevOps way of doing things take root. It becomes organic to the organization.

Actually this kind of engagement involving coming in and transforming IT from within, then leaving behind a team and new way of doing things while you move on to the next engagement was something I first saw put in action by a company called The Feld Group. Charlie Feld in many ways pioneered modern CIO models. He and his firm would come in and turn an IT department on its head, hire in some new leadership and when things were on track leave to the next engagement. But leave behind a solid team and culture.

This type of engagement is not only the purview of IBM, PwC and other big name firms. I had a chance to speak with Laszlo Szalvay, VP of SolutionsIQ. They are an Agile Enterprise Solutions company who today announced the acquisition of Boson-based Big Visible to form the largest agile consultancy focused on enterprise transformations.

Laszlo explained to me that in his view there are two kinds of IT transformation consulting engagements. You can tell them apart by a cup of coffee. In one type of engagement the consultant tells the customer to grab a cup of coffee and leave the consultant to his work. The consultant will let the customer know when finished. In the other type of engagement the customer and the consultant sit down over a cup of coffee and plan how they will work together to accomplish the goal which ends when the consultant hands the keys back to the customer after they have worked together to instill in the case of SolutionsIQ a culture, processes and tools for an agile development team.

Funny how though the names, faces, technology and tools change, the process stays the same. The same thing the Feld Group was doing 25-30 years ago is what these companies are doing today. You can graft a new culture and a new way of doing things into organizations. If planned well and done correctly you can grow your own organic engineering team.

Can you outsource DevOps though? Can you have DevOps-a-a-Service? I am not sold on that one. I guess someone will have to convince me on that before I buy into it.

December 11, 2014
DevOps in Action: MYOB

This is the first in a new series called DevOps in Action. A look at organizations who are putting DevOps into practice. Does your organization have experience putting DevOps into action that you think our readers might find useful or interesting? If so contact us at editor@devops.com and perhaps we can feature your story here.

Australia-based MYOB is a long-established developer of accounting systems and related software. During the last several years its original focus on desktop software has changed, and it now offers cloud-enabled and pure SaaS products. This change had two significant impacts on the company.

Firstly, it meant that instead of simply delivering packaged software to customers, MYOB now has to manage the infrastructure that its SaaS products run on.

Secondly, the traditional annual update cycle primarily driven by tax and compliance issues goes out of the window in a SaaS world, where the norm is continuous improvement based on customer feedback.

“Traditional operations management just doesn’t keep pace with that,” said MYOB business division product delivery manager John Sullivan. “The pace of change has changed dramatically.”

Not only are there more ideas for improvements and new features, but the changes are increasingly “externally fuelled,” he said. This makes it difficult to plan infrastructure requirements very far ahead.

MYOB joined the DevOps movement as it was seen as a way of releasing better software more quickly, and of ensuring that the hardware could be scaled to meet the demands of new software releases.

At some companies, DevOps is about building infrastructure for developers to use, or having operations build infrastructure in parallel with software development, he said.

But MYOB now has operations staff that literally work alongside developers to translate features requirements into infrastructure requirements, and then use repeatable methods to build that infrastructure.

“We scale our infrastructure at the time features are being built,” said Sullivan.

The company uses multiple cloud providers, reflecting the varying needs of its 40-plus products. AccountRight Live (a desktop program that stores its data in the cloud) was written using .NET, so it made sense to use Azure for the cloud element.

And it has adopted a microservices architecture so a single implementation of a function can be used by multiple products (read more about this in ‘MYOB moving ahead with microservices’ http://www.itwire.com/business-it-news/accounting-software/65878-myob-moving-ahead-with-microservices). These microservices mostly run on Amazon Web Services.

Everything else runs on virtual cloud infrastructure provided by Macquarie Telecom.

Using these larger providers means MYOB can be confident that the physical resources are available whenever they are needed, without having to invest large amounts of money in private infrastructure. Having the infrastructure “come alive” at the moment any particular piece of software goes live provides significant savings.

MYOB had an important advantage when starting its DevOps journey: the need for operations staff was new, as it was going from being a traditional software vendor to also offering SaaS products. Sullivan said that made it easier to put development and operations staff together than it would be for an organisation that has a history of managing its own infrastructure, especially with a steady release cadence as opposed to the rapid tempo associated with cloud services.

“We were very lucky that we needed to manage infrastructure for the first time,” he said.

But there were some cultural issues, particularly around making the delivery teams understand they had to take infrastructure into account when building features, as it is important that the cost of delivering the service is kept within reasonable limits. So developers need to tap into operational expertise to ensure that the patterns they use will scale adequately, and that the hooks for monitoring and managing the software are included.

MYOB started with an independent DevOps team with an oversight role, but this was less than optimal, Sullivan said. “If you say you have a ‘DevOps team,’ you probably haven’t” because the closer that development and operations staff are, the better the systems become – getting the right design means looking at application features from several different angles.

Those on the operations side need to be able to advise developers about the software patterns and tools that will be able to handle increasing load. Not only does that potentially avoid the need to build complex failover arrangements, it also saves you running into a situation where you have to re-architect the whole system when demand reaches a certain level. “That’s the advantage of having operations and development people close together.”

MYOB uses the Agile methodology, and infrastructure concerns and requirements are included on feature cards so the software and infrastructure can be built together. As the company gets better at DevOps, developers become increasingly aware of infrastructure requirements and so they build as they go – “we’re just about there with one of our teams.”

“You need operations people with the mindset of a developer … or you’re going to struggle,” he said. For example, you want systems that can automatically scale on demand, and that requires scriptable deployment processes: “a difficult thing to do at times.” Fortunately for MYOB, the company was able to find the right people. It can be hard for development staff to adapt to DevOps, Sullivan suggested, but it’s even harder for traditional operations staff.

One of the ‘tricks’ used by MYOB to build the DevOps culture was to rotate developers into operations roles after the software they had been working on entered production in order to learn about operational requirements. “Then they actually feel the pain of phone calls in the middle of the night if something goes wrong.”

Another is that the company favours ‘self-reflective teams’ (regardless of function) that review what has or has not worked for them during the last two weeks, and adapt their practices accordingly. DevOps is just one area where this practice is used. Sullivan notes that because the environment is so dynamic, ideas are not rejected permanently. So teams can decide that just because a certain technique or idea did not work on a previous project, it might be successful this time because the situation is different in a relevant way.

Early in the transition MYOB filled some key roles with DevOps practitioners that had Azure and AWS experience. But Sullivan warned “there are very few conferences or training courses on DevOps … so most companies don’t do DevOps correctly.” Consequently, special care is needed when hiring people that claim experience in the field.

“The term ‘DevOps’ is overused and sometimes misunderstood.” So some people say they have DevOps experience when all they really know about is doing scripted builds, and do not have the skills and practices to be embedded in a DevOps team.

“There are few people that have the right mindset,” said Sullivan, so it is very important to check a candidate’s skill set against your requirements – and on the flip side, to take the steps necessary to retain your good DevOps staff.

So did MYOB get what it wanted from DevOps?

One initial goal was to go from the traditional software-only annual release to a six-week ‘software and infrastructure’ cycle. “We met that pretty quickly,” said Sullivan, and the company is now targeting ‘any time’ releases for online services with minimal or no interruption. This will allow the company to pick and choose which new features are to be released and when according to marketing priorities, rather than being hamstrung by arbitrary release cycles.

The other was to achieve “one click deployment,” that is to be able to release code and infrastructure to customers with no manual tasks beyond initiating the process. The automated process should either run to completion, or back itself out if something goes wrong. This has been achieved for most products, the main exception being those for mobile devices because the requirements of the various app stores get in the way.

“Other companies may have different metrics, but ours were about release capability and infrastructure efficiency,” said Sullivan.

Asked if MYOB had any regrets about going down the DevOps route, the answer was succinct: “None whatsoever.”

And his advice for others considering following this path sounds sensible for any part of working life. “Embrace the idea that you have to grow and learn, and keep making the best decisions you can at every step.”

About the Author:

Stephen Withers is one of Australia¹s most experienced IT journalists, covering everything from gadgets to enterprise systems. In previous lives he has been an academic, a systems programmer, an IT support manager, and an online services manager. Stephen holds an honours degree in Management Sciences, a PhD in Industrial and Business Studies, and is a senior member of the Australian Computer Society. You can see his Linkedin profile here: http://au.linkedin.com/pub/stephen-withers/0/222/687/

December 10, 2014
Q&A with Ken Cheney: Chef and HP partner to bring DevOps to the enterprise

HP and Chef issued a joint press release last week revealing their new partnership. HP Technology Services now offers Chef’s IT automation platform as part of its Datacenter Care service. The move is big news for both HP and Chef, and is yet another illustration of how DevOps is transforming enterprise IT.

The partnership makes sense. It’s hard to talk about DevOps without mentioning Chef, and virtually impossible to talk about enterprise IT without bringing HP into the conversation. The press release explains, “Enterprises in every industry are under significant pressure to deliver on-demand digital products and services to remain competitive. In doing so, they are adopting web-scale practices, including DevOps, to create more agile and resilient infrastructure.”

How does bringing Chef into the mix enable HP to help its customers achieve those goals? With capabilities for high availability, replication, and analytics, as well as comprehensive integration with all major operating systems and cloud providers, Chef gives enterprises a comprehensive platform for automating IT infrastructure and application delivery across Windows, Linux, and Unix, whether in a local datacenter, or a cloud environment.

This isn’t the first big news of the year for Chef. Earlier this year Microsoft jumped on the DevOps bandwagon by integrating Chef and Puppet Labs into the Azure cloud platform. Teaming up with HP is another significant rung on the ladder to making DevOps a mainstream enterprise concept.

I conducted a little Q&A with Ken Cheney, vice president of business development for Chef, about the new relationship with HP:

DevOps: What does this new relationship with HP mean for Chef?

Cheney: This is a multimillion dollar deal with HP—it’s the largest partner deal we’ve ever done. HP Technology Services is packaging Chef’s IT automation platform into its Datacenter Care offering, serving more than 2000 customers globally.

This represents significant validation for both Chef’s commercial automation platform and our market leadership. With HP’s channel, we can enter and help transform very traditional datacenters to adopt DevOps patterns, increase efficiency and reduce costs.

DevOps: In your opinion, what does it mean for HP?

Cheney: HP Technology Services will now offer Chef’s IT automation platform as part of its Datacenter Care—Infrastructure Automation offering. By turning infrastructure into code, HP will now enable even more IT agility and reliability for businesses.

DevOps: There are benefits for both Chef and HP, but the customers are what really matter. How does this partnership benefit Chef or HP customers, or IT admins in general?

Cheney: This new offering will enable HP’s enterprise customers to unify development and operations on a single codebase, accelerating software and application delivery, as well as ensuring maximum security and eliminating costs related to human error or system breaches.

DevOps: What impact does Chef’s partnership with HP have on DevOps in the enterprise?

Cheney: Today, every business must transform into a software company to compete in the always-on digital economy. This shift toward what is called “web-scale IT,” requires companies to embrace web-oriented architectures and open source software, automate operations and move services to the cloud, and take on new cultural practices based on the DevOps movement.

Our new partnership with HP is further validation that the DevOps movement is hitting the bow of the enterprise – we are working together to help deliver the web-scale architecture enterprise customers are looking for.

Only time will tell the true impact of this partnership, or how DevOps culture evolves in the enterprise. It’s safe to say, though, that the combination of Chef and HP is a powerful force, and that the partnership will make DevOps available to many large enterprise customers.

December 8, 2014