devopsy.kinsta.cloud

Category: Features

  • Container Usage Survey

    Container Usage Survey

    Is there any topic in IT hotter than containers today? Hard to make a case that there is. Whether it is Docker or Rocket, LHX or something else, it seems everyone is talking about them. But what are people actually doing with them? devopsy.kinsta.cloud and ClusterHQ are conducting a survey on actual and planned container usage. The results will be published just before DockerCon next month.

    We need your help though to make the survey as insightful as possible. The more responses we have the better. So we are offering a $50 dollar Amazon gift card to one out of every 50 people who take the survey. The survey is only about 20 questions and shouldn’t take more than a few minutes to complete.  We don’t ask for contact info unless you want to be eligible for the gift card giveaway.  Either way taking the survey is a big help to us, ClusterHQ and everyone else as well.

    The link to take the survey is:https://www.surveymonkey.com/s/3TZ9LDC

    One of the key areas of inquiry is how many people are actually using containers in production environment. How many have already used containers versus how many are planning to do so? What about data management in containers? These areas and more are explored in the survey.  It should make for some excellent research.

    Thanks for your help in making this a great survey!

  • Release Management, a catalyst for DevOps

    Release Management, a catalyst for DevOps

    We have discussed in my last post “DEVOPS and Continuous Change” on the benefits of doing shorter and swifter runs instead of managing a few ‘Dinosaur’ releases. We have also touched upon the way to go about multiple releases by identifying component /functional dependencies without impacting functionality and the expected savings of time, resources and cost, reaching the goal of rapid delivery of software.

    It’s common to find a formal Release Management (RM) team in large organization, with a lot more of medium and some small ones following suite. We will not touch upon the benefits of having a core RM team helping plan and manage the complex journey of change from the handover of code by the Application teams, packaging and deploying across testing environments by Operations, validating testing signoffs, risk analysis, impact assessment and ultimately ensuring a successful Production deployment.

    Rather, let’s delve deeper to understand how this Confounding or the Third variable called RM fits into DEVOPS. In my opinion a very important actor which neither belongs to DEV nor OPS!

    I would call Release Management a catalyst in true sense, which enables change without itself undergoing such change, a neutral team dedicated in managing the change by brokering a formal hand-off between DEV and OPS.

    RM teams traditionally have been the most underrated, with the DEV and OPS hogging the credit of every successful run to production. They converge the business understanding of new features or fixing a functionality break with the Development team’s new version of code release or the Operations expectation of infrastructure upgrades which will also provide an enhanced user experience advantageous to business and development.

    So, back to my question, where does RM team fits in the cozy DEVOPS selfie?

    Wouldn’t it be incorrect to ignore RM from the DEVOPS equation?

    I would take the liberty of stating that the DEVOPS movement in an organization should be owned, driven and determined by RM, adding Change/Configuration and Release management expertise to the existing cross-functional mix of the DEVOPS team.

    RM should be driving this collaborative approach to rapid delivery, bringing together the right mix of people from both sides and ensuring efficient use of resources, empowered and with executive oversight supporting hard decisions.

    They would be able to add value in this rapid delivery process, by not just tasking them with managing a release cycle but empowering and enhancing their capabilities. RM team under the DEVOPS banner should also be involved in these broad points below:

    • Prioritizing the book of work, creating trade-off between Business needs and Development delivery.
    • Understanding complex business requirements along with adhering to geographical and geopolitical needs and rules.
    • Distributed Set of applications, architectural complexities, interdependent and independent components and dependencies,
    • Complete awareness of the infrastructure landscape, scalability, capacity planning, migrations.
    • Release planning, calendar management, process adherence, true agile sprints, optimal resource utilization, Contingency or Rollback strategy planning.
    • Post deployment/release lessons learnt, retrospection, streamline or integrate processes after every run.
    • Ensuring consistent message is sent across that the DEV and OPS are not adversaries rather, partners in this collaborative approach of software delivery.
    • Management reporting along with any risks assessed or highlights deficiencies.

    This take is based on the many DEVOPS discussions which I have come across where there is no mention of RM or is being assumed as ‘part of DEV and works with OPS’.

    My disagreement on this thought process is not about to which group it essentially belongs or will be part of, but in assigning an inconsequential role to RM in the whole release journey when in fact they should be the drivers!

    Thank you for your time.

    Visit my blog for earlier posts http://askhurram.com/

    Read On, Live On.

     

  • DevOps Leadership Series 2015

    DevOps Leadership Series 2015

    We recently hosted the 2nd annual “DevOps: Wine-ing, Not Whining” event in San Francisco to bring together a number of thought leaders and leading practitioners in the DevOps field.  We traded stories from the trenches, formed new insights, and offered visions of the future.

    This week, I’ll be sharing videos captured from the event that highlight key themes from our conversations. We call it the DevOps Leadership Series.

    Screen Shot 2015-05-15 at 11.19.59 AM

    You’ll get to hear:

    In the next few weeks, I’ll be on the west coast at the Velocity Conference and on the east coast covering DevOps Days DC.  I’ll be capturing more insight from DevOps thought leaders at these events.  If you are planning to attend one of these events, let’s capture your DevOps perspective for the series.  To participate, please leave a comment below or catch me on @weekstweets.

    Today, we’ll start the series with Gene Kim on important themes for DevOps in 2015:

    [youtube http://www.youtube.com/watch?v=0IXrnhcRNUk]

     

    PART 2 of the series is now available at: 

    https://devopsy.kinsta.cloud/2015/05/19/devops-leadership-series-software-supply-chains/

  • Survey: Fed gov cloud success hinges on DevOps

    Survey: Fed gov cloud success hinges on DevOps

    A new survey out last week by MeriTalk and Accenture Federal Services shows that the federal government is on the cusp of finally putting its money where its mouth is with regard to cloud and that DevOps will play a big role in the transformation. In fact the government’s cloud success may hinge on DevOps.

    The survey looked at perceptions of federal IT managers familiar with their agencies’ cloud progress. Many of them report difficulty in gaining cloud momentum, in spite of the fact that 66 percent believe that their agency needs to move IT services to the cloud faster to meet mission and constituent needs. Fewer than half of Federal agencies have made process or policy changes to facilitate successful transitions to the cloud, only 30 percent have made necessary cultural changes and just 28 percent have made organizational changes. What’s more just 12 percent of Feds report their departments have all the tools they need for a cloud transformation.

    However, 57 percent believe DevOps holds the key to help them succeed in the cloud and 63 percent say DevOps will help them speed up application delivery and migration. As things stand, only one in ten respondents believe that developers and administrators are highly collaborative in the Federal setting and only 5 percent say their agency has fully deployed DevOps patterns.

    “A DevOps model brings software engineering, quality assurance, and IT operations together as an integrated team,” stated Tim Hoechst, Accenture Federal Services’ chief technology officer (CTO).  “Federal agencies can start up new projects more quickly, improve resource utilization, and reduce downtime – ultimately driving the innovation needed for real change.”

    Respondents are hopeful of that vision, with 60 percent reporting they see DevOps in their agency’s future. Approximately one-third say they’ve already started adopting the model or plan to do so within a year.

    But it’s not clear whether this is pie-in-the-sky hopefulness for a silver bullet or a real commitment to DevOps. A big red flag from the survey was that some of the same barriers to Federal cloud adoption are those that typically stand in the way of DevOps transformations. For example 40 percent of Feds said fear of change and inflexible practices have stood in the way of their cloud efforts.

    Over half of respondents say they will need to train current personnel to implement DevOps, and 40 percent say DevOps success depends on incentivizing changes in culture to make it happen. Meanwhile 41 percent believe they’ll need to establish a new vision for the future to make DevOps possible.

  • Netuitive’s Transformation to SaaS Delivery, Powered by Its New DevOps Shop, Part Two

    Netuitive’s Transformation to SaaS Delivery, Powered by Its New DevOps Shop, Part Two

    In part one, devopsy.kinsta.cloud presented Netuitive’s DevOps transformation, which helped the company to automate its product development processes. Now in part two, we take a look at that development process as it stands today.

    Netuitive Development & Technical Debt Initiatives

    “We are using a Feature Branch Workflow model with Git to isolate our work while we develop and test it,” says Brian Spindler, VP of Engineering, Netuitive. As soon as all teams and team members are in agreement, Netuitive merges the feature with the master branch and deploys it to the internal pre-release staging environment for further testing and verification.

    “In the old world, we released software twice a year,” says Spindler. After using the Scrum development process religiously and exclusively to take the software to beta, Netuitive drove the pre-release software through the testing ringer prior to releasing the packaged final product.

    “Now we release on a weekly basis,” says Spindler. The development team shoulders the unit, integration, and acceptance testing while Netuitive reaches forward to increasingly add features and polish its architecture.

    Netuitive is continually bringing the increasing technical debt into the black side of the development ledger, so to speak. Netuitive offers two examples of being technically in the red. The first is in the area of unfinished business. Incomplete or hacked up implementations typically happen when the team is developing outside the core technology stack, is cutting corners for speed, has a poor understanding of the software requirements in a given area, has to rush testing, or does not create complete and appropriate documentation, according to Spindler.

    The second example circles library / dependency updates, where falling behind can lead to numerous interrupts and blocks for features / bugs / requests that are at the top of the queue, Spindler explains. “Often times in either of these two sample areas, a theme will emerge and that can be a good indicator that it is time to start paying down the technical debt in that area,” says Spindler.

    Netuitive puts that debt under a microscope where through transparency and visibility, it can review the bugs on a regular recurring basis and, as soon as an engineer catches something that is offensive to the software, he can create a ticket in JIRA to put it on the backlog to address, according to Spindler.

    “However, it doesn’t always make sense to ticket something if it is not actionable,” says Spindler. In such cases, Netuitive will catch those instances of technical debt in the Technical Roadmap Idea Board that it maintains, which captures three verticals: the development environment, the technical debt, and the architectural / structural vision, according to Spindler. “Any team member can contribute to this Board. To keep everything on it in perspective, we hold retrospectives regularly,” says Spindler.

    Changes, Experiences

    “It was a positive experience seeing how we handle bug fixes differently after switching to a SaaS model,” says Spindler. Now we understand bugs that we fix very well and we easily duplicate them for isolation and to excise them from the software. That’s because the developer has full access and freedom to debug the same system that he reports on. This eliminates the tedium of setting up systems and testing and repairing for edge case scenarios, according to Spindler.

    “We were also positively surprised at how quickly something can go from concept to production,” says Spindler. To make that possible, there is an opportunity cost in that development has to be willing to sacrifice, to try something while risking having to toss something aside. “In the SaaS world, it’s all about trade-offs, fast vs. refactoring, pay off that technical debt or put out feature XYZ. When you have a team like ours, everyone’s time is critical, and we work hard at prioritizing and balancing that,” Spindler says.

    To maintain the balance and improve the team’s precision and good judgement over time, Netuitive follows Kaizen, which is a continuous improvement methodology. “As the team grows and changes, so will our priorities and the way we work and build the system. We’ve got to be open to that and willing to embrace it,” says Spindler. Regular retrospectives and transparency help to make that possible.

  • Surprise! Broad Agreement on the Definition of DevOps

    Surprise! Broad Agreement on the Definition of DevOps

    For the past few years, the conventional wisdom has been that DevOps has no agreed upon definition. Instead, everyone has their own interpretation. However, the conventional wisdom is wrong.

    While everyone supplies slightly different words, the same working understanding of DevOps is nearly universal. Our understanding of DevOps goes back to a handful of common touch-points. There is the “CAMS” (Culture, Automation, Measurement and Sharing) acronym popularized by John Willis and Damon Edwards. CAMS has been extended repeatedly – Eveline Oehrlich of Forrester is now discussing CALMSS. Gene Kim offers his “3 Ways” and Patrick Debois has codified “4 Areas”. Allspaw and Hammond had 10 deploys a day, and Etsy made technologists fascinated by a virtual crafts bazaar.

    Analysts, authors, industry, and community leaders all agree. The ideas are big and mutually supportive. In short:

    • DevOps exists to help the business win
    • The scope is broad, but centered on IT 
    • The foundations are found in Agile and Lean.
    • Culture is very important
    • Feedback is fuel for innovation
    • Automation helps

    Why: To help the business win

    The most popular DevOps book today is The Phoenix Project, with a subtitle “A Novel about IT, DevOps and Helping Your Business Win”. DevOps is not a navel gazing IT methodology for making IT better for IT. It is focused on how stuff involving the technologists can deliver better results for the business.

    As Patrick Debois writes in his excellent post,”DevOps Areas – Codifying devops practices“:

    As rightly pointed out by Damon Edwards, devops is not about a technology, devops is about a business problem. The theory of Constraints tells us to optimize the whole and not the individual ‘silos’. For me that whole is the business to customer problem, or in lean speaks, the whole value chain.

    At IBM, our definition of DevOps calls out enabling the business to “seize market opportunities”.

    Today, technology is critically important to how a wide variety of companies compete. Retailers who neglected e-commerce have gone out of business. Logistics companies like UPS find a competitive edge by creating software to shave miles off delivery routes. In the automotive space, Ford is at tech conferences recruiting companies to write apps for its cars and Tesla’s first fix for their flammable battery problem was an over the air software update.

    DevOps is a response to what the IT is experiencing – from a back-office concern that needed to “keep the lights on” to key part in how the business competes every day.

    Who: A Broad Scope, beyond development and operations.

    Yes, DevOps is a portmanteau that brings development and operations together. Yes, the classic DevOps concern is the Wall of Confusion between development and operations. However, with the focus on business value, the scope must include everyone who is involved in taking an idea through IT to business value.

    In recent years, there’s been something of a cottage industry of community leaders and analysts writing, “No, no it really should be Dev_____Ops” where the blank is filled in by their own specialty. Examples: DevQAOps, DevOpsSec, DevSecOps, BizDevOps and of course Bussdevtestqanetsecnetops.

    When a Finance department must approve the provisioning of each new test environment, slowing the delivery of new applications by weeks or months, there is a DevOps problem that leaves narrow confines of development and operations. It leaves IT. Patrick’s quote above talks about optimizing the whole value chain. In that post he describes DevOps as being about “collaboration, optimization across the whole organization. Even beyond IT (HR, Finance…) and company borders (Suppliers)”.

    Based On: Agile and Lean

    In answering why he didn’t support a manifesto for DevOps, Patrick Debois points people back to the Agile manifesto and asks them to reread it more broadly. Agile’s shift towards a faster delivery of software, and focus on the end goals make it a natural spiritual ancestor for DevOps. However, from the perspective of Ops teams, Agile represented the barbarians at the gates. Agile generally took root in development side of the house, enabling it to create something the business wanted faster. But with operations still moving at a pace suited to waterfall, pressures grew and the need for something like DevOps became clear.

    Where much of DevOps energy seems to come from its Agile roots, the intellectual side of the movement is more rooted in Lean. It seems no accident that many of the Phoenix Project’s lessons are taught away from the cube farm and instead from the factory floor. While we can intuit that faster feedback is better, and the “continuous” in CI and CD is convenient because it lets us map feedback to fewer changes, it is in the Lean literature where we encounter Little’s Law. It provides the queue theory mathematics supporting the idea that smaller batch sizes supports shorter cycle time. And so Reinertsen’s “The Principals of Product Development Flow: 2nd Generation Lean Production Development” has become popular in DevOps circles and is well cited by Jez Humble in his new Lean Enterprise book.

    Culture: Collaboration and Experimentation

    The Agile and Lean roots show through in the DevOps culture as well. Lean and Agile are both focused on people first, systems second and heavily cross-train or work in cross-functional teams. They also emphasize continual improvement (kaizen) through approaches like retrospectives.

    In DevOps circles, collaboration is preached as an antidote to thinking in siloes which has resulted in antagonism between development and operations. For the systems thinking required to deliver for the business, teamwork is required. Developers make better decisions when they take advantage of operations knowledge (Patrick’s Area 3) and operations does better when application knowledge is embedded with them (Area 4).

    The other cultural emphasis is a push towards experimentation, constant learning and improvement. This is Gene Kim’s third way of DevOps. The Lean Startup ideas of Eric Reis permeate the DevOps world. Intuit has made great strides in showing how these approaches work in established companies by running experiments in the midst of tax season resulting in improved business outcomes.

    These cultural trends come together in things like blameless postmortems discussed by Etsy and Jake Loomis in Web Operations. When things go wrong, organizations need to understand why and take corrective action. In troubled corporate cultures, the messenger reporting the problem might be shot, and the hunt to assign blame for the problem will begin. Healthy organizations look to learn from failure and try to improve the systems that allowed (or guided) people to make a series of mistakes that lead to the failure.

    Feedback: Yes. Lots please!

    Experimentation, adaptation and learning tend to do better when there is ample feedback. Otherwise, there is too little to learn from. Developers tend to be more productive as well with rapid feedback. Their thirst to know whether the latest change diminished quality led to the practice of continuous integration and the tools that support CI and continuous delivery. That CD ends up facilitating faster delivery of value to the customer is something of a happy accident.

    Amplifying feedback loops is Gene’s “second way“. The “CAMS” concepts of measurement and sharing point to feedback. You gather lots of information, and make sure people see it so they have an opportunity to learn from it. And when Elisabeth Hendrickson presents at DevOps Enterprise 2014, as Cloud Foundry’s Director of Quality Engineering at Pivotal, she describes her job as “to see to the care and feeding of the feedback cycles“.

    As expected with a collaborative culture with ample sharing, feedback must cross the traditional silo lines. That means the awesome production monitoring that had lived in Ops land, is radiated back to development. This is “Area 2” in Patrick Debois codification.

    Automation: Of course.

    The “A” in CAMS is automation. Patrick’s “Area 1” points to continuous delivery techniques, well described by Dave Farley and Jez Humble in their book. A Lean initiative looking at IT will find waste in wait times, errors, and more and move towards automation as a natural result. Feedback craves automated deployment of changes, and automated tests or production monitoring that validates them. Automation is what makes a lot of DevOps possible. It doesn’t make cooperation between development and operations happen, but tensions are reduced when Ops gets out of the business of arguing about the SLA they have with Dev around server provisioning and instead provide Dev a button to press.

    DevOps isn’t about automation. Automation is just the natural result of DevOps.

    Additional Resources not cited above

    DevOps for Dummies – by Sanjeev Sharma

    8 Best Practices for DevOps – Ashok Reddy’s overview of the Forrester Report

     

    About the Author/ Eric Minick

    ericMinickEric Minick is a Technical Evangelist. He joined IBM through its acquisition of UrbanCode. He has spent the last ten years helping organizations – large and small – adopt continuous integration, delivery and now –DevOps. He is a frequent speaker on the topic, and co-authored Application Release and Deployment For Dummies.

    Prior to consulting, Eric was a developer, test automation engineer and support engineer, and has contributed to multiple generations of UrbanCode products.  Follow him on Twitter at @ericminick.

  • Netuitive’s Transformation to SaaS Delivery, Powered by DevOps, Part 1

    Netuitive’s Transformation to SaaS Delivery, Powered by DevOps, Part 1

    “Netuitive offers an adaptive monitoring solution for cloud infrastructure, applications, and services, powered by a patented self-learning correlation engine,” says Brian Spindler, VP of Engineering, Netuitive. The engine uses multivariate regression analysis to calculate correlation coefficients between metrics based on time-series data collected, leveraging the significance of certain relationships between metrics variables across the data.

    The Netuitive technology establishes a baseline of what metrics and their relationships have been historically. Netuitive policies enable development and operations teams to see anomalies that present with significance above the historical baseline and related to AWS EC2, Linux OS, MySQL, and other systems and technologies.

    “We have launched a SaaS version of our product wherein we now deliver code to production on a weekly basis,” says Spindler. This increase in deployment speed is thanks to the company’s new DevOps processes. Here is the first part of Netuitive’s two-part story.

    Before Moving to a DevOps Shop

    Before moving to a DevOps software development approach including continuous delivery, Netuitive used a Scrum framework together with Jenkins and Ant products and scripts the company developed in-house to develop Netuitive software and the patented self-learning correlation engine within. “We recognized the value of automation early on—using simple shell scripts and our own Web API we were able to orchestrate quite a lot,” says Spindler.

    Still, these methods presented challenges. The extensive planning process, including conjuring up development time estimates that Scrum requires for Sprints were challenging. In addition to and despite all that planning, numerous interrupts occurred surrounding testing and deployment issues that stemmed from software complexity and integration management. “These interrupts slowed us down and impacted our burn rate,” says Spindler. (The burn rate calculates the speed of the business spend above and beyond income.)

    Mountains of bug fixes were also a concern. With an exorbitant amount of time spent fixing bugs that would only present themselves in rare edge cases, Netuitive was putting too much effort into builds while getting too little in improvement or new features out of the process. “While we sought a solution for feedback loop issues, it became clear that this would require a more agile approach. We looked to DevOps for inspiration, which led us to migrate from Scrum to Kanban,” says Spindler. Kanban is an illustrative process for managing targeted production goals, including what to develop when and how far to extend that development.

    As enterprises adopting DevOps often mention, the cultural change necessary to move to DevOps approaches was the most challenging. In particular, getting everyone on board with automated testing as a proof of software validity was tricky. Internal teams such as QA and developers had to transform their existing relationship. “For years, these groups were adversaries; now I am asking them to be allies. This required the developers to take on the responsibility of writing tests while QA validates and ensures the tests are sufficiently broad,” says Spindler.

    Goals, Results, & Hazards: Method to Madness?

    Netuitive had specific outcomes in mind from switching to a more complete DevOps shop. Quality and productivity improvements were high on the list. In fact, the two would have to be mutually inclusive and close knit to ensure both. To accomplish this, Netuitive deploys on a weekly basis, using the full Atlassian suite for collaborations and issue tracking along with a combination of Bamboo, Gradle, and Salt Stack to automate the configuration, manage the infrastructure, and manage deployment operations. This orchestration helps quality and productivity to dance in harmony.

    The results are more than aesthetically pleasing. “We are increasing the velocity of deployments through continuous process improvements,” says Spindler. By deconstructing the Netuitive product and dividing it up into independent service modules and components, the company can move overall progress along more quickly by improving each individual component separately and simultaneously, making faster progress gradually than they would by waiting to finalize a single, complete product update.

    Netuitive experienced three fully distinct bumps in the road or ‘hazards’ on the way to morphing itself into a DevOps shop. “There were issues related to automating testing, changing the organizational workflows to accommodate automation, and deconstructing the product into granular enough services that we could deploy independently,” says Spindler. These were all related to the desire to automate the product production as a whole. Netuitive works continually to refine the necessary balance between layering on additional tests across those granular services to raise the bar on quality and excising some of those testing layers in order to increase deployment velocity.

    Tune next time when devopsy.kinsta.cloud reproduces the relative chronological Netuitive DevOps development process from release to release along with differentiations, benefits, and the payoffs that the company did not expect.

  • Step-By-Step: How DataArt Processes Application Backlogs, Converting Legacy Apps with DevOps

    Step-By-Step: How DataArt Processes Application Backlogs, Converting Legacy Apps with DevOps

    DataArt designs and custom builds software for industries such as travel & hospitality, media, IoT, financial services, and healthcare. “We build new products and services, improving existing solutions, re-engineering legacy systems, and providing expertise in technological and industrial domains,” says Anton Krasikov, Project Manager, DataArt.

    How DataArt Uses DevOps

    DataArt quickly delivers high quality software in support of operations performance. DataArt’s DevOps process enlists version control systems like Git and build systems such as Ant, Maven, and Gradle. “Almost every project relies on using these build systems inside CI servers like Jenkins, TeamCity, or TFS,” says Krasikov.

    DataArt’s development process also requires a lot of unit and integration testing using frameworks like TestNG and Junit. “There are also projects where we prefer TDD or BDD approaches, with the use of specific frameworks such as Selenium for UI testing,” says Krasikov. By using Puppet, Vagrant, and Chef, DataArt can swiftly deploy the development environment in the CD pipeline. DataArt applies these select DevOps development tools using methodologies, which could include Kanban or Scrum (Scrum was most frequently mentioned by DataArt for this story).

    Previously, DataArt relied on manual processes or scripting tools for build systems, source code, and unit tests. “It took a lot of management effort to tie everything including development, quality assurance, and operations together,” says Krasikov.

    Challenges to Adding Backlogged Applications to Those You Develop Using DevOps Approaches

    In the instance where DataArt returns to work on existing systems that include some legacy applications, it is possible to use agile development in maintaining these as they are, but excessive software dependencies could make it hard to align the software releases / release cycles.

    “Legacy software may not have good test coverage or a proper build system, so we usually take steps in a DevOps direction, if not to achieve continuous delivery, at least to implement proper continuous integration,” says Krasikov.

    There are challenges to adding backlogged apps to a DevOps development process. “These kinds of non-DevOps applications usually suffer from several issues, or in worst case scenarios from all of them,” says Krasikov. Backlogged apps waiting in a pre-DevOps state and hungry for transformation may have any or all of the following symptoms, according to Krasikov:

    *many dependencies between system modules
    *no automated environments
    *no continuous integration
    *no test automation
    *no build system or inefficient usage

    Transforming an Application, Step-By-Step

    To add apps to DevOps approaches, says Krasikov, start by analyzing the existing source code, system dependencies, and system environments. The steps that follow come from an example with a recent Java project that DataArt did.

    A) Select a source code control system such as Git.

    B) Do refactoring and perform unit tests. “These go hand-in-hand, so the main business logic was covered with tests. It does not make sense to rush to 100% test coverage, but we tested the main bits. This also allowed us to reduce system complexity and decouple system components,” says Krasikov.

    C) Build system – “The original application used huge and confusing Ant scripts to produce deployable packages. After we completed the refactoring, we split the application into modules with Maven as the build and dependency management system,” says Krasikov.

    D) Integration tests – “We introduced these as a separate test automation suite to ensure that application modules work correctly as a whole,” says Krasikov.

    E) Continuous Integration – DataArt set up a Jenkins server for multiple build jobs, building with each commit and targeting deployments to their intended environments with overnight integration testing. “CI allowed us to identify non-working builds early to save time on investigations,” says Krasikov.

    F) Environment setup – Finally, DataArt implemented infrastructure as code. The new development environment uses Puppet with Vagrant to check out source code, spin up a VM box / template, and deploy fast, according to Krasikov.

    DataArt runs some steps in parallel for additional time savings, for example moving to virtualized provisioned environments along with source code migration, according to Krasikov.

    DevOps Helps DataArt Speed Projects to Their Customers’ Delight

    By increasingly converting backlogged apps to DevOps development approaches, the enterprise benefits by achieving project goals and delivering a working software iteration with each pass through the DevOps processes. “The high level of automation reduces the risk of human errors and helps us to fail early in the development cycle. With continuous delivery, it is much easier for our clients to see the immediate impact of their ideas,” says Krasikov.

  • The DevOps Donkey represents the rest of us

    The DevOps Donkey represents the rest of us

    Yesterday I wrote about a session that I wanted to see by Michael Cote’ called The Coming Donkey Apocalypse. In my article I said on its face I didn’t agree with calling the mainstream adopters Donkeys. I also felt that their adoption would not result in an apocalypse.  I also said that you can’t judge a book by its cover and you can’t judge a session by its title and abstract. I was really looking forward to see what Cote’ was getting at in this session.

    So I eagerly grabbed a near front row seat when he took the stage right after Damon Edwards excellent opening keynote. One of the first things Michael addressed was his use of the term Donkey. Actually he started off with his conclusions which I thought was a very effective way of delivering a session as it focused the audience on what the takeaways are supposed to be. Anyway Michael says that donkeys are hard working beasts of burden, gets no respect, has to do more with less and represents the “rest of us”.  The donkey as a jackass connection is not part of this. It is more that while Unicorns are magical, horses are actually regal but most of us are more like donkeys.  Put better as one of Michael’s co-workers said “the donkey is real people with real problems”. DevOps Donkeys are what is coming as more mainstream adoption of DevOps takes place.

    Michael had a lot of data showing that DevOps adoption is very much on the rise. Multiple sources showed a continuing further penetration by DevOps in the marketplace.  Also the idea of what DevOps is understood enough to  move on beyond that. The jist of the message was that the people attending the session had to do what they could to help these donkey’s in their adoption and journey to DevOps.

    But what about the apocalypse? This is important.  It is not that “bending DevOps to their will” is the apocalyptic event. Rather the apocalypse is the fact that according Ian Head of Gartner says that “by 2018 90% of I & O organizations attempting to use DevOps without specifically addressing their cultural foundations will fail.”

    This is something I agree with. All too often the tendency is to “bag dive” into tools and features. While everyone gives lip service that DevOps has a strong cultural element which trumps tools, to too many people it is just lip service. They will talk about the cultural aspects then go run out and buy tools to implement.  Then they wonder when their DevOps projects fail.  That is apocalypse we need to avoid.

    I spoke with Cote’ briefly after his session and clarified some of his message and thoughts. I am embedding a slide share of his presentation from slideshare. Though marked draft, it seems to be complete. I will update it when a later copy becomes available.

    So the take away from me is something that I have seen time and time again. How do you help donkeys or anyone else implement a cultural foundation for DevOps in their organization. This is a different problem than picking the right tool. In talking with Michael he mentioned that transforming corporate culture is something that companies such as McKinsey and Bain literally wrote the book on.

    The problem is can donkey’s afford McKinsey or Bain? Probably not. So how can they get practical, affordable counseling and advice on transforming their culture.  I don’t know any companies offering that today.  But if any of you do, leave a comment so perhaps others looking for this will be aware of them.

    Anyway, it was a great session by Michael Cote’ in a day full of great sessions and DevOps learning in Austin today.  I am looking forward to another great day tomorrow.

  • Netskope Draws Strength from DevOps In Support Of Scrum Development

    Netskope Draws Strength from DevOps In Support Of Scrum Development

    Netskope keeps enterprises and their data secure despite the unbridled mix of approved and unapproved mobile apps in the market and on employee devices. Employees use unapproved mobile apps (BYOA) on their BYOD smartphones, tablets, and laptops when connected to networks away from the enterprise. IT professionals are keenly interested in tools that help them to have visibility into these apps and devices and to secure them.

    Enterprises face many issues with mobile apps. They confront challenges where data location and access as well as security are concerned when complying with PCI DSS, HIPAA, SOX, GLBA, FISMA, FERPA, and the ISO standards. Netskope works to control app usage, protect enterprise data, and maintain compliance on various endpoints whether the device is on premise or remote / mobile, all in real-time.

    “Netskope uses a distributed policy enforcement model that lets your people use these apps securely so you don’t have to block them completely,” says Abhay Kulkarni, VP of Engineering, Netskope.

    Netskope’s Security Duties Dictate Its Development Path

    “We do not follow Continuous Delivery,” says Kulkarni. Here are two of the many reasons for that.

    First, Netskope’s customers demand that software releases are predictable. “Our customers are large enterprises that want predictable change windows so they can ensure that they make the necessary changes on their side or that they prepare for our updates,” says Kulkarni.

    Second, because Netskope is a SOC-1 Type-II and SOC-2 Type-II certified company, it must comply with the requirements that comprise these certifications. Tightly-knit change management is critical to SOC certification. “As we do each change, we need to make sure that all changes meet the security, availability, integrity, and confidentiality criteria of these certifications by testing the entire product end-to-end,” says Kulkarni.

    But Netskope follows other DevOps principles such as using its own adaptation of a well-accepted source control workflow model as well as an automation model, a model for bootstrapping machines, and also models for software installation and change management.

    Netskope performs a balancing act between complying with SOC-1 and -2 trust services criteria and principles and maintaining rapid feature development. On the one hand, they manage to limit what individual developers can do on production machines—many times disallowing access. On the other hand, they use Sumo Logic to enable their coders to peruse anonymized logs in order to gauge how the service they instantiated is working.

    Goals, Tools, & Challenges

    “The key goal for Netskope’s DevOps approach is to ensure developer responsibility for running, monitoring, and scaling their services,” says Kulkarni. Netskope’s developers and operations staff work together from design through development while complying with a common framework, which helps in resolving development issues and brings consistency to data center operations, according to Kulkarni.

    Netskope holds developers individually accountable for making their software ready for automated deployment. To help with that, Netskope uses familiar tools such as Ansible, which helps Netskope to automate everything in production. “Right from bootstrapping a machine to changing network configurations, we automate all commonly repeated tasks,” says Kulkarni.

    Netskope deploys Sumo Logic agents using Ansible as a part of bootstrapping so that machines are configured from the get go to send logs to Sumo Logic. “Each service has standard logging methods that allow Netskope to create performance dashboards and to track errors,” says Kulkarni. Netskope uses a number of other tools that are commonplace in DevOps, such as Nagios, a monitoring tool and various external monitoring tools for tracking end user performance.

    The biggest challenge for Netskope was not in tooling but in building a DevOps culture where developers lead the charge in responsibility for quality, error-free operation of their services. “In a non-SaaS world, developers ship a version of their product and celebrate. However, shipping a SaaS product is just the beginning. There is really no celebration since you are now responsible for the upkeep of the services,” explains Kulkarni.

    A Trip Through Netskope’s Development Process

    Netskope starts with a Scrum sprint planning meeting where in they decide what features to address for the given sprint. “We try to be realistic about our expectations in delivering code in a sprint by applying quality criteria—i.e. we expect code to be of a certain quality or else we cannot deploy it,” says Kulkarni.

    Then developers bundle various modules on a single branch, which they test for functionality, security, performance, and integration. As the modules and the branch increase in stability, the developers upgrade them from one stack to the next until everything is deemed fit for a production deployment, explains Kulkarni.

    Results

    “Often we see proactive notifications from developers about issues they see in production. Detecting problems before our customers see them is a key benefit of the DevOps model,” says Kulkarni.