Category: Blogs

  • DevOps by Osmosis — Is That Even Possible?

    DevOps by Osmosis — Is That Even Possible?

    “We don’t do DevOps. We can’t, as our development plans are agreed so far in advance and our priority from a production point of view is to keep our systems available 24/7.”

    I heard this comment from a pal of mine as we had a quiet beer over the weekend, but I’m sure this comment, or a variation on it, has been uttered many times in many places. Here’s the irony: I had met my pal, who works as a developer for a major financial services company, after he had had the week from hell. His group tried—and succeeded—to fix a major system issue that had come out of the blue. He had certainly earned the beer that week.

    So, I asked him: “How did you get the problem sorted in the end?”

    “Well, we had to form a Tiger Team,” he replied. “Pulled together some of the DBA team, the key developers and the Ops guys, worked throughout the night and managed to keep the systems up and running the next day, while we used that time to address the root cause.”

    “How did you get around the Governance guys?” I asked.

    “Well, we brought one of them into the team for a few hours, and she was able to get us past some of the key bottlenecks,” he said.

    “And how about QA?” I asked.

    “Oh, they knew this was a major issue and so we cut out a lot of the QA and just went ahead,” he replied. “It was a bit risky, but we had all the right guys in the room to fix anything that cropped up.”

    “So, you manged to get a major change through in a couple days, kept your systems up and running, kept your customers happy, and earned that beer,” I said.

    “Yep, that’s about it, alright,” he said.

    “So, that’s pretty much what DevOps is all about—minus the emergency element,” I pointed out. “Seems to me like you guys are already on the DevOps road.”

    It was about then that the penny dropped with my pal. He suddenly realized that yes, they could do changes quickly, yes, they could do them in an orderly manner and yes, they could keep their systems up and running while all that happened. But, as quickly as the penny dropped and the realization dawned on him, the light went out again.

    “There’s still no way we can do DevOps in our place,” he said. “There’s too many hurdles to overcome, too many guys used to doing things the existing way, which to be fair, has worked well for us. To be honest, there’s too many guys who would feel under threat if we suddenly changed how we do things.”

    And that, in a nutshell, sums up the major challenge facing people promoting a DevOps approach. I have spent many years in the IT industry; I go so far back that I wrote my first code using punch cards (look them up) so I’ve seen change come and go, and change has been the one constant. I think many people in the IT industry have become too focused, whether it’s on the latest tools in their area of expertise, or on the newest piece of kit that has been bought, or indeed on the latest problem the business has put in front of them. The industry has become mature—if we compare the IT world to a field such as medicine, you pay passing attention—if you even notice at all—to things outside your specialty field.

    Now that’s right and proper. I don’t want an orthopedic surgeon trying to fix the problem with my eye, and I don’t want a DBA to write me some code. But when something comes along that affects the whole industry, I want them to pay attention and not ignore it.

    Daysha ran a lunchtime seminar last week, and we titled it “BizDevOps – DevOps isn’t compulsory but neither is survival.” We wanted to use this seminar to engage business people who, let’s face it, are the people we support and who need to understand how IT can make things better. Somewhat disappointingly, while we attracted a great crowd, they were all IT people. No one from the business attended. Now maybe we advertised this event in the wrong places, or maybe we held it in the wrong place, but here’s the thing: We attracted IT people from major companies, and none of those IT guys said to their business people, “You need to hear about this, come along for lunch.” I hope that after our seminar they went back to their business people and said just that, and maybe the next lunch will have the business people in the majority.

    When PRINCE came out into the project management world, it was not bought into by a majority of project managers. The same happened with ITIL. And the same will happen with DevOps. It is fair to say that PRINCE and ITIL are pretty much standard on a worldwide basis. I am old enough to remember the time I first heard of them, and the time I did my first course and passed my first tests, but nowadays, they’re part of the furniture—taken for granted, and rightly so.

    I think people are a little bit afraid of DevOps, and I can see why this is the case. Think about it for a second: Someone is now saying that the way we have done things for years has to change. That we can do things better, faster, with higher quality, but we just can’t do the massive 12-month long project in the same way. We might need some new tools, primarily on the Ops side. We might need some new skills—again, primarily on the Ops side. But, by the way, we’re not going to do away with the Ops guys (so don’t expect to make major budget savings there). But here’s the thing: Our end customers will notice the difference, and so will not be tempted by the new kid on the block—who, as it happens, is probably using a DevOps approach to develop his application, which is threatening our business.

    Change is scary, it’s not easy, and making a major change while you’re still having to churn out the new business requirements, and keep the current systems running … now that’s really scary. But, like my pal whom I shared the beer with last weekend, maybe you’re not as far away from that change as you think you are, maybe it’s not as scary as you think it is, maybe it will affect people positively, maybe you just have to do it.

    About the Author/John Brophy

    eJohn Brophy is currently Head of Professional Services at Daysha Consulting in Ireland, where he leads a team of 25 project managers and business analysts working on projects in blue-chip clients. He is a 35-year veteran of the IT industry, starting out as a database systems programmer, moving onto leading a team of systems programmers looking after what were then called midrange systems. He then became a senior project manager with a global IT company and followed that up by running his own IT infrastructure support company for 6 years. His background is Ops – and he is proud of that!

  • Q&A: Sematext’s Thies on Docker Monitoring, Part 1

    Q&A: Sematext’s Thies on Docker Monitoring, Part 1

    Stefan Thies is DevOps evangelist Sematext Group. I recently spoke with him about Docker’s use of Sematext Logsene logging, anomaly detecting and alerting software and SPM monitoring software. Read on for part one of our conversation.

    David Geer: Technically speaking, for the purposes of this article, what is logging?

    Thies: Logging means to store the output of programs (applications, servers) running on a computer. Typically, log messages that software applications produce are stored in log files. However, software running in Docker containers writes its log messages to a virtual terminal (console). The Docker Daemon (the server process managing the containers) has several facilities to forward log messages via log drivers or the Docker API to external applications, which are responsible for storing the log messages. This is not the traditional way of working with log data. This changes the log collection and storage mechanisms and forces many organizations to change their workflows for log management by introducing new tools.

    Typically, you would use a service called syslog to receive and store log messages on servers, but this requires special configuration for all Docker containers. While Docker made it easier to deploy applications, the new technology made logging and monitoring much harder as you could not use the traditional tools apart from integration with Docker.

    This is where companies like Sematext provide added value, by enabling:

    • Integration into the Docker Ecosystem
    • Cloud storage for log files
    • Indexing for full text search, analytics, visualization and alerting on log data

    Geer: What has Sematext Logsene contributed to logging, technically?

    Thies: Logsene Log Management includes all required workflows and security procedures to handle log data. Logsene provides role-based access controls for access to log data. Logs are collected and shipped to Logsene servers (running in the cloud or on premises) where the data gets stored and analyzed.

    Other important features are:

    • Full text search
    • Analytics
    • Alerts
    • Anomaly detection

    Geer: In what way is Logsene/Sematext available to users and organizations that seek solutions to capture logging for monitoring their Dockerized distributed applications?

    Thies: Logsene is available on premises and as SaaS in the cloud. It fits in hybrid environments as well such as running the Dockerized application in-house on bare metal while shipping logs to Logsene SaaS. In this case, only Sematext Agent for Docker runs on premises.

    Sematext Docker Agent automatically detects running Docker containers and automatically collects logs from the containers. The Sematext Docker Agent can transform unstructured log messages into structured data for analytics. We do this with integrated parsers and a pattern library, which detects log formats to structure logs. The final task of Sematext Docker Agent is to ship the structured logs from all containers securely to Logsene, where indexing for full text search and analytics queries takes place. The Logsene user interface provides all tools for search, visualization, anomaly detection and alerts on log data.

    Geer: What is SPM for Docker? What role does it play?

    Thies: Both Logsene and SPM collect data via the Sematext Docker Agent. SPM is a performance monitoring, anomaly detection and alerting solution for many server applications like Web servers (e.g., Apache, Nginx), databases (e.g., Cassandra, MongoDB, HBase, MySQL, etc.), big data engines (e.g., Spark, Storm, Hadoop, etc.) and full text search engines (e.g., Elasticsearch, Solr). Sematext Agent for Docker collects not only container logs but also performance metrics of all containers at the same time, plus Docker events.

    SPM for Docker provides operational insights into application performance on the Docker platform. It is not limited to performance monitoring—it integrates with Logsene for log management in a single user interface. DevOps teams can find log messages in correlation with potential performance bottlenecks. Most organizations use different tools for performance monitoring and logging. Sematext has recognized that metrics and logs must be available in a single platform to provide detailed operational insights without having to switch between different tools and without overhead for things like integration of multiple tools, multiple user interfaces, logins managing access permissions for multiple tools, and billing or licensing from multiple vendors such as NewRelic or Splunk.

    Watch this space for part two of my conversation with Stefan Thies of Sematext.

  • North Carolina May Meeting: DevOps – From the Hype to the Bottom Line

    DevOps is generally defined as software development movement that encourages automation, integration and greater collaboration between software developers and operations staff. By bridging the gap between those two sometimes warring factions, DevOps establishes a consistent, repeatable way for IT to manage its production environment, resulting in faster time to market, higher productivity levels and smoother server and application deployment.
    What does this definition really mean? Does the DevOps movement really deliver on the bottom line? Are organizations buying in to DevOps at all levels, or is it just the latest in a long line of the latest and greatest software development methodologies? What are the key elements of a successful DevOps initiative? What skillsets should IT professionals grow to be successful in a DevOps environment? Should organizations replace existing successful best practices with DevOps methodologies?
    Our panel discussion of IT leaders in May will answer these questions helping sort through misconceptions about DevOps and its impact on the software development and delivery cycle.  Panelists with links to their bios include:·
    Frank Isaacs DevOps Engineer at IBM
    Salill Soman  Director, Application Development at MetLife
    Josh Wyatt    Director, Global DevOps at LogicNow.

    The meeting will be on May 12 at The NC State University Club, 4200 Hillsborough Street in Raleigh.  Registration and networking starts at 5:30 p.m., dinner is at 6:30. The feature presentation will be from 7:15 pm to 8:30 p.m.

    Attention Job Seekers: Bring your resume

    http://rtp-aitp.org/next-meeting/
  • Pay Down the Skills Debt Before the Technical Debt

    Pay Down the Skills Debt Before the Technical Debt

    When we talk about DevOps, we talk about addressing technical debt. We talk about how modern delivery processes and solutions might help you cure that technical debt of monolithic systems that are built on obsolete technologies, using antiquated development methodologies. But that jumps past the first critical step toward transformation, as technical debt is not the biggest inhibitor to DevOps or faster delivery. By far the biggest inhibitor to companies moving forward is what I call the skills debt.

    I constantly hear this when talking to executives: “I’d like to move forward with a change, but … my team is …” and that statement always represents a skills debt. The primary limiting factor is not the systems in play, it is that the people aren’t ready or positioned to make the necessary changes. We need to make sure we have a strategy for eliminating the skills debt before addressing technical debt. This is the single-biggest risk to DevOps inside the organization.

    Why is the skills debt so endemic? It’s not that people don’t care about their jobs, or that they are not smart or capable enough. It is a self-fulfilling prophecy for most companies because people accrue habits, usually over the course of a decade or more, in the way things are done.

    Changing Mindsets to Fill the Skills Gap

    Ossified attitudes. Just like we talk about addressing the technical debt of hardwired, inflexibly constructed monolithic apps, organizational structures naturally will harden and develop into ossified attitudes as people favor procedures and skills that are already in place. An organization behaves according to Newtonian physics when it comes to change: An object in motion wants to stay in motion, an object at rest wants to stay at rest.

    It’s not as though companies want to have a skills debt, but over time they come to value “the way things are done around here” as part of their identity, and the way corporate structures handle career advancement can even play into this ossification. Think about the technical leadership of the company, the people who rise to the top might have managed successful projects in the past, and while they may not crack open an IDE and code anymore, they will still tend to advise teams to stick to what worked in the past.

    New people and ideas can’t stick. Let’s say you have this established skills debt running through your halls, and you bring in a new hotshot to shake things up. Once this person tries to introduce change, they will immediately find themselves lacking traction at every point. The skills debt is like a Teflon coating around the organization’s IT practices, protecting the core of the business from its own ability to change. A new person with new ideas will just skate off the top and bounce right off.

    If you work for a cutting-edge startup, or someplace like Google or Apple that naturally has innovative engineers lining up to get in, you probably don’t have a problem with recruiting and allowing new hires to make an impact. But what if you work at a big bank, or a large retailer or a consumer goods manufacturer? Changing attitudes and new skills will largely have to be cultivated from within.

    Consulting doesn’t address your skills debt. So if you can’t change fast enough, or hire the right people, how about getting some outside help, some “Pros from Dover” to do that transformation for you?

    Consultants will approach an engagement from two angles. Many times, the consultants are very academic. They’ll come in and show you a fancy PowerPoint, maybe teach a class on Agile Methodology and tell you what’s wrong with your processes, without a practical starting point. That solves neither the technical debt, nor the skills debt.

    Or in the outsourcing model, the consultant can offer to take all your problems away, and have their own resources do all the work in the project for you, even implement a whole new system. That may solve your short-term technical debt, but it will simply allow the skills debt to continue to accrue. You now have a permanent dependency on this external team, unless you allow the project to end. In any case the people responsible will someday move on to other projects and away from your company.

    Digging Your Own Grave

    So, give up and stand pat? Remaining static is a huge mistake in today’s economy. I don’t have to explain that standing still means certain death for your company. On average, a firm in the S&P 500 gets replaced every two weeks. You’ve got to be able to constantly outpace competition—and even your own past results—on a consistent basis.

    What you need to do is take the people that are stuck and unstick them. Yes, sometimes specific people will need to be replaced, but by and large the skills debt is something your whole team has to own responsibility for as a first priority before we talk about the technical debt of DevOps. You have to start breaking skills debt down from the inside out, because doing it from the outside in has been proven to fail.

    Start changing the DNA. You have to start somewhere. Nobody will be motivated by hearing that the way they are is “no bueno.” If you go and mandate a sweeping cultural transformation across the whole company, everyone’s heads will explode.

    The linchpin to making transformation work? A single project. A real piece of work done a different way, by a representative group of the people who need to change. You know how gene therapy is done? One project that allows teams to use the right tools for the job and modern techniques will insert a bit of code into that organization’s DNA. That replicates into beneficial changes elsewhere in the organization as the success and better experience of that team is propagated.

    Skills debt is only paid down through direct transference during actual work. This doesn’t happen without effort, but few worthwhile things are achieved without a little struggle. You can’t just tell people about it, or teach them about it—you have to get people involved in projects that resolve the skills debt of DevOps, before you solve the technical debt.

    One technology executive at a big media company who made this change said his teams used to ship code to release every seven months, and now they are shipping once every two weeks, with an eye to releasing once a week. That’s a huge impact that would not happen without paying down the skills debt. There’s not any one magical piece of software the company installed to make that change; in fact, it is still working on the technical debt side of the equation today. It made the change happen within, starting with one project done the right way.

    Getting people ready for DevOps-style transformation takes a focus on activity and real action on a real project. Non-passive learning cycles, and hands-on work with the new technologies and new methodologies can cure your skills debt, and then you can start paying down your technical debt.

  • Why Your SaaS Ecosystem Isn’t Delivering

    Why Your SaaS Ecosystem Isn’t Delivering

    Cloud computing and -as-a-service deliverables such as SaaS have the potential to be transformational, but their value depends on whether you see them as tactical, single-purpose tools or as part of a larger and more strategically integrated ecosystem.

    The cloud today is embraced almost universally at some level and seen as an essential component of nearly any enterprise, making it easier to enter new markets, transcend physical barriers and gain access to vital services. The most common, basic approach to SaaS may best be described as a JBOS approach, or “Just a Bunch of SaaS.” In this model, SaaS is used throughout the enterprise, but only as single-purpose tools, deployed to meet the needs of a particular siloed requirement. SaaS in that respect is a quick fix, a Band-Aid, lipstick on a pig—and it has not reached its full potential.

    There is a point however, where SaaS and DevOps intersect—that point where an integration framework is superimposed over a SaaS catalog, and enterprise users now gain the advantage of each SaaS being part of a greater whole. At this point where SaaS apps and data become integrated strategically, we begin to see collaboration between IT professionals, developers and line-of-business managers; we see a higher level of automation; a more agile solution that takes full advantage of the cloud model; and finally, an overall cultural shift—all components of the DevOps movement.

    SaaS’ Potential

    The potential of SaaS is well-known and includes lower costs, agile sourcing and deployment, lower CapEx and more functionality to a wider group of users. But, the greater potential has been lost on many corporate users. IT is moving to the cloud, but is it doing so for the right reasons? By focusing on the “quick fix” aspect of SaaS, companies are missing out on the bigger picture. A more successful SaaS strategy defines the future of the business, using SaaS to stay agile and to align with business needs not just the individual SaaS instances but the entire SaaS ecosystem.

    The greater potential of SaaS is achieved when it is deployed strategically, with an integration framework—and then the benefits go far beyond lower costs and CapEx. At that point, SaaS becomes transformational.

    Unexpected Results

    With all the potential, the rapid uptake of SaaS throughout the enterprise has had a few unexpected results. In addition to a marked increase in siloed information and isolated functionality that results from a lack of integration, the “dark data” phenomenon looms on the horizon, especially as the Internet of Things creates millions of new data points that never before existed. IBM Research’s John Kelly, known as the “Father of Watson,” noted recently at the San Francisco Cognitive Colloquium that 80 percent of data remains dark, and by 2020, the number will reach 93 percent.

    With the vast majority of this unused data being generated from SaaS apps and IoT deployments, the unexpected result is an enormously untapped potential. Regardless of what the app’s intended purpose may be, who it was meant to benefit and which department deployed it, useful data is being orphaned at the source. In many cases, that data may have practical uses beyond the original intent, but those uses are never seen because it never escapes its own little corner of the enterprise.

    In addition, incumbent integration platforms do not lend themselves to the cloud and lack the SaaS adapters necessary for ad hoc integration and line-of-business usage, and the solutions that do exist still lack tools for governance, API management and IoT integration support.

    The Role of IT is Shifting

    The IT department’s role is shifting dramatically. The need for agility and flexibility, along with a growing dependence on the cloud and as-a-service software, means IT no can longer continue its centralized role, lest the benefits of the cloud be thwarted. In a recent survey conducted by NetEnrich, respondents’ greatest concerns included IT is losing its relevance (18 percent) and  business applications are crashing more because IT is not involved (22 percent).

    While businesses recognize the shift, still, 39 percent are facing difficulty integrating traditional IT, cloud and DevOps, and there is no DevOps team in more than half of the IT departments surveyed.

    The Solution

    With more than half of companies lacking a DevOps team to accommodate the integration of traditional IT and the cloud, the most pressing need is to continue re-evaluating IT’s role from one of build-and-deliver to one of facilitator. Use of an as-a-service integration framework that can be guided by and facilitated by IT but used by ad hoc integrators or developers—whose specialization is not integration but have a need to connect applications and systems quickly from time to time—is the obvious next step to resolving the issues of a SaaS ecosystem that isn’t living up to its potential.

    Such a framework can be deployed effectively using a bimodal, do-it-yourself integration approach. Those integration platforms that have begun to emerge may still be inadequate to accommodate the pervasive requirements of this major shift, since they still require IT specialists to manage the integration. The use of integration platform as a service (iPaaS) overcomes this limitation by providing a framework that empowers non-specialist IT users. Specifically, IT’s role as facilitator of this self-service platform comes into play by creating and maintaining the integration platform and a library of connectors, which then can be used not only by the integration specialists, but also by ad hoc integrators and “citizen integrators” or power users within the lines of business. As a result, the inefficient priesthood of IT is then replaced with a new balance of power between IT specialists, IT non-specialists and line-of-business users, encouraging more flexibility of integration in response to different priorities.

    The iPaaS approach brings a new level of efficiency to the many new projects that are generating storms of data, such as IoT, and it supports a move away from the older IT approach that lacks the speed and agility required in today’s environment. To make the best use of SaaS, it’s time to rethink our approach and, through universal integration platforms, bring it out into the open.

    About the Author/Renat Zubairov

    Renat photoRenat Zubairov is CEO and co-founder of elastic.io, a leading innovator in as-a-service integration.

  • Defining What DevOps is Not

    Defining What DevOps is Not

    It’s not often that my emails turn into full-blown conversations on DevOps. But when it happens, it’s a great time to take advantage of the insights. John Purrier of Automic and I recently discussed DevOps sustainability. Both of us think that sustaining a modern delivery chain and adopting it are two very different things. However, most only consider the latter. And because of that, in two years (or much sooner), they will find themselves in the exact same place they started—in an environment that is unable to adapt, rooted in habit and suddenly the old guy on the block.

    (more…)

  • DevOps: Metrics Do Matter

    DevOps: Metrics Do Matter

    One of my favorite quotes about metrics or KPIs is from Jim Fowler, CIO of GE Capital: “Don’t measure me on cost or traditional IT metrics, but on the metrics of the business.”

    It’s a perfect tee-up for me to talk about metrics in the context of DevOps.

    DevOps is the movement of enabling a business via technology or the ability to look at business processes and issues to mitigate them or achieve a goal.

    The Importance of Metrics

    DevOps has as one if its five major principles the concept of measuring (CLAMS: culture, lean, automation, measurement and sharing). DevOps helps the business understand how it uses and benefits from its second major resource after people: technology. This is where key performance indicators (KPIs) come in, though hopefully not KPIs such as this:

    a

    This specific KPI was created to show the productivity of developers, using lines of code written as the measure. Could you use or explain this? It’s a great example of metrics or KPIs that are based on bygone times of being complicated, disconnected from business goals, or driving the wrong behaviors by placing quantity ahead of quality.

    Catch the Ball

    DevOps enables everyone to work together toward a common goal. The DevOps lifecycle approach of creating a strategy, figuring out how to do it, introducing those steps, checking the results and, via feedback (people or tools), improving that strategy or product aids in improving the use and proving the benefit of IT.

    DevOps can use a trick from lean (Hoshin Kanri), where the strategy from senior management is passed down to each level (“catchball”) such that each team can create its own way of supporting that strategy. Hoshin Kanri allows every person to set their own way of working to meet the goal. How DevOps is that?

    b

    By permission Steve Bell: LeanIT @All rights reserved worldwide

    KPIs should be designed to help answer these questions:

    • Did we improve or decline?
    • Are we on course?
    • Does the impact effect our organization?
    • Do we need help?

    KPIs provoke a Yes/No response and force an immediate action/decision. If not, they are simply metrics that probably support a KPI but do not key to help fulfill a strategy or request.

    How Do You Create a KPI?

    First, look at a business target. Can you see what’s required to help achieve the goal or remove the obstacle to achieving the goal? Then could you determine how you would know this was happening and whether you could you prove it? All answers have to be Yes/No.

    I like making a tree, which also acts as a dashboard to quickly show at the various levels the indicators of performance and their status.

    c

    This helps make KPIs such as Cost of Delay, Mean Time between Repair, Defects2Incidents and others relevant to the right people at the right time. You could even be brave and flash the status using red and green: Red = we have a problem and need help to overcome or improve and green = all ok. Don’t hide behind amber—see how using just two colors helps guide easier, faster, better and safer.

    KPIs should be collaboration and communication tools. Create and use them wisely to keep your organization in business and on course.

    About the Author/Daniel Breston

    dDaniel Breston is the Chief of DevOps Transformation for Ranger4. Daniel has 30+years leading IT Service, Operations, Applications, Data Center and service partners in management of technology enabling business processes and functions. He has also spent 15 years of consulting and coaching leadership teams across the EU and UK in developing their people, processes and use of technology to deliver technology services better, faster, safer. Daniel has qualifications in IT Service Management (ITIL, COBIT,etc.), Lean and DevOPS.