devopsy.kinsta.cloud

Category: Blogs

  • Why NoOps is a DevOps disaster waiting to happen

    Why NoOps is a DevOps disaster waiting to happen

    Ever since the first hunks of Big Iron hit the streets, we’ve been struggling (and failing) to keep up with the demand for software applications.

    Now it’s getting harder. Not only do we have to satisfy an insatiable appetite for new apps and services, but the nature of the business palate is changing. Business now demands a continuous flow of software innovation over back-office application support – or using the haute cuisine analogy – they want IT to serve a classy degustation menu versus just warming up a ready-made TV dinner.

    All this bodes well for the fast iterative style agile, with fully autonomous development teams empowered to increase the rate of software deployments.

    Sound great, but what does this mean for IT Ops?

    Many would argue that IT Ops is a completely defunct or redundant function. Even credible analysts and thought leaders have touted NoOps (No IT Operations) as a viable option. Some even suggesting that it accelerates DevOps collaboration since by removing the Ops function you remove all friction.

    There appears some validity to these arguments. After all, if developers can codify operational functions into their activities, why do we need the disciple?  Perhaps this logic explains why for some, IT Operations as a function is becoming disenfranchised; no longer having a voice into critical digital transformation initiatives – or if it does, just the sound of “No” – the last deployment bottleneck.

    The death of IT Ops has been greatly exaggerated

    All this NoOps thinking is flawed for a couple of reasons.

    Firstly, however operationally awesome developers think they are, building resilience, maintainability and supportability is not always top-of-mind. Worse still, these elements might be neglected if management is fixated on rewarding developers according to ‘speeds and feeds’. Secondly, even if these elements are addressed, they’re often conducted at the end of development cycles or bolted on after problems are discovered – that’s like baking a cake, forgetting the sugar, and then trying to compensate with a sickly sweet chocolate sauce.

    In reality, great operations engineers are best equipped to help incorporate operational excellence into all practices. After all they have years of experience supporting every new wave of technology – from Mainframes to Microservices. What must change, however, is how their expertise is developed and shared.

    • Become cloud connoisseurs – Like our development colleagues, operations’ emphasis will shift from on-premise to cloud. This means ensuring Ops expertise is fully leveraged to support the business nirvana of delivering high-quality digital customer engagement at scale. So if you’re an IT operations professional still configuring QoS policies on routers or manually provisioning development environments, it’s probably time to skill-up in PaaS, Amazon EC2 and containers. Either that or go and find a gig with a cloud service provider.
    • Embed Ops In Dev – This means being less separatist and silo’d and more inclusive and unified. Rather than focus on technical diagnostics and reacting to failures, the new operations professional will apply systems thinking to holistically look at how business applications and the all-important customer experience can be improved. This is easier said than done with skeptical development teams, so it’s essential that monitoring feedback is automatically shared and incorporated into all practices – like for example, agile sprints
    • Embrace Lean thinking – this involves being less interrupt-driven – fixing one technical problem after another, to working side-by-side with other teams to ensure a constant flow of value is delivered to business with all waste removed. As with our development colleagues, elements of agile (and common sense) thinking will come to the fore – with terms like “never done” and continuous improvement becoming established elements of the new operations and performance management mantra.

    Perhaps the biggest change IT Operations will undergo is how it interfaces with other teams. Here, IT Ops will morph into providing sets of easily accessible and repeatable processes other teams will use to bake quality into everything developed and tested. To this end, operations will transform from a separate “keeping the technical lights on” function, to a high-value to a craft that outfits the software factory and staff with new capabilities they need to ensure speed and quality at scale.

    The new age of Agile Operations “craftsmen”

    For IT Operations this means letting go of many traditional admin tasks – not just running things, but participating in crafting new automated processes. So rather than routinely monitoring applications in production, the Ops in DevOps will ensure this capability is available in pre-production. Rather than mundanely provisioning servers from change requests, new teams will equip the organizations with a complete release environments that support program level goals.

    Operating as DevOps “craftsmen” makes complete sense because it moves expertise out from behind the production curtain. The organizational focus of IT operations positively changes too — from being technically good at describing and fixing problems, to being awesome at prescribing improvements that drive better business outcomes.

    Great organizations exploit their business models with flawless operational execution. With these models now being constantly re-shaped by software applications, IT Operations is more important than ever. Relegating it with NoOps thinking isn’t an option.

     

     

     

     

     

     

  • Version your infrastructure

    Version your infrastructure

    Infrastructure is code, and like any code, you need to version it. Notice how that works:  If it’s infrastructure, then it’s a fundamental part of the system; nothing’s going to work without it, which makes it valuable. So valuable that in full-stack deployments, it is part of the application. And if it’s code, then it should be under version control.

    (more…)

  • APM on-premises or SaaS: What’s the best fit?

    APM on-premises or SaaS: What’s the best fit?

    Today’s DevOps teams need in-depth monitoring that offers insights into every aspect of the application-delivery chain. When you decide for an application performance monitoring (APM) solution, you can choose between an on-premises approach or SaaS-based solutions (“Monitoring as a Service”). Every deployment option brings its pros and cons or has advantages for special use cases. An installation on-premises for example works best if you want to monitor an intranet that you want to keep isolated.

    Which APM deployment option makes most sense for your organization? There‘s no simple answer to this question. A lot depends on your environment and requirements. Let’s explore the points to consider in deciding between SaaS and on-premises monitoring:

    Server location

    With in-house, on-premises systems, all servers and data are physically under your control. When you use SaaS-based monitoring, performance metrics, reliability data, and detected issues are stored in the cloud. If your industry has strong regulatory and policy requirements, SaaS may not be an option for your organization—you may need to keep all such monitoring data on-premises.

    Implementation

    To deploy on-premises solutions you need to provide your own hardware and storage in a private datacenter or cloud. You install and configure the monitoring software, databases, and other components of the APM solution. In general, SaaS offerings can be installed and set up within minutes using a self-service system. Some SaaS APM tools even auto-detect your entire server landscape within minutes following installation.

    Software updates

    SaaS solutions are typically updated automatically on a frequent basis by the vendors who provide them to deliver support for new technologies and feature enhancements. On-premises installations require that you manage updates yourself and stay current with the release of new product updates. Updates for on-premises solutions typically come out less frequently, which means you sometimes have to wait longer for support of new technologies and bug fixes.

    Server and database backups

    Maintaining your own servers for an on-premises APM solution requires that you manage the servers, data storage, and data backups yourself. SaaS solutions are maintained and backed up by their vendors.

    Costs

    On-premises software is usually licensed for longer periods of time. This means that the initial investment is often higher, but costs are recouped over time. SaaS-based APM often features “pay as you go” pricing, so you only pay for what you use. This model can be cheaper, but it depends heavily on your usage and your tool vendor’s pricing model. If you run extra servers during periods of peak demand, it makes a big difference if your billing is based on the maximum number of servers you use rather than your actual usage. Because they’re built on cloud infrastructure and often don’t require long-term contracts, SaaS make it much easier for you to opt out of the system and cancel any time. As you aren’t required to purchase and maintain your own hardware, they’re also typically more flexible and easier to scale.

    Support

    When your monitoring solution runs on-premises it’s often harder to get vendor support because the system may not be reachable from outside your organization. With SaaS-based solutions you can grant access to your tool vendor’s support staff and work with them on resolving issues. This makes the SaaS approach faster and easier if you run into special cases that require vendor support.

    Mobility

    With on-premises solutions you must connect to your internal network via VPN to view monitoring data on mobile devices (unless your organization exposes this service to the Internet). Conversely, you can access monitoring data supplied from a SaaS-based solution anywhere you have an Internet connection.

    Customization

    With an on-premises APM solution, you can expect in-depth customizations that suit the exact requirements of your organization. Customization of SaaS solutions on the other hand is limited, and dictated by rules defined by your tool vendor.

    Distributed applications

    If you need to monitor applications that run in a public or hybrid cloud scenario, it’s likely that you’ll also want to run your monitoring in the cloud. SaaS-based solutions enable you to seamlessly monitor cloud-based environments because this is what they’re designed to do. You only need to install agent software on the involved servers.

    (No) single point of failure

    On-premises solutions are vulnerable to critical problems like power outages and broken network connections, potentially leaving you with no way of receiving performance and availability problem alerts. On the other hand, SaaS monitoring in the cloud runs in a different geographical location and so remains available to send you alerts whenever problems are detected.

    Reliability and redundancy

    When your APM monitoring solution runs in your own data center, your IT department can determine appropriate service availability levels. SaaS monitoring providers often use software architectures that can be updated without downtime and reach service availability levels of up to 100%. Well-designed SaaS solutions use redundancy to further improve availability. It’s recommended that you take a close look at any SaaS-based solution vendor’s claim of providing “always on” availability.

    Scalability

    SaaS tool vendors ensure that needed resources like disk space and CPU will scale up with your demand. This means you won’t need to budget for additional hardware purchases and typically leads to lower total cost of ownership (TCO) with cloud-based SaaS solutions.

    Conclusion

    Deploying monitoring software in house (on-premises) gives you full control over your data. “Monitoring as a Service” in the cloud brings new possibilities for many monitoring use cases. Depending on your requirements, it could even make sense to use both. Some tool vendors are now providing monitoring solutions that combine the best of both worlds.

    With the points covered in this article, you should now know everything you need to consider when deciding which monitoring approach makes most sense for your organization.

  • DevOps Impact on IT Operations Management

    DevOps Impact on IT Operations Management

    Most of the focus of the DevOps movement is on the application development to deploy cycle, and that’s appropriate.  When folks talk about the IT Ops side of DevOps, you’ll hear talk about IT process management tools like Puppet, Chef, and Ansible that are used to automate the deployment, update and remediation of software and VMs.  Yet, DevOps is widely acknowledged to be more than just one process—it is a culture of collaboration, enabled by automation that results in high quality outcomes.  As such, more IT disciplines are looking at how to apply DevOps principles to their domain.  These domains include those that can be put under the umbrella of IT operations management (ITOM): network engineering and operations, disaster recovery, and infrastructure operations management.  Many of these disciplines and organizations are currently highly siloed in traditional IT organizations, and tool suites are from an era that precedes the current emphasis on API automation.  Some may wonder whether these IT arenas can adapt and adopt DevOps.  The more pertinent question may be “can IT organizations afford not to?”

    ITOM is a huge topic, and there’s no way to do an exhaustive treatment of how DevOps is impacting it in one article.  However, we can at least take a look from a couple of angles.  The first angle is to consider DevOps impact in terms of functional areas of operational management.  ITOM can be roughly broken down into managing provisioning, capacity, availability and performance.  Of these, provisioning is probably the area where DevOps has had the most impact.  This only makes sense, since without automated provisioning of infrastructure and application components, it would be very hard to achieve the continuous processes that are a hallmark practice of DevOps teams. The trend of all IT infrastructure to become “software-defined” aligns with this provisioning automation movement.

    Once we stray outside of provisioning, the reach of DevOps becomes more sparse.  In the win column for DevOps is Application Performance Management (APM).   Five years ago, there was a tremendous dissatisfaction level felt by engineers trying to understand application performance.  So much so that a #monitoringsucks meme developed.  Since then, APM has significantly modernized and moved profoundly towards DevOps-friendly approaches.  Open Source Software (OSS) options such as the ELK stack (ElasticSearch + Logstash + Kibana) are seeing increasing adoption.  Meanwhile, commercial SaaS offerings from New Relic, AppDynamics, BMC and Riverbed are offering nextgen, highly API-enabled options.  Note that even though APM is considered a category of ITOM, it’s certainly the most dev-centric.

    Infrastructure monitoring has lagged by comparison.   The typical operations center arranges itself in a multi-level skills hierarchy.  Practices are shaped around monolithic, GUI-driven tools, and manual processes are the norm.  In fact, it’s fair to say that much of the institutional knowledge in many ITOM practices is literally “muscle memory.”  That is to say, knowing where to click on the screen.  Stovepipes of these hierarchies are formed around different types of infrastructure–compute, storage and network.  Collaboration between stovepipes is the exception, and ‘hot potato’ troubleshooting routines are the rule.  Rather than emanating from organizational DNA, collaboration often requires top-down pressure, usually to respond to crises. On the tools side, monolithic, enterprise software deployments and appliance-based architectures are common.  These products have been primarily designed to deliver a GUI experience.  Their APIs are usually second or third class citizens from the vendor point of view, and aren’t meant for end-users.  Rather, APIs primarily exist for vendor to vendor partnership integration and high cost custom integration projects. Clearly, this picture is not DevOps nirvana—so how is DevOps impacting it?  The answer is, so far, incrementally.

    One of the clear signs that an IT arena is getting primed for DevOps is cloud and open source disruption, and by disruption, I mean going beyond narrow, point tools to general purpose platforms.   Those same open source tools used for APM are making their way into systems management, and to a lesser degree in network management.  We’re also starting to see “green shoots” of change in systems and network management tools, with new SaaS offerings, though we’re far from a tipping point in these segments.  However, an interesting challenge for true conversion to cloud-based DevOps approaches is the need to address a Big Data problem.  The amount of data generated by IT infrastructure and networking equipment is now so voluminous that traditional systems and approaches can’t keep pace.  Legacy systems that have their architectural provenance in the 1990’s are based on scale-up models, and were never built for today’s data volumes.  Even cloud-based solutions may not necessarily be scoped to address IT Big Data. Until ITOM faces this technology challenge broadly, it will remain an impediment to transformational change.

    About the Author/Alex Henthorn-Iwane

    AlexAlex Henthorn-Iwane is the Vice President of Marketing at Kentik . Henthorn-Iwane has more than 20 years of experience bringing new technologies in networking, security and software to global markets. Henthorn-Iwane leads the global marketing strategy for Kentik and help the company bring its innovative story and solutions to network-dependent organizations around the world.

    Prior to joining Kentik, Henthorn-Iwane was VP of Marketing at QualiSystems, where he oversaw worldwide marketing at the industry’s leading supplier of DevOps cloud orchestration software for hybrid IT infrastructure, and Packet Design Inc., a provider of network management software founded by serial entrepreneur Judy Estrin. Prior to that, he held product management roles at CoSine Communications, Corona Networks, Lucent Technologies and Livingston Enterprises.

    Through his roles at QualiSystems, Packet Design, CoSine Communications, Corona Networks and Lucent Technologies he has acquired expertise in cloud computing and the opportunities presented through virtualization. He has written for Embedded Computing, Virtual Strategy Magazine, Datamation, SDN Central, Datacenter Knowledge and InformationWeek.

    Connect with him on LinkedIn, Twitter, or SlideShare.

  • For more mobile wins, enterprises must find their MVPs

    For more mobile wins, enterprises must find their MVPs

    The demand for enterprise mobile apps just isn’t letting up. The research firm Gartner expects that by the end of 2017 enterprise user thirst for mobile apps will be growing five times more quickly than internal IT organizations can deliver those apps. And a recent Opinion Matters survey found that enterprises are already hitting deep backlogs, with 85% of enterprises surveyed reporting that they already have a significant mobile app development backlog.

    Such a backlog, as we’ve covered earlier, suffocates an enterprise’s ability to innovate and be competitive. When enterprises take too long to give development teams the app specs they need, and those apps take weeks and months to build and ship, the enterprise just lumbers along in its digital efforts. That means worker productivity lags, too. Most of the time customers aren’t happy with their experience, either.

    Why are enterprises having difficulty fulfilling their app demand? There are a number of reasons, notably that enterprise development teams often work the way things were accomplished a decade ago when IT was more centralized. First, the app needs approval from IT and then timelines are set. And the app needs to be prioritized along with everything else on development team’s plate. When apps are approved, everyone tends to try to shove his or her pet features into them and the apps end up trying to achieve too much right out of the gate.

    Enter the MVP

    The result of all of this, of course, is that the time between the app request and its delivery is extended considerably, and when the app finally does arrive – if it doesn’t meet the required needs or is less than optimally executed – a big and time-consuming effort turns into a big and costly setback. It’s one of the reasons why so many enterprise mobile apps fail, with 50 percent of respondents to a recent survey saying that their mobile apps failed because of poor user experience.

    It’s also one of the reasons why more enterprises are embracing the Minimum Viable Product, or MVP, as a way to get try to get apps with the features users need as quickly as possible, while not investing too much time in apps that users don’t desire. “In the strictest definition of MVP, it refers to a product that delivers the highest ROI relative to risk,” says Scott Amyx, CEO at Amyx McKinsey, a smart wearables and Internet of Things strategy and development agency.

    What does that mean, practically, to CIOs and development teams trying to develop more apps that are more on target to user needs? Asaf Yigal, cofounder and VP of product at data analytics platform provider Logz.io, says it means that developers should start with a minimalist, but useful, definition of the app that is to be created.

    At Logz.io, Yigal explains, the company develops with an agile methodology. “It’s critical for us to be able to define the MVP before we start a project,” he says. “There are two key objectives when defining a MVP. The first objective is that the majority of the users need to be able to benefit from a new feature – this means that we don’t address edge cases in development at all. The second objective is that we need to be able to analyze how users will be using the new feature. We are a data-driven company, which means that we log all relevant events in every feature we develop and build internal dashboards that display how users are interacting with a given feature,” he says.

    Small teams, small apps, tight focus

    Greg Archbald, founder & CEO of oil and gas industry software provider GreaseBook, has worked with dozens of oil and gas companies, ranging in size from those with tens of billions in revenue to those with less than one billion in revenue. Archbald explains that while the “super majors” of the industry have plenty of success extracting hydrocarbons, they don’t necessarily excel in building software. “Even the companies that have managed to land themselves some really strong development talent tend to struggle with project scope,” says Archbald.

    Why? It’s the sheer number of people that become involved in the development process, he explains: ”I’ve seen first hand how layers of management, engineers, and production supervisors (all with the best intentions) become involved, and this is where issues begin. Everyone needs the app to do something different, and pretty soon feature creep settles in.”

    This is especially true for development teams that find building systems easy, but dealing with people more difficult. “For much of software to be successful, it must focus on the human elements, not the technical. And, most larger companies don’t realize this soon enough and ultimately set themselves up for failure,” Archbald says.

    So who’s going to keep app development on track? It could start with learning when to say “no,” Archbald says. “This is very tricky because, as part of a development team working for a large company, you’re paid to develop, not to tell upper-management “no,” he says. But to be able to succeed with the MVP and clearing app backlog, it’s about clearing both unnecessary people and features. “The word ‘no’ is exactly what MVP stands for. And at larger companies, a successful MVP isn’t just about a limited set of features, but also about a limited number of people involved.

    “It sounds counterintuitive, but many projects in the development succeed not in spite of the limitations but because the limitations were put on them. Limitations of the amount of input, manpower, beta-testers, and even funding – all of which are unheard of at the enterprise level – force a team to figure out more creative solutions.”

    Jordhy Ledesma, CEO at creative design, information architecture, and custom programming consultancy Information Providers would likely agree. “Keep the development team to less than a handful of teammates. Develop using the SCRUM project management framework (which advises to code in sprints and ship functionalities in batches) and limit the project duration to less than six months,” says Ledesma. “What is sometimes misconstrued by the DevOps community is that MVP is about a tangible, minimal product more than a methodology to leverage. In other words, it’s an iterative process that helps us to refine our concepts, prototypes, data, analyses, and learnings to deliver high value to our target customers. It’s a framework.”

    It’s not just about enterprise software; as more things and devices become interconnected, it’s going to be even more important that enterprises reduce risk with apps. “Our agencies work with both large enterprises and startups. When they come to us to conceptualize and develop wearables and IoT products and services, they almost always start with an aspirational vision. Sometimes, that means a laundry list of sensor technologies or data analytics capabilities,” explains Amyx.

    “We work with them to understand, from the customer perspective, what is the biggest pain point and then to focus on delivering a focused solution that addresses the top problem statement. After we learn from an iteration, we do it again to incorporate customer feedback into the product roadmap. A tight MVP methodology turns a potentially daunting process into doable mini-sprints while helping our clients to focus. They are often surprised at the nth iteration how their product/service is vastly different from what they had originally envisioned,” Amyx says.

  • Librato Seeks to Unlock DevOps Application Monitoring

    Librato Seeks to Unlock DevOps Application Monitoring

    As wider adoption of DevOps methodologies continues to foster demand for new tooling to help streamline evolving workflows, at least one vendor is attempting to address the growing need for related monitoring capabilities.

    Librato, a San Francisco-based provider of Amazon AWS applications performance monitoring (APM) services, claims that it is finding favor among the rapidly expanding DevOps community driven by its ability to inject critical visibility into the engineering lifecycle.

    Acquired by Austin, Texas-based SolarWinds in January for a lofty $40 million and renamed Librato SolarWinds Cloud, the company is touting its recent adoption by well-known providers such as Slack for monitoring of DevOps-centric applications performance and health metrics.

    “At its core, DevOps is destroying conventional wisdom around ritualized processes and the need to perform them, mostly in the domain of scale,” said Dave Josephsen, Developer Evangelist at Librato. “By attempting to include monitoring as a key element of the development lifecycle, this has created a whole new range of challenges, and opportunities.”

    By providing a centralized Web interface and open API to help create visibility into critical applications engineering requirements, Librato maintains that it is creating the tooling necessary to empower integrated development and deployment processes, such as continuous change control.

    “There are a lot of disparate tools for monitoring individual engineering tasks, but there hasn’t been any practical way to bubble up that data for efficient monitoring,” Josephsen said. “Within the context of DevOps, change control is a ritual that we’ve abandoned as we’ve embraced test driven development and automated away the human element, so there’s a distinct need for monitoring of feedback and metrics.”

    For example, cloud-based productivity messaging specialists Slack are using the Librato “StatsD” lightweight daemon process to stream business and app-health metrics regarding PHP applications into the monitoring platform. Slack also leverages Librato to chart data regarding connect, query and fetch latencies for its server clusters, and to monitor internal metrics about overall system health.

    “Having those graphs side-by-side lets us clearly see and compare the effects of network latency on one of our new services, and that’s very cool,” Richard Crowley, director of operations at Slack, wrote in a recent testimonial posted to the Librato Solarwinds blog.

    Industry experts agree that broader adoption of DevOps strategies will continue to drive demand for relevant monitoring tools, with solutions such as Librato seeking to deliver “modern stack architecture for looking at the modern stack.”

    “I think that APM is always interesting because it’s always new,” said James Governor, industry analyst with RedMonk. “The pace with which we’re building new architectures to deliver services to end users is increasing all the time, and with every new wave of technology, we tend to get new and associated models of APM tooling.”

    Librato is taking advantage of the need for a system that brings together the elements necessary for cloud native monitoring, and in an agentless manner different from other competing tools such as New Relic, the analyst noted.

    However, whether or not the platform is able to break down traditional engineering siloes to enable more efficient DevOps management, as Librato officials claim, remains to be seen, he said.

    “Each wave of technology has promised to do this, to remove the inherent finger pointing, and each has always made the same claim to break down organizational firewalls,” Governor said. “DevOps hasn’t boiled the ocean in this manner yet, but there are some unique opportunities; unlike previous waves, where automation was intended to replace humans, this is now about augmenting people, and we are seeing that teams are choosing new tools from which developers can truly derive value.”

  • 9 Open Source DevOps Tools We Love

    9 Open Source DevOps Tools We Love

    While we have been building our SaaS company, we have been investing much time in researching and deciding which tools to include in our DevOps toolkit. We’ve based these decisions on our years of experience in the IT industry, dealing with infrastructure for the most part. From building a petabyte-scale, data analytics infrastructure, our architecture, tools, and processes have become key components of our technology and operations. We’ve taken great care in selecting, benchmarking and constantly improving our tool selection.

    As a company that is building a solution on top of an open-source stack (ELK), our team is heavily involved in the open source community, contributing to multiple projects such as Camel and Kafka while customizing tools to fit our needs. The vast majority of tools we use internally are open-source ones. By sharing the toolset that we’ve collected and honed over time, we hope to foster a discussion within the DevOps community on what further improvements can be made.

    With that, we welcome you to enjoy browsing through the following list that we’ve created. Some tools you may have known about for years, while others may be new. Of course, we invite any and all feedback, especially from those of you who are using alternatives!

    Must-Have DevOps Tools

    1. Nagios (& Icinga) 

    nagiosInfrastructure monitoring is a field that has so many solutions… from Zabbix to Nagios to dozens of other open-source tools. Despite the fact that there are now much newer kids on the block, Nagios is a veteran monitoring solution that is highly effective because of the large community of contributors who create plugins for the tool. Nagios does not include all the abilities that we had wanted around the automatic discovery of new instances and services, so we had to work around these issues with the community’s plugins. Fortunately, it wasn’t too hard, and Nagios works great.

    We also looked into Icinga, which was originally created as a fork of Nagios. Its creators aim to take Nagois to the next level with new features and a modern user experience. There is a debate within the open source community about the merits of Nagios and its stepchild, but for now we are continuing to use Nagios and are satisfied with its scale and performance. The switch to newer technology, such as Icinga, may be appropriate in the future as we progress.

    2. Monit

    monitSometimes the simplest tools are the most useful, as proven by the simple watchdog Monit. Its role is to ensure that any given process on a machine is up and running appropriately. For example, a failure occurs in Apache, Monit will help to restart the Apache process. It is very easy to setup and configure and is especially useful for multi-service architecture with hundreds of micro-services. If you are using Monit, make sure to monitor the restarts that it executes in order to surface problems and implement solutions (rather than just restarting and ignoring the failure). You can do this by monitoring Monit’s log files and ensuring that you are alerted to every restart.

    3. ELK – Elasticsearch, Logstash, Kibana – via Logz.io

    logz.ioThe ELK Stack is the most common log analytics solution in the modern IT world. It collects logs from all services, applications, networks, tools, servers, and more in an environment into a single, centralized location for processing and analysis. We use it for analytical purposes (e.g., to troubleshoot problems, monitor services, and reduce the time it takes to solve operational issues). Another use for this tool is for security and auditing (e.g., to monitor changes in security groups and changes in permissions). After receiving alerts on these issues, it is easy to act on unauthorized users and activities. We also use ELK for business intelligence, such as monitoring our users and their behavior. You can set up your own ELK or buy it as-a-service. We’ve written a guide for the community on using ELK to monitor your application performance.

    Disclaimer: Logz.io is our ELK-as-a-service that we use in our own environment. You can say that we eat our own dog food.

    4. Consul.io

    consulConsul is a great fit for service discovery and configuration in modern, elastic applications that are built from microservices. The open-source tool makes use of the latest technology in providing internal DNS names for services. It acts as a kind of broker to help you sign and register names, enabling you to access service names instead of specific machines. If, for example, you have a cluster of multiple machines, you can simply register them as a single entity under Consul and access the cluster easily. We praise this tool for its efficiency, although we still feel there is more that can be done with it. If you also use it, it would be great to hear about your own use case.

    5. Jenkins

    jenkinsEveryone knows Jenkins, right? It’s not the fastest or the fanciest, but it’s really easy to start to use and it has a great ecosystem of plugins and add-ons. It is also optimized for easy customization. We have configured Jenkins to build code, create Docker containers (see the next item), run tons of tests, and push to staging/production. It’s a great tool, but there are some issues regarding scaling and performance (which isn’t so unusual). We’ve explored other cool solutions such as Travis and CircleCI, which are both hosted solutions that don’t require any maintenance on our side. For now, however, since we’ve already invested in Jenkins, we’ll continue with it.

    6. Docker

    dockerEverything that can be said about how Docker is transforming IT environments has already been said. It’s great…life changing, even — (although we’re still experiencing some challenges with it). We use Docker in production for most services. It eases configuration management, control issues, and scaling by allowing containers to be moved from one place to another.

    We have developed our SaaS solution with a twelve-layer pipeline of data processing. Together with Jenkins and Docker, we have been able to run a full pipeline across all layers on a single Mac. It would be wrong to say that there aren’t any complications with Docker, as even small containers can take a significant amount of time to build. However, we want to ensure that our developers are as satisfied as possible and enable them to work rapidly. With all of the management involved in storage, security, networking — and everything surrounding containers — this can be a challenge.

    We see Docker progressing and look forward to welcoming the company’s new management and orchestration solutions. For those who might be having issues with Docker, we’ve also compiled a list of challenges and solutions when migrating to Docker.

    7. Ansible

    ansibleAgain, simplicity is key. Ansible is a configuration management tool that is similar to Puppet and Chef.  Personally, we found those two to have more overhead and complexity to our use case– so we decided to go with Ansible instead. We know that Puppet and Chef probably have a richer feature set, but simplicity was our desired KPI here. We see some tradeoffs between configuration management using Ansible and the option to simply kill and spin new application instances using a Docker container. With Docker, we almost never upgrade machines but opt to spin new machines instead, which reduces the need to upgrade our EC2 cloud instances. Ansible is used mostly for deployment configuration mostly. We use it to push changes and re-configure newly-deployed machines. In addition, its ecosystem is great, with an easy option to write custom applications.

    8. Collectd/Collectl

    Collectd/l are nifty little tools that gather and store statistics about the system on which they run and are much more flexible than other tools. They allow users to measure the values of multiple system metrics and unlike other log collection tools that are designed to measure specific system parameters, Collectd/l can monitor different parameters in parallel. We use these two tools to measure customer performance parameters and ship them to our ELK-as-a-Service platform. We’ve specifically wrapped a Collectl agent in a Docker container and push it with Ansible to all of our servers. It collects information every couple of seconds and then ships it to ELK to allow us to run reports and send alerts. If you’d like to see a specific example of how we do this process in our environment and how others can do the same, we’ve created a guide for everyone.

    9. Git (GitHub)

    githubGit was created 10 years ago following the Linux community’s need for SCM (Source Control Management) software that could support distributed systems. Git is probably the most common source management tool available today. After running Git internally for a short period of time, we realized that we were better suited with GitHub. In addition to its great forking and pull request features, GitHub also has plugins that can connect with Jenkins to facilitate integration and deployment. I assume that mentioning Git to modern IT teams is not breaking news, but I decided to add to it to the list due to its great value to us.

    Other picks?

    The modern DevOps world is full of outstanding and unique open source tools—it’s a jungle out there. We found the tools listed here to be the best in breed and think they should be included in every DevOps engineer’s shortlist.

    Or am I wrong? What open source DevOps tools are in your toolkit? I’d love to hear your own recommendations and experiences in the comments below.

  • Ten ways DevOps will benefit your IT department

    Ten ways DevOps will benefit your IT department

    When implemented properly, DevOps can transform your software engineering process and create value for both employees and customers, producing business performance and value.

    A fully working DevOps methodology will bring the following benefits:

    1. Increased service quality

    Perceived service quality and reliability depend on both the availability of a service in the absence of error (MTTF: Mean-Time-To-Failure) and the ability to restore the service to an error-free-state (MTTR: Mean-Time-To-Repair).

    Due to fast feedback loops and high release velocity, service deficiencies are removed much faster than in the past (shortened MTTR), which leads to an improvement of perceived service quality and reliability.

    1. Increased reliability of service delivery

    The perceived timeliness of delivery is an extremely important satisfaction factor for your internal or external customers Usable results are delivered early and frequently, which is critical in uncertain and changing business environments.

    With agile and DevOps, big projects are cut into small chunks, which are then continuously delivered. Both initial and successive requirements and changes are delivered with short lead times, while changes are possible at any time in the process.

    1. Increased customer value through responsiveness to change

    To stay relevant, businesses need to constantly adopt to a rapidly changing market environment, respond to competitors and stay ahead in the innovation cycle. Your business needs to act in a world of uncertainty.

    This means that at the time a new idea is brought up, most of the project details cannot be known. Even once they are defined later on, they are subject to constant change. Complex upfront specifications – a necessary deliverable in the absence of short release cycles – often become obsolete.

    1. Better usability increases customer satisfaction

    Shorter release cycles lead to shorter feedback cycles. Your customers are receiving frequent and timely updates, which gives you the ability to evaluate and test their satisfaction and response early on.

    You can even test different versions of your product at the same time with different customer groups by A/B testing. This gives you the ability to compare different new features or capabilities of your products to each other.

    1. Efficient operations

    Agile and DevOps are built on lean principles. The reduction of waste, such as unnecessary waiting times or process overhead through continuous improvement, is a core principle.

    This not only helps agility, but also has a positive impact on costs by helps to limit their growth.

    1. Reduce bottlenecks through cross-pollination of skills

    Blurring the line between dev and ops involves each side learning about the other’s trade. This enables staff to understand the whole process end-to-end and see where it can be tweaked and improved.

    If ops know the basics of coding and vice-versa, bottlenecks can be removed by not having to rely on highly specialized individuals. Automation also helps here, for instance in the way a dev can use a service catalogue within ServiceNow to spin up environments on demand, without waiting for an ops member to have free time.

    1. Holistic value creation increases employee satisfaction

    In a conventional IT organization, both dev and ops are often limited to executing specialized tasks. The value these tasks create for customers is often difficult for customers to measure, as it is one small aspect of a complex process.

    Whether internal or external customers are served, the impact created by agile dev and ops teams is far more direct and holistic and can thus be more easily observed and “felt” by employees.

    1. Collaboration improves employee attitude

    Whereas previously dev would hand code over to ops then switch to the next project, DevOps encourages collaboration throughout the team along the entire process. Not just exchanging emails, but face-to-face and through collaborative devices such as Kanban boards and Scrum meetings.

    In the past, the two teams could have been working in different offices. DevOps brings IT together, and this shift in culture is likely to spread outside the department. This reduces waste through misaligned incentives causing political behaviour and mutual blame and increases employee satisfaction.

    1. Constant learning and improvement

    DevOps promotes a culture of continuous learning and improvement through lean principles. In a fast-moving and unpredictable environment, continuous improvement is the single most important source of sustained competitive advantage and superior economic value.

    An continuous learning environment also attracts and sustains outstanding talent. It is an important ingredient to employee satisfaction. 

    1. Reinvigorate your employees

    Apart from the new technical practices DevOps brings, perhaps one of the main advantages for IT is the shake-up it gives working lives. Like a new sports coach can bring a fresh perspective and increased work-rate amongst the team, so DevOps can reinvigorate software engineering teams.

    New people to interact with, perhaps even a new office and a new way of working – DevOps can be just what IT needs to breathe life into stagnating staff who have been following the same routine for far too long.

    **************************

    In order find out how to implement DevOps in your IT department and see these benefits for yourself, I recommend attending my free webinar on Tuesday 28th July. Click to find out more.

  • SaaS: Building Tools for the DevOps-Minded

    SaaS: Building Tools for the DevOps-Minded

    DevOps, Continuous Delivery, Continuous Integration and Continuous Testing are no longer just methodologies and development strategies — they’re all a part of a growing economy within the tech industry.

    Great Software-as-a-Service (SaaS) companies such as Chef, Docker and New Relic have done an amazing job of spotting the trends early and building their tools and services around a new wave of development processes. Chef and Docker alone have reportedly raised nearly $250 million, and New Relic went public in 2014.

    In fact, this very website is another example of how impactful these new development practices are — as several specialized news outlets have been born out of these revolutionary development strategies. So, not only has the DevOps methodology helped developers create better products, faster; but the process has opened opportunities for technical experts to share their experiences on sites like this, e-books, webinars, conferences and any other way people communicate.

    On DevOps.com, you’ll see the best articles on the web discussing how DevOps is defined in different organizations, why Fortune-500 companies are shifting old practices toward CD and the best practices of CI from industry experts.

    In the race to be the most DevOps-friendly solution — no matter the niche — hundreds of startups have been founded and funded, which has only enhanced the toolboxes of thousands of developers. You’d be hard-pressed to find a good SaaS company that doesn’t offer integration with Jenkins CI, TeamCity or Chef, and that’s because not only should a SaaS product solve an important problem, but it should have minimum disruption of your current development process.

    With all that in mind, Continuous Testing is one of the most exciting spaces in technology today. Continuous Testing is a process that aims to synchronize testing and QA with dev and ops, which ultimately yields better products more quickly.

    There are several factors that make Continuous Testing successful in a team’s software development lifecycle — such as open-source collaboration solutions, test automation APIs and tools to track the historic performance of applications .

    With the release of Load Impact Version 3.0, the global performance and load testing SaaS company has announced its latest feature that fits right in with the DevOps mindset: Performance trending.

    Performance trending allows users to plot runs of the same test over time. That gives companies a clear view into data that locates patterns of performance degradation or improvement in order to easily validate code and infrastructure changes.

    In the spirit of Continuous Testing, DevOps, CD and CI — performance trending improves users’ ability to gain both a low-level and high-level view into the performance of their systems. And with the increased number of tests run by DevOps professionals, performance trending yields valuable information for companies concerned with their application performance — which should be everyone.

    Between DevOps.com and plenty of great conferences and meetups around the world, it’s safe to say this economy of DevOps-minded tools will only continue to grow in the coming years.

  • OCP is a victory for CoreOS and Containers in general

    OCP is a victory for CoreOS and Containers in general

    A broad coalition of major tech companies have joined forces to create the Open Container Project—or OCP. The move to develop and maintain common standards around software containers vindicates the CoreOS split from Docker and reunites the two in a common cause that will make all container platforms and technologies stronger.

    A press release announcing the Open Container Project explains the overall vision for OCP. “Housed under the Linux Foundation, the OCP’s mission is to enable users and companies to continue to innovate and develop container-based solutions, with confidence that their pre-existing development efforts will be protected and without industry fragmentation.”

    Docker and CoreOS are at the heart of OCP but they’re by no means alone. A veritable “Who’s Who” of DevOps and container technologies has enlisted to support OCP, including Amazon Web Services, Cisco, EMC, Google, HP, IBM, Linux Foundation, Microsoft, Red Hat, and VMware.

    At one point Docker had a Standard Container Manifesto that spelled out some basic attributes it felt all containers should have. The Standard Container Manifesto called for containers to operate in a standard fashion, be content agnostic, infrastructure agnostic, and designed for automation among other things. When Docker removed the manifesto and seemed to shift to a more proprietary strategy CoreOS split off and launched its own container technology to continue the vision of a container standard.

    “An open, well-defined specification is required for the overall success and adoption of containers. At CoreOS, when we started the App Container spec (appc) our goal was to have a well-designed software container specification. With the OCP, we have now come together with Docker and many industry giants in an effort to unify this goal.”

    The launch of OCP and the laundry list of companies backing it show that CoreOS was not alone in its pursuit of the vision for a container standard. “We are excited to support this effort to bring the industry together to agree on a core standard for containers, which we feel will benefit all customers no matter what cloud provider they use,” said Jason Zander, CVP, Microsoft Azure. “Through our close work with Docker over the last year we have demonstrated our commitment to openness and welcome this opportunity to help create an open platform for customers.”

    “Customers need an open, interoperable future for software containers and we are happy to see the industry coming together to establish a common standard,” explained Kit Colbert, vice president and CTO, Cloud-Native Applications, VMware. “Our participation in the Open Container Project reinforces our goal to help customers innovate more rapidly and benefit from containers as they adopt the technology in their environments. We look forward to working with the community on container portability and security which are areas of focus for enterprise IT.”

    According to the OCP press release Docker will donate the code for its software container format and runtime—as well as the associated specifications—to the project. The leadership team from the Application Container spec (appc) will also contribute technical leadership and support for OCP.

    Ben Golub, CEO of Docker, summed up by acknowledging the effort and support for OCP. “We’d like to thank the many people in the industry who came together to support this important initiative. Our thanks to the Linux Foundation for the incredible work they have done in helping to pull this broad group together. Given the criticality of the intersection of containers and operating systems, we’d also like to thank Red Hat and Microsoft for their early leadership on this initiative. Our sincere appreciation to IBM and Intel for their ongoing encouragement of open governance. Finally, we’d like to recognize the members of the appc community, including Google and CoreOS, for their willingness to join all of us to constructively address these critical issues.”

    Now that all of the major players are united in a common cause to define and maintain an industry standard for containers it will be much easier for developers and organizations to embrace container technologies without reservation. The confidence that your investment of time, money, and effort won’t be wasted or paint you into a corner is very liberating.