devopsy.kinsta.cloud

Category: Doin’ DevOps

Performance Monitoring, another view

This is my first post to devopsy.kinsta.cloud, for those who haven’t read my other writings, I currently work as the VP of Market Development and Insights at AppDynamics. I joined in February 2015 after being at Gartner for 4 years as a Research VP covering all things monitoring. During my time as a Gartner analyst I would help buyers make decisions about purchasing monitoring software. When I read articles on the internet which blatantly disregard best practices I either smell that something is fishy (sponsorship) or they just didn’t follow a process. Conversely some people do follow a proper process. I specifically wrote this up due to reading a really fishy article.

Step 1 – Admit it

The first step to determining you have a problem is admitting it. Most monitoring tool buyers realize they have too many tools, and none of them help isolate root cause. This is a result of most buying happening at the infrastructure layer, when the application layer is either lightly exercised synthetically or disregarded. People soon realize you must find an application layer monitoring tool, or write your own instrumentation. The vendors who sell logging software would love for you to generate a lot of rich logs, so they can help you solve problems, but realistically this doesn’t work in most enterprises as they are using a combination of custom and packaged software, and forcing developers to write consistent logging is a frustrating and futile task.

Step 2 – Document it

The next phase is to document what you are using today, most organizations have over 10 monitoring tools, and they typically want to buy another one. As they consider APM tools they should document what they have today, and determine a plan to consolidate and eliminate redundant tools. Having tools within silos will happen until you can fix the organizational structure.

Step 3 – Requirements

I can’t tell you how many times vendors set requirements for buyers. You can tell when vendors do this versus actually understanding the requirements the user has, and if the solution will scale and be operable by the team. Purchasing a tool which requires full time consultants on staff to keep it running is no longer feasible today, sorry to the old big ITOM vendors (formerly the big-4) but people don’t want that anymore.

Step 4 – Test it!

If you are evaluating software you MUST test it, and make sure it works in your environment. Better yet load test the software to determine if the vendors claims of overhead and visibility are in fact true. I see far too many people buy software they do not test.

Getting back to the article which prompted this post:

https://devopsy.kinsta.cloud/features/devops-style-performance-monitoring-for-net/

The author is located in Austria and I believe there is a connection to Dynatrace. Dynatraces’ R&D and technical leadership are in Austria, but this is just conjecture. The article is written with a bias, which is obvious to any reader who has used these tools. The evaluator did not test these products, or if he did he didn’t have a testing methodology.

Finally the author’s employer is new and all employees have come from Automic Software in the past year. I don’t need to explain to those who work with Automic the state of things there. http://www.glassdoor.com/Reviews/Automic-Software-Reviews-E269431.htm#

I think I’ve spent enough time on the background, here are my specific issues with the content.

The screenshots are taken from various websites, and are not current or accurate. If someone writes an evaluation of technology, at least test it, and use your screenshots from the testing!

This post was specifically to discuss DevOps, but yet the author doesn’t discuss the need to monitor microservices or asynchronous application patterns, which are clearly the current architecture of choice. This is especially the case in DevOps or continuous release environments. In fact Microsoft themselves announced new products last week at Ignite including nano server, and Azure Fabric Service for building these patterns. The difficulty with microservices is that each request externally spawns a large number of internal requests, these often start and stop out of order creating issues when monitoring these. This causes major issues for APM products, and very few handle this effectively today.

The article calls out user interface and deployment models inconsistently, and fails to mention Dynatrace is an on-premises software product. Dynatrace is also a thick Java client, built on Eclipse. This is the definition of a heavy client, which the author calls out for other products in this article.

Continuing the incorrect facts in this article the author calls out Dell (formerly Quest) Foglight. This is not Dell’s current product for modern architectures. Dell has moved to a SaaS product called Foglight APM SaaS, which is not what was evaluated. The evaluator should look at the current product (noticing a trend here).

Finally the AppDynamics review is inaccurate:

Editors note: the following is this authors opinions, not the opinions of devopsy.kinsta.cloud. As the author states, he is employed by AppDynamics.

“code level visibility may be limited here to reduce overhead.”

The default in order to scale AppDynamics to monitor thousands of systems with one single on premises controller (something no other APM vendor delivers) is done based on advanced smart instrumentation, but every transaction is monitored and measured. All products do limit the amount of data they request in some way. AppDynamics has several modes including one which captures full call traces for every execution.

“ lack of IIS request visibility”

AppDynamics provides excellent visibility to IIS requests, I’m not sure what the author means here. AppDynamics also supports implementation on Azure using nuget.

“features walk clock time with no CPU breakdown”

Once again, I’m not sure what the author is referring to here. The product provides several ways of viewing CPU usage.

“Its flash-based web UI and rather cumbersome agent configuration”

The agent installation is no different than other products, showing the author has not installed the product. Finally the product is HTML5, and has been for a while, there are still some flash views in configuration screens, but those are being removed with each release. I’d much rather have a web based UI with a little remaining flash than a fat client UI requiring a large download.

AppDynamics has a single UI which goes far deeper into database performance than the other products in this review. AppDynamics also provides far more capabilities than what was highlighted in this article including synthetic. AppDynamics provides deployment of the same software via SaaS or on premises installation.

Finally this review did not look at analytics, which is clearly an area of increasing demand within APM. That being said this review is far from factual or useful.

Hopefully this sets some of this straight, please leave comments here, or @jkowall on twitter!

Thanks.

May 20, 2015
Lean Value Stream Mapping for DevOps
Value Stream Mapping is a crucial step in assessing an organization’s DevOps capability. We create a value stream map of the software development lifecycle early in any DevOps engagement, because it helps us to-
- Provide context for our architects and technical experts who are, in parallel, examining tools and technology.
- Establish a performance baseline, against which the improvements delivered by adopting DevOps will be measured. (See my other blog post on Box Scores for more on this).
- Begin the process of winning hearts and minds, and engender cultural change. By working side-by-side in a workshop, Dev and Ops resources, as well as those involved in planning, test scheduling, and other activities, start to gain an understanding of how the work they do is linked to steps upstream and downstream. This lays the groundwork for closer coupling of those activities in the future.
What’s interesting is that the template we use for our DevOps Value Stream Map is almost identical to that which we use for other service processes. It’s been used by IBM Lean Coaches to assess processes as diverse as Marketing Campaign Development, Order-to-Cash, Mortgage Applications, and Production Support to name but a few. This template was evolved from more traditional Value Stream Maps used to capture manufacturing processes, and a slightly different template also exists for innovation or product development processes.

At the start of every Lean project – whether it’s in DevOps, Insurance Claims, New Product Development, or Financial Forecasting, I review the Value Stream Map template. I ask myself and my team if the questions make sense for this process and if the boxes we are seeking to fill are relevant. Some of the language might be slightly different, and the outputs will certainly differ – from a high frequency process where a step takes minutes and seconds, to a creative process where a step could take weeks. But almost without exception, the template remains the same.

We ask the same questions across all these types of processes – and now we ask those questions of the Software Development Lifecycle. Those questions are:
- What activities are being performed?
- Who delivers those activities? How much of their time is dedicated to those activities?
- What is the yield, i.e. ‘right first time’? What % of rework is generated? What causes rework?
- What tools are required to complete the step? And what other inputs are required?
- How many units (e.g. work packages) are ‘work in progress’ within this step?
- How many units are waiting in the ‘inbox’? How will they be prioritized or scheduled? How long will they, typically, wait for?
- How long does the step take? How much of that time is effort, or touch time, versus duration, or in-process lead-time?
When all’s said and done, all Value Stream Maps have something fundamentally common: they are all about flow – and DevOps is no different.

About the Author/ Katie Sotheran

Katie is an IBM Lean Master based out of Los Angeles, California. Katie has many years applying Lean principles to processes as diverse as beverage distribution in Spain, lubricant product development in the Netherlands, Insurance Sales in the UK … and IT operations, all over the world.
May 19, 2015
Moving from Infrastructure Automation to True DevOps

Deciding on two separate teams, or an integrated end-to-end team

While a lot has been said about “the right way” to do DevOps, what we’re seeing in practice are two different approaches being tried to enable development and operations to work together as a team. On the one hand, there’s the approach to create an end-to-end, integrated development team, which you might call ‘Devops’. On the other, there’s ‘DevOps’, featuring two development teams – one working on application code, with another developing a platform using Infrastructure as Code techniques in a virtualized environment.

DevOps: providing a platform

Perhaps the most obvious way to go is to have a platform team to provide a runtime environment that “just works”, and a separate application team that develops the business logic to run on the platform.

For some organizations, particularly where one of the teams is outsourced, this separation is going to make more sense. You might buy an off-the-shelf product, build your own platform in-house (which seems to be what many organizations are looking to do with Docker right now – warning: it’s harder than it looks!) … or sign up with a PaaS vendor.

Whatever the situation, the platform should be versioned, just like any other piece of software.

This approach can be ideal for developers because they are free to work on code and focus on the business logic, and they really don’t need to worry about how it runs in production. It’s up to the operations team to make sure that the platform is stable and that it delivers what the developers need.

Communication is still key

Even with a physical or organizational separation, the importance of effective dialogue between the two teams cannot be overstated. This won’t work if the relationship is adversarial. There has to be a genuine awareness of common goals. If the operations team doesn’t understand where the development team is going, and what they need from the platform, then there’s going to be trouble. The idea that one platform is going to cater for every development need is often unrealistic, so versioning and the ability to run multiple platform versions side-by-side, or cater for exceptional situations where a particular application might need its own stack, is very important.

There has to be a clearly defined boundary, which should be close to the application code and configuration layer. The application handles the business logic, the platform handles the deployment, logging, monitoring, failover and auto-scaling.

The development team can focus on getting those new features and improvements into the live product, while operations handles the actual roll-out and manages traffic. For this to work, the platform must enable you to quickly determine who is responsible for any problem that crops up. It’s worth investing time in the setup to ensure that errors are explicit, so you don’t waste time tracking them down.

Arguments between the two teams are bad for progress and morale.

Devops: A full stack development team

For some businesses, it’s going to make more sense to put together integrated development teams that work directly on delivering the full stack from a shared code base, from virtual machine to application code. This allows you to design and optimize each system in a way that takes the application’s specific perspective and requirement into account, and can be an especially good fit if your applications differ from each other greatly in terms of the operating systems, middleware, etc., that they run on.

This approach can only work if the overall group is small (Amazon’s “two pizza” teams are a good example) and has sufficient knowledge and experience to see the big picture of the entire system.

It’s not possible for everyone to understand every part of the full stack in detail, but there are areas of overlap where greater collaboration will increase the level of shared knowledge and allow the team to build a better overall system.

Pulling together

The shared foundation of the Devops approach builds mutual respect and enables more free-flowing communication, but it must be based on shared goals and a vision of what the overall purpose of the system is. This integrated approach can give you much more flexibility, but it also means holding the whole team responsible for keeping everything running. Shared code equals shared responsibility.

The right approach

You need to take a baseline, and then put metrics in place to track your progress. You have clear goals in mind and a system to measure them. So how do you find the right approach to achieve those goals?

It would be wrong to say that one approach is better than the other. What you need to do is assess what will work in your organization. Both are means to achieving similar goals, but finding the right approach is about understanding what your company is looking to achieve. You might choose common goals, like reduced expenditure, or reduced error, or you might choose something more specific to your industry. Establish your goals at the outset, and make sure that they’re measurable.

The right approach is going to be the one that gets results for you — that’s why setting up a baseline and metrics is so important. You may find that a hybrid of the two approaches is going to be the right fit, but you aren’t going to know for sure until you have some hard data to back up your expectations.

Be sure to join devopsy.kinsta.cloud editor-in-chief Alan Shimel on Thursday, May 21^st as he hosts the webinar, “Ship Faster without Breaking Everything” to further your smarts on Continuous Delivery.

About the Author/Andrew Phillips

Andrew Phillips heads up product management at XebiaLabs, a provider of continuous delivery automation. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

May 14, 2015
Documenting DevOps: Agile, Automation and Continuous Documentation
Chris Riley former GigaOm analyst, O’Reilly author, devopsy.kinsta.cloud featured writer and founder of Fixate I/O has written an outstanding report on the key role documentation must play in the modern software development lifecyle. Agile or DevOps, automated or not, we live in a continuous IT era. So why shouldn’t documentation be continuous as well? Riley’s analysis includes key findings including:
- Even the definition of documentation is subjective. Despite several decades of implementation, the methods, drivers, and even language of documentation are nearly limitless. Standards such as ITSM help stabilize the look, feel, and process of documentation, but it does not help an organization understand how to use that documentation and integrate it into development.
- The modern software delivery pipeline introduces new complexities in administration. New tooling in the DevOps space has improved application efficiency and quality, but in some cases, this has also reduced the manageability of the environment. This creates new challenges to knowing what you have, how it functions, and how to extend it into the future.
- As the delivery pipeline moves to a continuous stream of releases, classic documentation processes become impossible. Documentation must happen automatically within the process, or not at all.
- In order for documentation to be useful, it needs to be actionable.
Download the complete report on continuous documentation for an in depth analysis, conclusions and guidance:
Documenting DevOps
Email
This field is for validation purposes and should be left unchanged.
Name*
First Last
Company
Email*
May 13, 2015
4 Ways DevOps Teams Create Value.

Sven Malvik – I’ve written thousand of lines of code based on lies, lies I hoped were the truth. The truth is that we make false assumptions. Assumptions that kill our business. I’ve built software that was too expensive, I’ve built software nobody wanted, and I’ve built software that never saw the light. If you are like any other developer you probably have lied too. But it doesn’t matter. What matters is the application we haven’t built yet and that we will build based on the truth. We need the truth to create value and to survive. This article will show you 4 ways of how DevOps teams create value.

(more…)

May 12, 2015
Is DevOps on the cusp of Disillusionment?

If you are familiar with, or buy into the idea of the Hype Cycle, you are familiar with the Trough of Disillusionment. When it comes to adoption of pretty much anything, there seems to be a point, just after the peak of enthusiasm, where those in the know are disappointed with the practicality and adoption rate of their thing or cause. Now that DevOps is several years old, and, in the last year enthusiasm has been at an all time high, is DevOps on the cusp of Disillusionment?

Hype Cycle – Wikipedia

(more…)

May 11, 2015
DevOps Teams: Containers Will Help You Become Future Ready

Container frameworks are emerging as a superior choice to create, manage, and run distributed applications in a scalable and secure fashion. Docker’s growth is testimony to the fact. There are 100 million downloads of the Docker Engine, 75,000 Dockerized applications, 770 contributors, and 50,000 third-party projects on GitHub using Docker in less than two years.

The State of Containers 2015 Docker Adoption Survey found that more than 70 percent of respondents are either using Docker or evaluating for future use. Given Docker’s overwhelming popularity, here are three trends that DevOps teams should prepare for:

–THIS STORY HAS BEEN MOVED TO CONTAINER JOURNAL–

About the Author / Varma Kunaparaju

Varma Kunaparaju is CTO at Vistara and a seasoned technologist with over 20 years of experience leading engineering teams to deliver enterprise software. Reach him on Twitter @varma1

May 8, 2015
Social Skills for DevOps: Whose responsibility?

In his post “DevOps developers; don´t be a DevGoof” on devopsy.kinsta.cloud Parker Yates pointed out, that developers have to become more social. Devs should better realize that they need to become a normal part of the business social structure. That is really true and extremely necessary to make DevOps more efficient.

But is this the whole truth? And who is responsible for the social structures and the relationships?

First let´s have a look at the devs. For me as a consultant working with people from IT I always get the impression that developers are creating a cult in being a “nerd”. They use being “nerds” as an excuse for not being compatible with the “others”! To be honest, this is a very cheap way of remaining within their own comfort zones! But perhaps we should take some time to ask for the “why?”! Developers are human beings, too. As is the case for all human beings, there is always a reason, why someone acts and behaves the way they do! So what could the “why?” be? Often experiences in our past are the reasons for our behavior and our actions. But perhaps it is a little bit different with the devs. Most of the devs start sitting in front of PC´s and producing code mostly at the age of puberty, at a time when their own personality is being shaped. This time in front of PC´s is not used to improve their own social skills and interaction with other people. So why do they sit in front of the PC at that age? This could be simply because they are really thrilled by the tech stuff or because they like to deal with PC´s much more than interacting with other kids, perhaps because of experience. Additionally, along with the possibility that they do not like to interact with others they also have a lack of time in practicing social skills and communication. But this lack will have the effect that when “contact happens” the result could be disappointing on both sides. And for the later developers a formative experience is made. All that stuff about being the only smart person in the room could be some kind of self-protective mechanism for not being responsible!

And now, what about the “others”? They call themselves “the experts in social skills” in comparison with the “nerds”. One very important part of social skills is empathy. But empathy doesn´t mean to blame other people for a bad relationship. Rather, it means to change the point of view for a better understanding! To point a finger at the developers and make jokes about them is not the way, somebody with good social skills will act! The reason for their ways of acting and behaving would also lay in experiences and preferences. A relationship always depends on both parts!

When people choose their profession they look for jobs, which fit with their interests and personalities. And with bringing different professions together, different types of personalities are brought together, so of course there could be a gap between them. And this gap often is filled up with stereotypical thinking leading to prejudices!

One important requirement to bridge this gap is, that all parts are made to feel responsible for building up that bridge. The attitude of “I am okay – you are okay” is the only useful way! It is a process of a personal change, which has to start!

When I once started my work on IT and DevOps I was awestruck that all people were talking about tools, but nearly no one was speaking about how to train people to improve communication and social skills! When someone has a backlog in such important personal spaces, how can we expect more of this person without giving support? And why do we expect that a team full of different personalities will grow together and will reach its full potential that easily without any support from an expert?

I have made the observation that developers especially are very interested in ways they can improve their social and communication skills and would really enjoy it if their companies would offer them, for example, workshops or team supervisions.

All parts of stakeholders are responsible for their success; and in particular the company which wants to run DevOps! So do not always think about technical tools; think of the human contribution!

May 5, 2015
Why I’m Happy To Have DevOps Engineer In My Job Title

A recurring argument in the DevOps community is whether the word DevOps should exist within a job title. Most people close to DevOps think it’s a potentially harmful anti-pattern which indicates a misunderstanding of what DevOps is all about. A few years ago I agreed with this, but I’ve now changed my mind.

Yes, DevOps is a culture, a mindset and all of those good things. It is about how people work together much more than automation. I also appreciate the argument that labelling individuals and teams with DevOps potentially introduces a silo where we are trying to solve a problem of siloes.

With all of that recognised, I’m still more than happy to now call myself a DevOps Engineer.

There were lots of job titles that didn’t exist 10 years ago. Agile Coaches, Social Media Managers, Data Scientists and Mobile App Developers for instance. Those fields evolved and there are now recognised experts in those fields who know how to apply the tools and help organisations be successful with them.

DevOps is the same. It is maturing as a movement and there is now enough specialist knowledge in the niche that people can build significant niche expertise, work within it full time, and even build long term careers as DevOps Engineers.

Definition Of A DevOps Engineer

DevOps Engineers are the people who sit between or within Development and Operations and use automation across the SDLC to reduce the cycle time from development to production.

These Engineers will bring knowledge across Development, Testing and Operations and help with gluing those areas together to ease the transition of code across them.

They will have deep technical skills in tools such as Puppet, Chef, Ansible, Vagrant and Docker, and use these tools to help their organisations deliver better software, faster and more reliably. They will also likely understand cloud and how to use infrastructure automation to leverage this to best effect.

As important as the technical knowledge is what we call the ‘DevOps Mindset’ – a personality type and approach that can help the IT organisation move from traditional ways of working, break down siloes, and collaborate more effectively. Technical skills alone do not make an effective DevOps Engineer.

Why This Matters

To me, this all sounds meaty enough to be a full time role and a career path. Therefore, why shouldn’t it also have it’s own name to identify it by?

I’ve always been pragmatic about this, but my main interest in this is how accepting DevOps Engineers as a job title might help to overcome the DevOps skills crisis – the huge shortage of people in industry who have this skill set and the fact that as an industry, we have still have no sustainable path to creating these people.

By accepting DevOps Engineers as a job and a career path, I think the definition of one would become more concrete. There would be more of a shared understanding over what it means, rather than savvy build and release engineers or system administrators re-badging themselves.

More people could align themselves to DevOps Engineering and work towards obtaining this recognised skill set.

Visibility of DevOps activities within the business would grow, and budgets would be created to support DevOps rather than it being an under the radar activity happening under other functional siloes.

Training for DevOps Engineers might emerge, and one day, it might even end up as a University Level course.

All of these things would help DevOps expand as a movement and practice in industry, which is ultimately a good thing for the sponsoring organisations and anyone who works in IT. Here at Contino, the more we interact with customers around DevOps, the more I realize this is the path things should take.

April 30, 2015
Successful Failure

In software development, success is all about failure.

That may sound as if we are carrying the “fail fast, fail hard” philosophy a little too far, but which would you rather have — a quick, definite failure that you can identify, repeat, and analyze, or a series of semi-failures and partial recoveries that leave your application and your data in a corrupt and unstable state? Anybody who’s spent much time testing or debugging software should know the answer: errors that are masked by near-recovery are almost as hard to trace as intermittent errors — and all too often, there’s no real difference between the two. Slow failures masked by partial recovery show up as on-again, off-again bugs.

(more…)

April 28, 2015