devopsy.kinsta.cloud

Author: David Geer

Loggly Derived Fields Add Structure To Unstructured Logs

The Gist of Loggly’s Derived Fields Gesture

A top log management product vendor, Loggly announced an update to its tooling last month in the form of its new Derived Fields. The vendor has drilled down past the summaries in its Dynamic Field Explorer interface into Derived Fields that use metadata to structure unstructured log data.

Where traditional log management and analysis products have typically required you to create one-off regular expressions for custom analyses—expressions that you must build anew each time you want to put log data under a microscope—Loggly now enables you to use rules rooted in regular expressions that you create once to structure unstructured data when your systems first enter logs into the tool. You can look at resulting Derived Fields instantly, every time you need to do log analysis, according to Hector Angulo, Head of Products, Loggly.

The necessity for the Derived Fields capability arises from the fact that logs that escape structured creation contain so much untapped value. Until now, Loggly’s approach has been to automatically apply structure to common log types. “But we find that more than one-third of customer logs are still not structured. Even worse, many of these logs were created by developers for their own use, without the expectation that others would need to rely on them for critical troubleshooting. As such, it can be really hard for everyone but the original developer to decipher what specific logs mean,” says Angulo. Derived Fields attempt to retrieve 100-percent of that value from the log data by making analysis possible for any team member.

How Loggly Dogs Data Using Derived Fields

Loggly absorbs data from common log types such as Apache or JSON. By adding metadata that describes unstructured log data and then “injecting” context and structure into logs or parts of a log, Loggly’s Derived Fields reclaim obscure developer logs. Fields that uniformly describe the same kinds of data elements across logs enable users to compare data within logs to see what is happening across a DevOps environment.

Staff members who can create custom parsing rules based on regular expressions can set up the rules the DevOps teams will need at the start. Then, unless you come up with a new rule you want to add, your team members won’t have to create regular expressions again, and certainly not in order to search log data on the other end of that pipe.

Using Derived Fields

Using Loggly’s Derived Fields, you can drill down on a specific value in a field. Click on the field name and you can inspect all log events that have the same type of value. A summary chart above the individual event log will automatically refresh to summarize the events of the data from the given value, says Angulo.

You can easily see other log events that share similar data or characteristics in order to arrive at insights about what is going wrong, or right with the software.

How Derived Fields Help DevOps

“Rather than starting with an empty search box, Loggly starts off with bird’s-eye-view summaries using Dynamic Field Explorer,” says Angulo. From there, you drill down to fine grain details.

The time that Loggly’s Derived Fields save, time that it would normally take to create one-off queries for every analysis, is critical if you’re trying to solve a problem that is preventing end users from completing tasks and generating revenue for the business, says Angulo.

According to Angulo, Derived Fields specifically help DevOps teams to:

• Resolve issues faster because it is easier to spot the data that matters.

• Perform advanced analytics even with legacy applications that send out unstructured, text-based logs that you cannot update or don’t have the bandwidth to update.

• Extend advanced log analysis to more team members. You don’t have to be a regex wizard to gain insight from unstructured logs.

• Finally, the intuitive, navigable summaries generated in Field Explorer equate to less training time, less time with reference guides, and more time on data analysis and problem solving.

A Cautionary Note

As with any DevOps tool update, the proof lies in how Derived Fields work in your development process. Loggly touts 5K customers using its existing product. I’m sure they’re not trying to chase anyone away.

Are you using this tool? Do you have other input on this topic? See that empty comments field below?

July 14, 2015
Appster Must Produce Mobile Apps Using Nimble DevOps Practices (It Has No Choice)!

In order to meet the many challenges of producing loads of different app/server projects large and small in extremely rapid succession, Appster collaborates on, automates, and dashboards everything it can. “Unlike traditional product companies (i.e. one product with lots of developers) or corporate IT shops (i.e. silos of products and services that rarely change), Appster has to handle a wide spectrum of software projects across a myriad of industries and vastly different client profiles,” says Martin Halford, CTO, Appster. Many of the projects Appster works on are mobile apps.

To do this, the DevOps tools and processes must tackle each new and unique development project in a continuous project stream with the same fervor that any large development project requires. “Our DevOps focuses on automating testing, builds, and deployments for highly custom, highly complex app/server software development projects,” says Halford.

The tools that help Appster to accomplish this include the XCode IDE, the Eclipse IDE, Atlassian’s BitBucket, SourceTree (for source control), and Jira (for Agile project and issue tracking), Jenkins for build automation, Amazon Web Services, and Klipfolio for dash-boarding, lists Halford. “And due to the short turn time and short duration of app/server projects—between 6 to 16 weeks—Appster does not at present practice Continuous Integration,” says Halford.

Appster has three main objectives: to deploy only from BitBucket, to protect the non-development servers from the developers, and to display the build statuses via dashboard to all. Deploying only from BitBucket holds the source code repository accountable as the “single source of truth”, says Halford. In order to protect the non-development servers, developers do not have root level access to the testing, staging, or production servers, only to the development server in this four server stack, Halford explains. Further, the QA Team has access to the development and test servers as well as limited access to the staging server, but no access to the production server. Only the Dev Ops Team / Linux Admins have access to the production server; they also have responsibility for the maintenance and administration of the entire server stack, Halford says.

App Dev At Appster Prior To DevOps

Prior to moving to DevOps approaches, Appster developed apps using manual steps and documented processes that were prone to human error, says Halford. “The process was also prone to human shortcuts and attempts to hot fix builds by circumventing source control and documented protocols such as those that demanded that no one make direct updates to the production server,” Halford says. These issues were not immediately uncovered.

A Typical Trip Through Today’s Appster Development Sprint

CTO Martin Halford shared his take on a round trip through Appster’s development process today. Appster will first generate development servers from saved generic Linux server images, which the Dev Ops Admins will also secure. Servers will ultimately include development, testing, staging, and production servers (or DEV, TST, STG, PROD). Appster will then generate source repositories in BitBucket, and Jira Projects that it will link to BitBucket. Appster will also create a product backlog using an Agile Scrum Software Methodology.

According to Halford, the development team will begin to develop software using Sprints, and each Sprint will follow this process:

A. The developers will check out their code and associate tasks with User Stories and / or Bugs in Jira.
B. They check code in as Changesets that are attached to tasks, which are in turn attached to User Stories or Bugs.
C. Developers automate and deploy builds as needed to the development server.
D. At the completion of a Sprint, the QA Manager selects completed tasks (User Stories/Bug Fixes) for deployment to the test server.
E. BuildMaster initiates the build, which pulls Changesets from the source repository, generates the build, and deploys it to the target server for testing.
F. After a successful Smoke Test, the QA team performs more involved tests to verify the build.
G. BuildMaster follows a similar process to deploy the build to the staging server for User Acceptance Testing.
H. The next Sprint begins and the cycle repeats itself.

Concluding Thoughts

“DevOps can heighten the effectiveness of mobile development by forcing and enforcing discipline on all parties, preventing human error, speeding QA testing cycles, effectively stopping defects, and highlighting inefficiencies in the existing development process such as could be alleviated through additional developer training and mentoring,” says Halford. Further, having a dashboard that is visible to all parties providing transparency into server issues so that Dev Ops Admins can respond immediately during build or production is critical to the DevOps process.

July 8, 2015
Internap’s DevOps Culture: PrivateStack + CD = ? [Read On & Draw Your Own Conclusions]

Engineers at Internap, a hosting company and public cloud vendor designed a new OpenStack-based cloud platform and development environment. devopsy.kinsta.cloud tells the story of Internap’s DevOps cultural evolution, which grew virally, interwoven with the company’s development of PrivateStack.

Challenges in Internap’s Common Development Environment

Challenges that slowed Internap’s cloud development process triggered a hunger for change and for a new / altered development scheme. “The Internap public cloud product is essentially composed of micro services that make cloud resources available to our customers,” says Mathieu Mitchell, Senior Software Developer, Internap. Internap engineering teams tested their cloud services simultaneously with each affecting the other and transferring adverse effects to production or pre-production. Internap needed to let the billing team test their services, integrated with a duplicate of the production software and environment, without affecting the virtualization team.

These challenges were rooted in a commonly shared development environment for all the engineering teams and team members, which introduced tedium for engineers as they worked to build and deploy stable code.

When sharing a common development environment, each engineer’s tests depended on environment consistency and reliability. When engineers / developers tested two changes at the same time, there was no way to tell which change introduced a regression. The most reasonable way to address this prior to PrivateStack was to push all changes into the environment and dedicate specific engineers to identify and fix the issues that occurred.

The challenge with dedicating specific engineers to troubleshoot these regressions was that they had a diminished context to work with when compared to the original developer’s understanding. The team that introduced a change needed feedback from the environment and to shoulder responsibility for fixing the issue. This would result in a stable codebase and the ability to write more thorough, automated tests.

“This is why we created PrivateStack–to allow us to independently test a single change, integrated with other services, in an environment that is the equivalent of a private production setup,” says Mitchell. PrivateStack enables development teams to innovate while ensuring that changes behave correctly in production. This in turn enables CD and speeds development.

DevOps Culture Leads Teams to Success

As Internap developed PrivateStack, engineering observed a DevOps belief system spreading across the project and the teams. “At first, it was only a minority among us, mostly people with an Agile background, who advocated the Continuous Delivery approach while we worked on this project. We really believe in delivering value to our customers as quickly as possible. To achieve Continuous Delivery, we needed people to understand that automating everything was the way to go,” says Mitchell.

As more engineers realized that they were the solution, they became increasingly motivated to enhance processes, create the new development environment, and use CD to develop the Internap public cloud product.

Results with PrivateStack

PrivateStack enables Internap teams to share the same code and system configurations while individual developers have each their own private production environment to test their software without affecting others. “We heavily leverage virtualization to be able to recreate our environments, since buying racks and racks of hardware to make this available to all of our developers would be cost prohibitive,” says Mitchell.

With PrivateStack, Internap isolates production issues during development, keeping R&D efficient and slashing pre-production troubleshooting time. “PrivateStack also improves our time-to-market, is much less costly, and most importantly, reduces the chance of any issues reaching the customer,” says Mitchell.

DevOps Culture Wrap Up

To finish the PrivateStack project and foster a DevOps culture at the same time, Internap’s internal core of CD true believers lead by example. Mitchell and his colleagues adhered to consistent principles while hearing out other team members on their concerns. This helped them to encourage the larger engineering department including billing and virtualization teams to adopt a DevOps and CD frame of mind.

“We dedicated two people to drive this initiative, along with a few others who believed in the Continuous Delivery approach but were not involved directly,” says Mitchell. That was enough to get the job done. Now the vast majority of the engineering department is targeting CD.

“We are eager to share our PrivateStack platform with the open source community to enable other developers to run production-like environments for development in their day-to-day operations,” says Mitchell.

June 19, 2015
GE Capital Releases Mobile App Twice Weekly, Automating Compliance at Velocity, Thanks to DevOps/CD/CI

GE Capital offers equipment leases, mobile apps, and financial products for corporate customers. GE Capital’s Matt Merchant, Global Managing Director—DevOps, spoke with devopsy.kinsta.cloud about development initiatives and processes at the financial services giant.

According to Merchant, his team leads the charge in driving

(A) the DevOps cultural transformation throughout IT,
(B) the adoption and expansion of DevOps tooling, and
(C) the change in governing structures in order to operate at a much higher velocity while continuing to meet regulatory requirements.

That last mouthful speaks to a key pain point that GE Capital’s DevOps processes relieve: how to speed development while remaining compliant. GE Capital does it by automating security and compliance. You’ll read how shortly, but first, the mobile app.

The Mobile App for Fleet Drivers

GE Capital provides vehicle fleet leases globally for enterprises with delivery drivers, sales people, and others who travel on company business. The mobile apps that GE Capital provides as part of the leasing platform are many and vary from country to country. For this discussion, Merchant takes an app deployed in Sweden as the example.

The web-based app enables drivers to locate nearby gas stations, tire shops, repair shops, and windshield replacement services. Drivers also use the app to download routes, to collect refueling, mileage, and oil change / service data, and to log their hours. The app creates a number of efficiencies.

Drivers used to get shop data from a physical driver’s manual that they had to pore through to find the closest service provider. “The instant you hand them that catalog it’s out of date. Now they get real-time updates on their phones,” says Merchant. They also used to hand log their hours with pad and pen and then return to the office at day’s end to type this data into a computer. “Now they do it in real-time, and are done at the end of the day,” says Merchant.

Mobile App Development Before & After DevOps

In GE Capital’s old Waterfall mobile app development process, releases took six to eight weeks. Challenges included an inconsistent code repository, manual scripting and testing, and sitting before change, architectural, and security review boards every time developers evolved the code. “Because all that is so painful, you don’t want to repeat the process very often, and so you bundle more changes into a single release, which increases the risk of error,” says Merchant.

With DevOps, GE Capital added Chef configuration management and created an “over-arching umbrella” development process that the company calls ArchOps. ArchOps covers CD/CI and virtualized cloud platform management while leveraging a Lean Six Sigma approach to swift Agile application development, explains Merchant. This enables fast iterations, consistent code, automated testing and, using Chef, fast VM spin-ups. And there are no more change management reviews.

Achieving “Compliance at Velocity”

To achieve what Chef calls “Compliance at Velocity” and deliver compliant, auditable code at a high speed, GE Capital reconsidered its toll gate processes and artifacts. “We applied a value stream map to our governance model, removing 60-percent of our toll gates and 70-percent of our artifacts,” says Merchant. This shrunk mobile app release delivery to days.

GE Capital automated compliance by automating test output analysis. When the analysis shows that a portion of the code failed a test, the code goes back to development, which must fix it before retesting, further analysis, and eventual promotion to production. “That removed the need for a change advisory board,” says Merchant.

And when an auditor requests test results, release dates, or copies of specific code releases, Merchant simply pulls the given code and all the test results, design artifacts, and governance data that are bundled with it from the Git repository and hands it over.

Once Through Development, Please

Now with a DevOps mindset in command at GE Capital, all developers use the Eclipse IDE, which transfers the code directly into the company’s Git repository. Once the code enters Git, developers can run jobs using tools such as Jenkins, do Fortify application scanning, and perform UI testing using GE Capital’s home grown tool called Rainbow.

Once the code passes all tests, the system sends it into a pre-production staging environment. “GE Capital built the staging environment using Chef, which calls out to servers, spins up the environment, and places the code inside it,” says Merchant. User acceptance and remaining tests happen there. Then the Chef code that created the internal environment creates the production environment, ensuring application stability upon release.

June 16, 2015
RedHat Culture Keeping the DevOps Faith

Red Hat is readily recognized for its accomplishments with Linux and software design. The Red Hat DevOps culture itself started by design as a program built on training developers in operations skills and increasing operations’ faith in these developers. Red Hat’s IT Platform Operations Manager, Anderson Silva tells devopsy.kinsta.cloud how internal teams initiated the cultural change inside this Linux development icon.

Get Back In Line!

The Platform Operations team in IT manages and executes a lot of the deployments of IT-managed applications into these environments, says Silva. On the road to establishing Red Hat’s internal DevOps dynasty and prior to running the Platform Operations team, Silva captained a small band of operations folk in a department known as Production Control. There Silva and crew established controls to ensure that any new deployments would not and could not break the production environment.

“Sometimes this meant we had to halt the whole of a deployment because a developer forgot to add one link to our release documentation, putting them back in line with everyone else who was requesting permission to deploy,” says Silva. Feeling that Production Control as such was more the bottleneck than a deployment solution, Silva saw this as a challenge to Red Hat that would be increasingly important and increasingly difficult to meet.
In those days before DevOps was cool, a Red Hat development manager came to the rescue with a program to turn that bottleneck into a solution that could speed up the deployment process. According to Silva, this first initiative was the root of what became known at Red Hat as the Lieutenant program, which supports developers willing to gain the Red Hat Certified Engineer certification so they could enjoy production and pre-production access and manage each their own deployments as frequently as they liked. Red Hat eventually added a “corporals” component as an additional and similar program. These developers in “corporals” gained QA access and did not have to earn the RHCE certification to get it.

Riveting Results

“These two programs added significant value over the years. By building trust between developers and operations, we addressed many technical issues that we had historically viewed as ‘another person’s problem’,” says Silva, who also holds the RHCE and RHCA certifications and actively maintains a Fedora Package. Red Hat continues to foster a flexible DevOps culture and to nurture the flexibility that sprouts from added trust between engineers and developers.

That flexibility is paying dividends. Where as it once took Red Hat a week to develop and deploy software for about 20 servers to its staging environment, today the Linux titan can complete such a release a few times a day, says Silva. “I wouldn’t be surprised if in the near future we are able to do it a few times an hour when necessary,” Silva says.

And by keeping the faith between operations and development, Red Hat can limit the risk that might otherwise come with its migration to its hybrid cloud. This DevOps-relationship-based culture will also benefit Red Hat’s new image-based deployments, which count on tools that are common among members of this culture such as the Red Hat Enterprise Linux OpenStack Platform, OpenShift, Kubernetes (a data center tool), and Docker.

“As an open source company, we use open source solutions where possible and we have written our own tools to help us automate deployments,” says Silva. As Red Hat’s Platform Operations team is about to retire some of these tools in favor of others, it is also reaching out to other internal IT departments to spread the use of image-based deployments, which also leverage Jenkins, AWS, Puppet, Ansible, and Git.

The DevOps culture that a few managers seeded within departments has borne fruit in the form of a top-down leadership example that rewards and encourages developers who innovate. Red Hat draws temporary ad hoc teams including sys admins and developers together to address nagging development issues. The key to success, says Silva, is in working on these challenges together instead of finding someone to blame.

Free Advice

Silva advises that enterprise leaders who want to wade into fostering a DevOps culture focus on their people before whatever looks new and shiny among the many choices for tackling DevOps approaches. “Trust your people and let them do their jobs. When mistakes happen, don’t look to blame; look for a solution. Then take a look at what’s new and shiny,” Silva says.

June 3, 2015
Atlantic.Net Credits DevOps Culture to Mental Agility

Atlantic.Net (www.atlantic.net) is a Solid State Drive-based cloud hosting provider with locations in New York City, San Francisco, Dallas, Toronto, Orlando, and (coming soon) London and Singapore, says Marty Puranik, Founder & CEO, Atlantic.Net. The Atlantic.Net cloud system is an intuitive IaaS server environment for customers who are developers. The company backs its 24/7/365 operation with a deep, full stack DevOps integration throughout the organization.

That means that everyone has enough understanding of everyone else’s area within the company and of how everything is orchestrated so that when there is a change for the organization to make, everyone appreciates everything everyone else needs to touch to make the change successful. “That’s versus operating in a silo as only a database administrator or only a web developer,” says Puranik.

The Atlantic.Net software stack that makes change possible includes Linux, Nginx (http://nginx.org/), PHP (http://php.net/) /Python (https://www.python.org/), RESTful APIs for internal apps linkages, jQuery (https://jquery.com/), Ajax (http://www.w3schools.com/ajax/), MongoDB (/www.mongodb.org), node.js (https://nodejs.org), and KVM Virtualization. “We use serial protocols to communicate over Infiniband (a fast type of networking) so I would add SRP or iSER serial communications protocols, as well,” explains Puranik. (Infiniband is typically applied to high-performance computing applications.) Atlantic.Net applies a particular mindset as it uses these tools to serve its software, its customers, and its DevOps culture.

Mental Agility Key to DevOps Culture

Atlantic.Net teams feel that maintaining a mental agility across human resources and encompassing all stakeholders—customers, departments, and vendors, who often run DevOps teams, too—is a principal element of a thriving DevOps culture, according to Puranik. “Mental agility here means the ability to reformulate your best case scenario using new tools or upgrades to existing tools. Mental agility also means not designing systems that are so married to a legacy piece of technology that it is impossible to move them off that technology,” explains Puranik.

Atlantic.Net strives to design software such that they can easily add, change, or remove old and new technologies from the stack without making major revisions to the codebase. To do this, the Atlantic.Net team puts considerable time into mulling over design considerations prior to implementation to ensure flexibility in the relationships between technologies and the development stack. The team takes the additional time in order to abstract different models of how a new feature could work so they can make the best selection for everyone. That’s a better choice than tightly integrating the new technology simply because it’s easier for them to think through and makes development go faster. “In this sense, mental agility means keeping your options open so you don’t end up with your back against the wall and your best path cut off,” says Puranik.

The benefits of this type of mental agility are two-fold: the ability to continually evolve the codebase without any drag or friction from permanent ties to dated development moves; and, the ability to add functionality for customers with minimal code changes and without breaking the code.

Stretching Out Strategy to Satisfy Stakeholders

The nimble minds at Atlantic.Net take stakeholders into account because of the repercussions on customers, employees, and vendors of misguided changes to the software. “If, for example, we alter how a customer must access a particular feature, this creates a learning process for the customer,” says Puranik. If the change doesn’t carry enough value to make it worth the learning process, then the development team should not have put the customer or themselves through the whole mess.

In the same way, adding a new expression of a software component such as an API from a third-party vendor brings the reliability of the vendor into question. “Can the vendor handle the number of queries we are going to make? If not, it’s bad for the vendor, for us, and ultimately for our customers,” says Puranik.

Atlantic.Net must look at the entire development process from everyone’s angle, putting on their shoes and considering the outcomes of development choices. “Sometimes different people inside Atlantic.Net must be the champion on behalf of different stakeholders internally and externally to make the whole team think differently or at least think through what the outcome of specific actions might be,” says Puranik. That way continuous deployment is fast, and harmless—a win-win!

May 29, 2015
Netuitive’s Transformation to SaaS Delivery, Powered by Its New DevOps Shop, Part Two

In part one, devopsy.kinsta.cloud presented Netuitive’s DevOps transformation, which helped the company to automate its product development processes. Now in part two, we take a look at that development process as it stands today.

Netuitive Development & Technical Debt Initiatives

“We are using a Feature Branch Workflow model with Git to isolate our work while we develop and test it,” says Brian Spindler, VP of Engineering, Netuitive. As soon as all teams and team members are in agreement, Netuitive merges the feature with the master branch and deploys it to the internal pre-release staging environment for further testing and verification.

“In the old world, we released software twice a year,” says Spindler. After using the Scrum development process religiously and exclusively to take the software to beta, Netuitive drove the pre-release software through the testing ringer prior to releasing the packaged final product.

“Now we release on a weekly basis,” says Spindler. The development team shoulders the unit, integration, and acceptance testing while Netuitive reaches forward to increasingly add features and polish its architecture.

Netuitive is continually bringing the increasing technical debt into the black side of the development ledger, so to speak. Netuitive offers two examples of being technically in the red. The first is in the area of unfinished business. Incomplete or hacked up implementations typically happen when the team is developing outside the core technology stack, is cutting corners for speed, has a poor understanding of the software requirements in a given area, has to rush testing, or does not create complete and appropriate documentation, according to Spindler.

The second example circles library / dependency updates, where falling behind can lead to numerous interrupts and blocks for features / bugs / requests that are at the top of the queue, Spindler explains. “Often times in either of these two sample areas, a theme will emerge and that can be a good indicator that it is time to start paying down the technical debt in that area,” says Spindler.

Netuitive puts that debt under a microscope where through transparency and visibility, it can review the bugs on a regular recurring basis and, as soon as an engineer catches something that is offensive to the software, he can create a ticket in JIRA to put it on the backlog to address, according to Spindler.

“However, it doesn’t always make sense to ticket something if it is not actionable,” says Spindler. In such cases, Netuitive will catch those instances of technical debt in the Technical Roadmap Idea Board that it maintains, which captures three verticals: the development environment, the technical debt, and the architectural / structural vision, according to Spindler. “Any team member can contribute to this Board. To keep everything on it in perspective, we hold retrospectives regularly,” says Spindler.

Changes, Experiences

“It was a positive experience seeing how we handle bug fixes differently after switching to a SaaS model,” says Spindler. Now we understand bugs that we fix very well and we easily duplicate them for isolation and to excise them from the software. That’s because the developer has full access and freedom to debug the same system that he reports on. This eliminates the tedium of setting up systems and testing and repairing for edge case scenarios, according to Spindler.

“We were also positively surprised at how quickly something can go from concept to production,” says Spindler. To make that possible, there is an opportunity cost in that development has to be willing to sacrifice, to try something while risking having to toss something aside. “In the SaaS world, it’s all about trade-offs, fast vs. refactoring, pay off that technical debt or put out feature XYZ. When you have a team like ours, everyone’s time is critical, and we work hard at prioritizing and balancing that,” Spindler says.

To maintain the balance and improve the team’s precision and good judgement over time, Netuitive follows Kaizen, which is a continuous improvement methodology. “As the team grows and changes, so will our priorities and the way we work and build the system. We’ve got to be open to that and willing to embrace it,” says Spindler. Regular retrospectives and transparency help to make that possible.

May 14, 2015
Netuitive’s Transformation to SaaS Delivery, Powered by DevOps, Part 1

“Netuitive offers an adaptive monitoring solution for cloud infrastructure, applications, and services, powered by a patented self-learning correlation engine,” says Brian Spindler, VP of Engineering, Netuitive. The engine uses multivariate regression analysis to calculate correlation coefficients between metrics based on time-series data collected, leveraging the significance of certain relationships between metrics variables across the data.

The Netuitive technology establishes a baseline of what metrics and their relationships have been historically. Netuitive policies enable development and operations teams to see anomalies that present with significance above the historical baseline and related to AWS EC2, Linux OS, MySQL, and other systems and technologies.

“We have launched a SaaS version of our product wherein we now deliver code to production on a weekly basis,” says Spindler. This increase in deployment speed is thanks to the company’s new DevOps processes. Here is the first part of Netuitive’s two-part story.

Before Moving to a DevOps Shop

Before moving to a DevOps software development approach including continuous delivery, Netuitive used a Scrum framework together with Jenkins and Ant products and scripts the company developed in-house to develop Netuitive software and the patented self-learning correlation engine within. “We recognized the value of automation early on—using simple shell scripts and our own Web API we were able to orchestrate quite a lot,” says Spindler.

Still, these methods presented challenges. The extensive planning process, including conjuring up development time estimates that Scrum requires for Sprints were challenging. In addition to and despite all that planning, numerous interrupts occurred surrounding testing and deployment issues that stemmed from software complexity and integration management. “These interrupts slowed us down and impacted our burn rate,” says Spindler. (The burn rate calculates the speed of the business spend above and beyond income.)

Mountains of bug fixes were also a concern. With an exorbitant amount of time spent fixing bugs that would only present themselves in rare edge cases, Netuitive was putting too much effort into builds while getting too little in improvement or new features out of the process. “While we sought a solution for feedback loop issues, it became clear that this would require a more agile approach. We looked to DevOps for inspiration, which led us to migrate from Scrum to Kanban,” says Spindler. Kanban is an illustrative process for managing targeted production goals, including what to develop when and how far to extend that development.

As enterprises adopting DevOps often mention, the cultural change necessary to move to DevOps approaches was the most challenging. In particular, getting everyone on board with automated testing as a proof of software validity was tricky. Internal teams such as QA and developers had to transform their existing relationship. “For years, these groups were adversaries; now I am asking them to be allies. This required the developers to take on the responsibility of writing tests while QA validates and ensures the tests are sufficiently broad,” says Spindler.

Goals, Results, & Hazards: Method to Madness?

Netuitive had specific outcomes in mind from switching to a more complete DevOps shop. Quality and productivity improvements were high on the list. In fact, the two would have to be mutually inclusive and close knit to ensure both. To accomplish this, Netuitive deploys on a weekly basis, using the full Atlassian suite for collaborations and issue tracking along with a combination of Bamboo, Gradle, and Salt Stack to automate the configuration, manage the infrastructure, and manage deployment operations. This orchestration helps quality and productivity to dance in harmony.

The results are more than aesthetically pleasing. “We are increasing the velocity of deployments through continuous process improvements,” says Spindler. By deconstructing the Netuitive product and dividing it up into independent service modules and components, the company can move overall progress along more quickly by improving each individual component separately and simultaneously, making faster progress gradually than they would by waiting to finalize a single, complete product update.

Netuitive experienced three fully distinct bumps in the road or ‘hazards’ on the way to morphing itself into a DevOps shop. “There were issues related to automating testing, changing the organizational workflows to accommodate automation, and deconstructing the product into granular enough services that we could deploy independently,” says Spindler. These were all related to the desire to automate the product production as a whole. Netuitive works continually to refine the necessary balance between layering on additional tests across those granular services to raise the bar on quality and excising some of those testing layers in order to increase deployment velocity.

Tune next time when devopsy.kinsta.cloud reproduces the relative chronological Netuitive DevOps development process from release to release along with differentiations, benefits, and the payoffs that the company did not expect.

May 11, 2015
Step-By-Step: How DataArt Processes Application Backlogs, Converting Legacy Apps with DevOps

DataArt designs and custom builds software for industries such as travel & hospitality, media, IoT, financial services, and healthcare. “We build new products and services, improving existing solutions, re-engineering legacy systems, and providing expertise in technological and industrial domains,” says Anton Krasikov, Project Manager, DataArt.

How DataArt Uses DevOps

DataArt quickly delivers high quality software in support of operations performance. DataArt’s DevOps process enlists version control systems like Git and build systems such as Ant, Maven, and Gradle. “Almost every project relies on using these build systems inside CI servers like Jenkins, TeamCity, or TFS,” says Krasikov.

DataArt’s development process also requires a lot of unit and integration testing using frameworks like TestNG and Junit. “There are also projects where we prefer TDD or BDD approaches, with the use of specific frameworks such as Selenium for UI testing,” says Krasikov. By using Puppet, Vagrant, and Chef, DataArt can swiftly deploy the development environment in the CD pipeline. DataArt applies these select DevOps development tools using methodologies, which could include Kanban or Scrum (Scrum was most frequently mentioned by DataArt for this story).

Previously, DataArt relied on manual processes or scripting tools for build systems, source code, and unit tests. “It took a lot of management effort to tie everything including development, quality assurance, and operations together,” says Krasikov.

Challenges to Adding Backlogged Applications to Those You Develop Using DevOps Approaches

In the instance where DataArt returns to work on existing systems that include some legacy applications, it is possible to use agile development in maintaining these as they are, but excessive software dependencies could make it hard to align the software releases / release cycles.

“Legacy software may not have good test coverage or a proper build system, so we usually take steps in a DevOps direction, if not to achieve continuous delivery, at least to implement proper continuous integration,” says Krasikov.

There are challenges to adding backlogged apps to a DevOps development process. “These kinds of non-DevOps applications usually suffer from several issues, or in worst case scenarios from all of them,” says Krasikov. Backlogged apps waiting in a pre-DevOps state and hungry for transformation may have any or all of the following symptoms, according to Krasikov:

*many dependencies between system modules
*no automated environments
*no continuous integration
*no test automation
*no build system or inefficient usage

Transforming an Application, Step-By-Step

To add apps to DevOps approaches, says Krasikov, start by analyzing the existing source code, system dependencies, and system environments. The steps that follow come from an example with a recent Java project that DataArt did.

A) Select a source code control system such as Git.

B) Do refactoring and perform unit tests. “These go hand-in-hand, so the main business logic was covered with tests. It does not make sense to rush to 100% test coverage, but we tested the main bits. This also allowed us to reduce system complexity and decouple system components,” says Krasikov.

C) Build system – “The original application used huge and confusing Ant scripts to produce deployable packages. After we completed the refactoring, we split the application into modules with Maven as the build and dependency management system,” says Krasikov.

D) Integration tests – “We introduced these as a separate test automation suite to ensure that application modules work correctly as a whole,” says Krasikov.

E) Continuous Integration – DataArt set up a Jenkins server for multiple build jobs, building with each commit and targeting deployments to their intended environments with overnight integration testing. “CI allowed us to identify non-working builds early to save time on investigations,” says Krasikov.

F) Environment setup – Finally, DataArt implemented infrastructure as code. The new development environment uses Puppet with Vagrant to check out source code, spin up a VM box / template, and deploy fast, according to Krasikov.

DataArt runs some steps in parallel for additional time savings, for example moving to virtualized provisioned environments along with source code migration, according to Krasikov.

DevOps Helps DataArt Speed Projects to Their Customers’ Delight

By increasingly converting backlogged apps to DevOps development approaches, the enterprise benefits by achieving project goals and delivering a working software iteration with each pass through the DevOps processes. “The high level of automation reduces the risk of human errors and helps us to fail early in the development cycle. With continuous delivery, it is much easier for our clients to see the immediate impact of their ideas,” says Krasikov.

May 6, 2015
Netskope Draws Strength from DevOps In Support Of Scrum Development

Netskope keeps enterprises and their data secure despite the unbridled mix of approved and unapproved mobile apps in the market and on employee devices. Employees use unapproved mobile apps (BYOA) on their BYOD smartphones, tablets, and laptops when connected to networks away from the enterprise. IT professionals are keenly interested in tools that help them to have visibility into these apps and devices and to secure them.

Enterprises face many issues with mobile apps. They confront challenges where data location and access as well as security are concerned when complying with PCI DSS, HIPAA, SOX, GLBA, FISMA, FERPA, and the ISO standards. Netskope works to control app usage, protect enterprise data, and maintain compliance on various endpoints whether the device is on premise or remote / mobile, all in real-time.

“Netskope uses a distributed policy enforcement model that lets your people use these apps securely so you don’t have to block them completely,” says Abhay Kulkarni, VP of Engineering, Netskope.

Netskope’s Security Duties Dictate Its Development Path

“We do not follow Continuous Delivery,” says Kulkarni. Here are two of the many reasons for that.

First, Netskope’s customers demand that software releases are predictable. “Our customers are large enterprises that want predictable change windows so they can ensure that they make the necessary changes on their side or that they prepare for our updates,” says Kulkarni.

Second, because Netskope is a SOC-1 Type-II and SOC-2 Type-II certified company, it must comply with the requirements that comprise these certifications. Tightly-knit change management is critical to SOC certification. “As we do each change, we need to make sure that all changes meet the security, availability, integrity, and confidentiality criteria of these certifications by testing the entire product end-to-end,” says Kulkarni.

But Netskope follows other DevOps principles such as using its own adaptation of a well-accepted source control workflow model as well as an automation model, a model for bootstrapping machines, and also models for software installation and change management.

Netskope performs a balancing act between complying with SOC-1 and -2 trust services criteria and principles and maintaining rapid feature development. On the one hand, they manage to limit what individual developers can do on production machines—many times disallowing access. On the other hand, they use Sumo Logic to enable their coders to peruse anonymized logs in order to gauge how the service they instantiated is working.

Goals, Tools, & Challenges

“The key goal for Netskope’s DevOps approach is to ensure developer responsibility for running, monitoring, and scaling their services,” says Kulkarni. Netskope’s developers and operations staff work together from design through development while complying with a common framework, which helps in resolving development issues and brings consistency to data center operations, according to Kulkarni.

Netskope holds developers individually accountable for making their software ready for automated deployment. To help with that, Netskope uses familiar tools such as Ansible, which helps Netskope to automate everything in production. “Right from bootstrapping a machine to changing network configurations, we automate all commonly repeated tasks,” says Kulkarni.

Netskope deploys Sumo Logic agents using Ansible as a part of bootstrapping so that machines are configured from the get go to send logs to Sumo Logic. “Each service has standard logging methods that allow Netskope to create performance dashboards and to track errors,” says Kulkarni. Netskope uses a number of other tools that are commonplace in DevOps, such as Nagios, a monitoring tool and various external monitoring tools for tracking end user performance.

The biggest challenge for Netskope was not in tooling but in building a DevOps culture where developers lead the charge in responsibility for quality, error-free operation of their services. “In a non-SaaS world, developers ship a version of their product and celebrate. However, shipping a SaaS product is just the beginning. There is really no celebration since you are now responsible for the upkeep of the services,” explains Kulkarni.

A Trip Through Netskope’s Development Process

Netskope starts with a Scrum sprint planning meeting where in they decide what features to address for the given sprint. “We try to be realistic about our expectations in delivering code in a sprint by applying quality criteria—i.e. we expect code to be of a certain quality or else we cannot deploy it,” says Kulkarni.

Then developers bundle various modules on a single branch, which they test for functionality, security, performance, and integration. As the modules and the branch increase in stability, the developers upgrade them from one stack to the next until everything is deemed fit for a production deployment, explains Kulkarni.

Results

“Often we see proactive notifications from developers about issues they see in production. Detecting problems before our customers see them is a key benefit of the DevOps model,” says Kulkarni.

May 4, 2015