devopsy.kinsta.cloud

Author: David Geer

  • PicsArt’s APM Transition Brings Clarity for Customers

    PicsArt’s APM Transition Brings Clarity for Customers

    PicsArt’s Technology Landscape

    PicsArt enables its creative user base to freely upload or draw, edit, and exchange digital images in a worldwide, web-based community. PicsArt works on the web and on Android, iOS, and Windows Phone platforms.

    PicsArt applies state-of-the-art backend infrastructure to provide its editing applications and social reach for its community. PicsArt uses popular DevOps technologies such as the MongoDB database, the Node.JS JavaScript runtime, and the Redis in-memory data tool. With a foundation of these tools and infrastructure, PicsArt builds and supports apps that avail its users of many free options for creating and appreciating original digital art.

    According to Mikayel Vardanyan, Chief Technology Officer, PicsArt, PicsArt apps and technologies process millions of user requests each day. “We will also soon provide a public API to all of our developers, giving them an opportunity to use our backend services to create their own apps,” says Vardanyan. Though this will satisfy customer requests for additional functionality, it will also further multiply the intricacies of the PicsArt systems, making it ever more critical that they perform properly from foundation to pinnacle.

    PicsArt Experiences the Same Pitfalls as Other Tech Firms, and Responds

    As with any technological system, PicsArt’s infrastructure is at risk of downtime, which looms ready to strike the business and to challenge customer satisfaction at any time. Unforeseeable events as well as planned maintenance outages create downtime and are an expected part of the systems lifecycle, even though they are not always welcome.

    But malfunctions and downtime can exacerbate DevOps processes and corrupt and diminish the user experience. Downtime can temporarily take out entire services, ones that customers will definitely notice and miss when they become unavailable. “If, for example, the users cannot upload their pictures, that would certainly be bad for us,” says Vardanyan.

    Sufficient downtime on the PicsArt backend systems could also cause user registrations functions to fail. The backend also supports PicsArt’s community activities such as sharing artwork, liking others’ artwork, and commenting on people’s creations, which could suffer. “We try to prevent downtime by providing redundancy and scalable systems. We properly test our systems on a regular basis,” says Vardanyan.

    “When we start to sense issues such as these, our operations team goes to work immediately to fix what we need to fix. We investigate all the issues we come across and respond by adding more layers of redundancy and scalability,” says Vardanyan.

    PicsArt Sought Picture Perfect Improvements from APM

    PicsArt was looking for increasing visibility into its infrastructure in order to monitor for the aforementioned problems at their very outset, to create alerts on trouble issues and status reports, and to derive metrics about what is happening across the entire infrastructure from nation to nation, and region to region.

    PicsArt knew that it needed extensive monitoring and that the best way to go would be to have that monitoring centralized in a single product. “We needed to monitor the infrastructure 24/7 in order to provide the best possible service for our community,” says Vardanyan; “we needed external as well as internal monitoring and we needed to monitor the user experience”.

    APM Frames Performance Challenges Nicely

    PicsArt currently uses Monitis. At PicsArt, systems are continually wrestling with pressing loads and with load balancing. If systems performance should lag for any reason, the network can take center stage as the weakest link and the strongest bottleneck. “We utilize the different alerting capabilities of Monitis, such as voice calls and SMS, using all the escalation procedures that its notification systems offer, too. We also use Monitis to create custom monitors for certain scripts and for Redis jobs,” says Vardanyan.

    The cloud-based monitoring solution enables the digital art editing and sharing platform provider to leverage views from a number of global vantage points, clarifying transparency into PicsArt infrastructure. “With Monitis, we can check our global CDN locations and find out what location may have an issue. The benefit is that a potentially issue-stricken location does not escape the monitoring routine,” says Vardanyan. The sooner the enterprise sees it, the faster it can fix it.

    The conveniences and efficiencies of dissecting it all inside a single dashboard enable PicsArt to repair the real problems in its DevOps and platform delivery processes without touching anything that is not out of tune.

  • The Scalable Path to DevOps Career Training & Development – A Premium Talent Marketplace

    The Scalable Path to DevOps Career Training & Development – A Premium Talent Marketplace

    “Our company is a premium talent marketplace with on-demand leadership and clients mostly in the Bay Area, Silicon Valley, New York, and other metropolitan areas,” says Tomas Gutierrez, Partner, Scalable Path, a geographically-distributed network of more than a thousand software architects and developers. Scalable Path serves clients such as tech startups and product companies by helping them build software teams and fill technological gaps. Given that Scalable Path assists with software design, its clients most certainly have needs in DevOps approaches. Scalable Path’s Gutierrez informs devopsy.kinsta.cloud about DevOps career training and development priorities and building blocks.

    Foundational Knowledge

    At Scalable Path, a good working knowledge of the seven layers of the Open Systems Interconnect model is important for software engineers who want to work with DevOps approaches. “In order to have a full mental picture of what is happening where, software engineers must have a conversational knowledge across all these layers,” says Gutierrez.

    With any application of software, you will host it on hardware, you will have an OS, and hosts will talk to each other across a network. “The DevOps engineer must know the seven layers involved in order to consider goals for infrastructure as code and to help with software releases, deployment of infrastructure, networking, and configurations,” explains Gutierrez.

    Applicable Traits

    On the job, a good DevOps engineer must be a good problem solver, be able to think outside the box, and have empathy for the needs of the software product and the organization, says Gutierrez. Attention to detail and a security focus are also worth accounting for.

    “Even though it’s good to have formal training, college degrees, and certifications, the right personality, that of an autodidact (one who is self-taught) is key for good DevOps talent,” says Gutierrez. These engineers will examine a technical challenge globally and in its component parts and seek and assemble the tools and processes that will provide a complete solution.

    Further, someone who is empathetic to the needs of the software and the enterprise deploying it will realize when they need to use a simple, cost-effective software deployment without all the extras.

    The ability to direct a focused attention to the details of software engineering is another important trait for a DevOps engineer. Otherwise, programming errors occur that can create issues down the road. “If you don’t architect something correctly, you may end up with part of your architecture failing because you made some mistake in programming,” says Gutierrez. These things can propagate and scale up into bigger issues as the architecture grows.

    It’s also important for the DevOps engineer to be strong in security such that their programming does not expose critical data. It’s about how the programmer constructs the data layer, how they work with the transport layer, and what the end to end route is for the data, says Gutierrez.

    Certifications

    Despite the emphasis on traits and approaches to challenges on the job, there are certifications of value to the DevOps engineer. “The Cisco certifications, for example, especially the ones for networking will be helpful since a lot of what you will be doing in DevOps is about data in flight,” says Gutierrez. DevOps engineers need to be mindful of how traffic is moving over the Internet and to and through the data center, explains Gutierrez. Network configuration and network security expertise especially such as is validated through the higher Cisco certifications will be important for the DevOps engineer, Gutierrez adds.

    Application security certifications are also useful as are vendor certifications and courses from Chef, Puppet, and Docker as well as platform vendor training from AWS, Digital Ocean, and the like.

    Individualized Learning

    User documentation and “try me” opportunities for different tools such as containers and configuration management, automation, and deployment tools are helpful for self-starters and people who run into problems or have questions along the way. “People engaging in these kinds of training can simply create local laboratories for these tools on a local machine to simulate the larger application or architecture they are trying to learn,” says Gutierrez.

    In order to learn how to navigate the in-depth areas between the system and application architectures, the DevOps student should consider the books in and around these subjects. Just a few examples include, “DevOps: A Software Architect’s Perspective (SEI Series in Software Engineering)”, “Effective DevOps: Building a Culture of Collaboration, Affinity, and Tooling at Scale”, and “Salt Essentials”.

  • The Scalable Path to DevOps Career Training & Development

    The Scalable Path to DevOps Career Training & Development

    “Our company is a premium talent marketplace with on-demand leadership and clients mostly in the Bay Area, Silicon Valley, New York, and other metropolitan areas,” says Tomas Gutierrez, Partner, Scalable Path, a geographically-distributed network of more than a thousand software architects and developers. Scalable Path serves clients such as tech startups and product companies by helping them build software teams and fill technological gaps. Given that Scalable Path assists with software design, its clients most certainly have needs in DevOps approaches. Scalable Path’s Gutierrez informs devopsy.kinsta.cloud about DevOps career training and development priorities and building blocks.

    Foundational Knowledge

    At Scalable Path, a good working knowledge of the seven layers of the Open Systems Interconnect model is important for software engineers who want to work with DevOps approaches. “In order to have a full mental picture of what is happening where, software engineers must have a conversational knowledge across all these layers,” says Gutierrez.

    With any application of software, you will host it on hardware, you will have an OS, and hosts will talk to each other across a network. “The DevOps engineer must know the seven layers involved in order to consider goals for infrastructure as code and to help with software releases, deployment of infrastructure, networking, and configurations,” explains Gutierrez.

    Applicable Traits

    On the job, a good DevOps engineer must be a good problem solver, be able to think outside the box, and have empathy for the needs of the software product and the organization, says Gutierrez. Attention to detail and a security focus are also worth accounting for.

    “Even though it’s good to have formal training, college degrees, and certifications, the right personality, that of an autodidact (one who is self-taught) is key for good DevOps talent,” says Gutierrez. These engineers will examine a technical challenge globally and in its component parts and seek and assemble the tools and processes that will provide a complete solution.

    Further, someone who is empathetic to the needs of the software and the enterprise deploying it will realize when they need to use a simple, cost-effective software deployment without all the extras.

    The ability to direct a focused attention to the details of software engineering is another important trait for a DevOps engineer. Otherwise, programming errors occur that can create issues down the road. “If you don’t architect something correctly, you may end up with part of your architecture failing because you made some mistake in programming,” says Gutierrez. These things can propagate and scale up into bigger issues as the architecture grows.

    It’s also important for the DevOps engineer to be strong in security such that their programming does not expose critical data. It’s about how the programmer constructs the data layer, how they work with the transport layer, and what the end to end route is for the data, says Gutierrez.

    Certifications

    Despite the emphasis on traits and approaches to challenges on the job, there are certifications of value to the DevOps engineer. “The Cisco certifications, for example, especially the ones for networking will be helpful since a lot of what you will be doing in DevOps is about data in flight,” says Gutierrez. DevOps engineers need to be mindful of how traffic is moving over the Internet and to and through the data center, explains Gutierrez. Network configuration and network security expertise especially such as is validated through the higher Cisco certifications will be important for the DevOps engineer, Gutierrez adds.

    Application security certifications are also useful as are vendor certifications and courses from Chef, Puppet, and Docker as well as platform vendor training from AWS, Digital Ocean, and the like.

    Individualized Learning

    User documentation and “try me” opportunities for different tools such as containers and configuration management, automation, and deployment tools are helpful for self-starters and people who run into problems or have questions along the way. “People engaging in these kinds of training can simply create local laboratories for these tools on a local machine to simulate the larger application or architecture they are trying to learn,” says Gutierrez.

    In order to learn how to navigate the in-depth areas between the system and application architectures, the DevOps student should consider the books in and around these subjects. Just a few examples include, “DevOps: A Software Architect’s Perspective (SEI Series in Software Engineering)”, “Effective DevOps: Building a Culture of Collaboration, Affinity, and Tooling at Scale”, and “Salt Essentials”.

  • LinkedIn Preps Site Reliability Engineers (SREs) For Exciting Careers

    LinkedIn Preps Site Reliability Engineers (SREs) For Exciting Careers

    Find and Fix, Sink or Swim

    A few LinkedIn NOC engineers including Ariel Casas had a hand in forming what the once fledgling business / professional networking company today calls its Site Reliability Engineering organization. “The main difference between SRE and devops is that SREs are basically developers who happen to be very good at operations. So we call them Site Reliability Engineers,” says Ariel Casas, now Manager, Site Reliability Engineering, LinkedIn, explaining the choice of terminology.

    In those days, circumstances such as outages would thrust Casas and his colleagues into the middle of the excitement, forcing them to devise solutions to the operations challenges at hand. This gave them opportunity to engage developers about how to work together to avoid future issues.

    “Getting thrown into the middle of the storm and having to reverse engineer a service not knowing what it does, not knowing why it’s broken, we basically had to identify the root cause. We had a very, I mean, we still do, but we had a very, very complex infrastructure back when I first started. It was like descrambling an omelet,” says Casas.

    Casas and crew located and scraped through log data, combed through config files, identified service dependencies, and figured out how services worked on the spot in order to understand an unexpectedly broken service. “Those opportunities were a great way for us to learn and figure out how our infrastructure worked,” says Casas, whose early training at LinkedIn was largely on-the-job, experiential learning.

    Early team members like Casas received a lot of hands-on exposure to the various stages in LinkedIn’s application lifecycles, from conception to design to maintenance. Team members touched the application stack at many points and worked with developers to help them understand how the code they wrote would affect the infrastructure that the NOC had to support.

    Playing With Fire

    Speaking of particular fires he and his co-workers had to extinguish, Casas recalls an early example when only a few engineers were on hand as they watched the site, waiting for something anomalous to happen. And something did. “We had a company widget go down,” says Casas.

    The widget served data that LinkedIn collected about specific enterprises that external media outlets were reporting on. In this instance, the CNN news service’s application of the widget stopped working. “We had no idea about the service or how the widget served that data, we simply knew it was broken and causing a poor experience for the customer,” says Casas.

    The team used a Firebug debugging tool to determine the context path for the widget. They connected the dots from there to the service and the port that it was listening on. Then with some log mining relative to that service, they found errors that confirmed that the service could benefit and perhaps fully recover with the aid of some additional memory resources. “We increased the memory for the service, deployed it, restarted it, and it was really rewarding to see the widget behaving the way it should,” says Casas.

    On With the Formalities

    With more than 150 engineers at LinkedIn now, the company has fewer fires to put out and more need for formal training, including external trainings such as conferences and internal trainings at LinkedIn. SREs attend yearly conferences like Velocity, a WebOps-focused event where speakers flock from different organizations to share on topics where they have subject matter expertise. SREcon is another conference and SREs come to seed knowledge and training in their fellow workers there. LinkedIn also sends speakers to each of these conferences as well as attendees. “We also go to Python trainings to learn how to better code and to understand new concepts that we can use to improve our tooling,” says Casas.

    For internal training at LinkedIn the company’s SREs who support select services, such as the internal SaltStack will provide education about developments in those areas. Other training includes Java programming, as the LinkedIn software stack is formed with Java. LinkedIn’s Java SMEs also contribute code back to Java.

    LinkedIn also trains SRE staff on Kafka, the messaging system that supports much of that group’s data streaming. “We have some pretty smart engineers who support, own, and develop Kafka here. They provide good training around how we implement and use it,” says Casas. LinkedIn sends Kafka SMEs to meet-ups to train other enterprises in its use.

    An Open Door to the SRE Career Advancement Ladder

    LinkedIn offers internal trainings with an open door to any high-performing NOC engineer or developer in SRE who wants to attend. “We also have more general trainings called TechTalks and these are high-level and open to anyone in the company,” says Casas.

  • Test-Driven Development (TDD) Drives Smart Developers at Smartling

    Test-Driven Development (TDD) Drives Smart Developers at Smartling

    Supporting Swift Cyber Business Ramp Ups Around the World with Translation and Localization Sped by CD

    Smartling technology enables the business to quickly translate content in websites, apps, and documents into any language in the world. “We streamline all the non-linguistic work associated with the translation,” says Andrey Akselrod, CTO / Co-Founder, Smartling, a translation management platform provider for localization anywhere, globally. Using the Smartling platform, companies upload content, making it instantly accessible by a Smartling translator partner or a translator that the company acquires independently.

    I viewed the Smartling demo and I can tell you that the platform offers many innate, seamless, intuitive abilities. Briefly, it allows the translator to largely automate translation, localization, and content re-use and editing against a backdrop of how it will appear on a website, in an app, and in electronic documents. The platform includes workflow automations to speed translation along to the next stage in the project. “This allows for a reduction in complexity, cost, and time for the translation process, without sacrificing the high-quality translation that only a human can do,” says Akselrod.

    The TDD Testing Secret Sauce

    Smartling developers first write automated regression tests for the code they are about to create. Then they confidently code their software knowing how the tests will challenge the code because they wrote the tests themselves AND did so before they wrote the new feature or innovation. This process accelerates development because the programmers write the code knowing what it takes to ensure that it won’t break anything. This enables Smartling to deploy live code as many as ten times per day, according to Akselrod.

    Test-Driven Development (TDD) saves on refactoring and validates the tests. Smartling discovered this when certain test and development issues disappeared after they moved to TDD. When you code the tests after writing the software that you need to test, you may have to refactor your code so that it is suitable for testing. “You find yourself in a position of doing refactoring without tests to validate that the code still works,” says Akselrod. There’s an old carpenter’s saying that you should “measure twice and cut once” to make sure you don’t waste what you cut or have to cut again. But this sounds like the reverse, expecting the coder to cut (code) twice and then measure. The absurdity in that is self-evident.

    Developers who must create tests after they innovate their software don’t test whether the tests could fail. “This results in low-quality tests. We’ve discovered this the hard way—we had pretty good test coverage, but a number of bugs in the code base led us to discover that our tests were failing,” says Akselrod. TDD resolved all this, saving time, energy, and investment, and satisfying customers.

    Though these automated tests run much faster than manual regression tests do, Smartling gauges test performance over time and refines testing efficiencies to ensure that tests keep pace with accelerating deployments. “We distribute test execution among multiple servers to make sure that the tests run as quickly as possible,” says Akselrod.

    Can Get Enough of That TDD

    Developers find themselves obsessed with the TDD development process for the prompt props they receive as they deploy code directly rather than wait to see it run successfully. “TDD makes for quick customer feedback. Customers are thrilled because we react to their requests quickly. This fosters an exciting and fulfilling conversation between developers and customers,” adds Akselrod.

    TDD’s competitive advantage wets the appetite for market penetration. TDD moves Smartling through the OODA decision cycle at an accelerated rate. “Modern software development cycles through an OODA loop such that developers Observe, Orient, Decide, and Act. If your OODA loop is faster than your competitor’s is, you win,” says Akselrod. TDD enables Smartling to rapidly deploy, respond to the customer base, and instantiate innovations, expediting aggressive market growth.

    Results

    Customers are seizing the benefits that savvy platform provider Smartling serves. As this throng of growing businesses increasingly hitches its wagons to Smartling, Smartling in turn sees three-billion page views per month in traffic flowing through its Global Delivery Network of translated dynamic pages, according to Akselrod. No matter how you slice it, three-billion is a number you simply cannot explain away.

  • American Psychological Association Finds CI & CD Therapeutic

    American Psychological Association Finds CI & CD Therapeutic

    The American Psychological Association (APA), the largest scientific and professional organization for psychology in the U.S. publishes the PsycINFO database with 4-million records and more than 90 peer reviewed journals, books, and other publications, starts Beverly Jamison, Sr. Director, IT Architecture, The APA. “We maintain a database of psychological tests and measures and psychotherapy demonstration videos,” says Jamison. The APA publishes member information and other data as well.

    Because a broad audience including researchers, practitioners, members of other professions, students, the media, and the public use this data, the APA keeps a precise inventory of the data and its elements, checking it for quality and delivering it in a number of ways including as integrated and multimedia materials in B2B and B2C contexts.

    APA Dev and Ops Before & After CI and CD

    When Jamison joined the APA, some odd number of perl / shell-based coders like her used the VI editor as their IDE while others of the ColdFusion persuasion used Dreamweaver as their IDE. Developers deployed direct to the live system, at off hours, with the developers and perhaps one QA person testing AFTER the deployment was completed, says Jamison.

    Today, the APA uses svn for version control for legacy development projects where they are accustomed to that and Git and BitBucket for newer projects that seem to benefit from that. “We use Ant for the scripts to deploy code from version control. For unit testing, we use X-Ray for xquery, JUnit for java, and mocha for javascript,” says Jamison.

    The APA uses the Roxy community tool for deployments to MarkLogic to address MarkLogic database instances and configurations, which provides uniformity of the MarkLogic databases, explains Jamison.

    The nature and purpose of each team inside Dev and Ops determines the usage of the tools. The work of the MarkLogic team dictates one application usage while the priorities of the identity and integration services team require another. The MarkLogic team deals with data services and data models that are often complex; as such they make a great deal of use of unit testing to ensure that a single change doesn’t lead to broken software throughout, explains Jamison. Identity and integration services uses Jenkins to script their builds and to ensure that each build is current; QA analysts test to ensure that their complex scenarios are reliable, says Jamison. Likewise, the UI team has still another set of priorities and usages.

    A Panoramic View of the APA’s New Feature Development Process

    First the developer will check out a branch from subversion to a local IDE, make and then test the altered code and run unit and regression tests on existing code, and update unit tests, configuration scripts, and any script changes needed, explains Jamison. The process would then include updating and tagging any JIRA tickets with the appropriate version number for the forthcoming release.

    The code would deploy from svn to the given IDE for QA testing, which developers or engineers would run depending on the size and complexity of the system and the current system state with respect to release to the outside world, says Jamison.

    “Then the code would deploy to the integrated UAT environment for testing by the business team. Once approved there, we would schedule a production release with the systems engineers handling deployment to live public facing servers,” says Jamison.

    How CI & CD Impacts / Empowers This Development Process

    CI & CD processes hand the developers and engineers greater freedom in adding new features, relieving their fears of doing extensive damage to existing software releases. A minor feature change or bug fix could see the steps in the above process happen in close succession as CI and CD processes avoid the risks of having to revert the code. “A major new feature or change to a data model would mean taking these steps on separate days with distinct and formal signoffs on each,” says Jamison.

    Final Thoughts

    In a word to the wise, Jamison reminds that while tools matter, it is how the teams commit to using them in disciplined CI and CD practices that counts.

  • The Jenkins JUC is coming to Santa Clara

    The Jenkins JUC is coming to Santa Clara

    sacha-laboureydevopsy.kinsta.cloud spoke with Sacha Labourey, CEO, CloudBees about the upcoming Jenkins User Conference, its value, purpose, and focus, and about CD. It’s fitting coming into the JUC to highlight Mr. Labourey’s recent accomplishments. “Last year, we announced the Jenkins workflow, so the ability to build pipelines to create Continuous Delivery environments,” says Labourey. Noting Jenkins’ focus heretofore on CI and the notion of single jobs, Labourey offered that CI was not sophisticated enough to build those pipelines. “So the workflow enabling CD is receiving a lot of attention from the community and customers,” concludes Labourey.

    Moving on to more recent news, Labourey agrees that Jenkins’ support now for Docker is leading the community to consider how it can better leverage Docker. “Or, if they are already using Docker, they want to see how they can streamline the process by which they are going to build those images, register those, test them and so on,” says Labourey. These developments are key to understanding what customers and the DevOps-minded community will likely bring to the JUC, and what they can hope to take away from it.

    Great Expectations

    Labourey has great expectations for the JUC. “First, it’s a nice party. A lot of those people discuss together at the forums or through other means and that’s a great way to connect,” says Labourey. The JUC will be one big multi-topic Jenkins / CD conversation between and among attendees and presenters. There’s something to be said for face-to-face meetings despite the many unified communications channels and presence technologies available today for remote exchanges, collaborations, and white boarding.

    Second, and if the Washington and London Jenkins UCs are any clue, the Santa Clara Conference will see more of what Labourey calls “traditional companies” sending teams to this event to learn new Jenkins features, how other companies applied best practices to solve problems, and what did and did not work for them.

    Addressing the Community: the State of CI &CD

    devopsy.kinsta.cloud specifically asked Mr. Labourey if as we come into the conference he would briefly address the state of CI & CD. Labourey details the changing Jenkins audience and what waits in store for them.

    “The historical users of Jenkins can have a pretty strong focus on Continuous Integration. That’s how the project was born. That’s how it became successful. I think we are seeing a new wave of users now, a fresh wave coming from a Continuous Delivery standpoint looking at the full lifecycle rather than just the development section of it. Sometimes I feel like Jenkins CI is the wrong name. It should be Jenkins CD now because Jenkins CI sends the wrong message. I hope a lot of people can spend quality time thinking about that and educating themselves about those new features because they’re not just new features. It’s Jenkins tomorrow. It’s really the problems that Jenkins will have to solve tomorrow. It’s like a new era for the Jenkins project. I really hope that people attending the conference can realize this.” – Sacha Labourey, CEO, CloudBees.

    The JUC keynote and events will mirror Labourey’s statement in that the content, especially the presentations will include real use cases with innovative deployments and CD as well as technical discussions of how to master new features. “We see that people attending for their second year have done some quality work with Jenkins within their companies. They come back to present their experience in the form of a company use case or some extensions they’ve created,” says Labourey.

    A Vision for Jenkins and CD

    “It’s very important for everybody to realize the success of their company and also the success of Jenkins itself will happen if everybody makes the transition to CD and understands it,” says Labourey. Labourey sees the JUC sessions as covering the new tools that make Jenkins the leader of CD. The JUC committee carefully selected presenters and topics in order to target that goal.

    And since not everyone can attend a JUC, says Labourey, CloudBees as a company will be hiring people that will be much more active in educating the Jenkins community about this new era of Jenkins.

  • A Slice of Groovy’s Hip Use of JFrog Artifactory & Bintray, and of Gradle and TeamCity

    A Slice of Groovy’s Hip Use of JFrog Artifactory & Bintray, and of Gradle and TeamCity

    Grrroooooovy, Baby!

    Groovy supporters describe the Groovy programming language as dynamic and statically typed, supporting functional programming and OOP as a superset that includes the Java programming language. “It’s a kind of Java on steroids if you will!” says Guillaume Laforge, an Apache Groovy project PPMC member. A few examples of common applications of Groovy will help to illustrate its position in the programming landscape.

    Developers use Groovy in the build and test phases of a variety of projects even though they did not code their applications in Groovy. Coders find Groovy an inviting language for writing and maintaining tests due to its reader-friendly concision. “Developers use Groovy with the Spock Framework for unit testing and behavior-driven development and with Geb for Web integration testing,” says Laforge.

    Programmers use Groovy scripts for application extensibility for integration, configuration, customizations, and plug-ins. “The Jenkins Build Flow plug-in is a prime example as it enables users to orchestrate jobs by using the Groovy language to extract those jobs’ flow logic,” says Laforge. Other uses for Groovy include domain-specific languages, administration, automation, and Web, desktop, and mobile application development.

    devopsy.kinsta.cloud explores elements of Groovy’s trip from manual to automated development.

    Manual Processes, Not So Groovy

    When the developers of the Groovy language released software from individual developer machines and using other manual processes, this created several issues. If a developer neglected to give birth to a dedicated release branch, this could equate to non-committed files and local changes transferring directly into the release prior to joining the mainline source tree, skirting prudent source control requirements.

    “Or we could simply forget to set tags for releases or we could commit errant version number data,” says Laforge; “although the release’s version is x.y.z., the version in the manifest would be x.y.w.”

    In some cases with individual developer machines, new developers would experience discrepancies. For example, a new developer performing a project build could see dependency-related challenges such as when one project depends on another that has corrupted “pom” (Project Object Model) files. “Local repositories would fill up with odd dependencies,” says Laforge. There were other issues, as well.

    Groovy Fully Automates Its Release Process

    One of the progressions that Groovy made toward automated bliss was in the realm of version control. Groovy eventually moved from CVS to Subversion to Git and from Codehause to Github as its repository. “More recently, we moved the sources to Apache’s Git, as Groovy entered the Apache Software Foundation incubation process,” says Laforge.

    Today, Groovy developers send snapshots to the JFrog open source Maven repository, Artifactory, to test their bug fixes and features. According to Laforge, Groovy applies the JetBrains TeamCity CI server to build artifacts and distributions; the Artifactory plug-in for TeamCity enables Groovy to stage releases using Artifactory. Artifactory and Bintray integration enables Groovy to push everything to Bintray. Dependency management occurs on Bintray’s JCenter Maven repository. Synchronization with Maven Central supports developers who are retrieving dependencies.

    The Gradle build automation tool supports a mature build setup for Groovy, allowing the language developer to bootstrap its own compiler, enabling it to build Groovy code. “This is part of our deliverable, i.e., Groovy basically compiling itself,” says Laforge; “this would be much more difficult using other build solutions.”

    Drilling down, there are more benefits to having Gradle and TeamCity working in tandem. For one, it affords Groovy the opportunity to test different flavors and functionalities of Groovy on different platforms and environments. “We can do this using different JDK versions from 6 up to the as yet unreleased ver. 9, testing against our own snapshots and using different flags for testing and building Groovy with or without the invoke dynamic support from JDK 7 onward, with different branches,” says Laforge.

    Reinstituting Manual Processes for Apache

    On entering the Apache Software Foundation, Groovy had to re-introduce manual steps in order meet Apache guidelines that state that Apache committers must build projects themselves and that they need to sign their deliverables with their own keys. “We are still using Gradle, JetBrains Team City, and JFrog Artifactory and Bintray, but not in as much an automated fashion as before,” says Laforge.

  • Q & A: AppDynamics on commercial monitoring applications

    Q & A: AppDynamics on commercial monitoring applications

    AppDynamics hired Enterprise Management Associates (EMA) to produce a study about why the enterprise is plagued with numerous commercial monitoring applications that are not in use. According to AppDynamics, the enterprise is encumbered with disparate applications, resulting in vast challenges for IT teams including underperformance, monitoring gaps, and siloed tools—not to mention that many IT managers do not have time to address these application-based issues.

    AppDynamics emphasizes what it identifies as a finding that almost two-thirds of respondents report that it takes at least three hours to determine the root cause of an application performance issue. AppDynamics concludes that very few IT organizations have an accurate, comprehensive view of today’s complex applications environment, business transactions, and dependencies.

    devopsy.kinsta.cloud spoke with AppDynamics’ VP of Product Marketing, Kalyan Ramanathan about this study.

    devopsy.kinsta.cloud: How did you select and reach study participants?

    Kalyan Ramanathan: We worked with the analysts from Enterprise Management Associates (EMA) to do this survey and study. They have a standard database of people who they always work with, who they interact with. EMA did the heavy lifting, surveying the audience for the study. We did, however, put criteria in place in order to make sure that the survey is relevant.

    We made sure that the survey targeted enterprises of all sizes. So as you may have read, 80-percent of the surveyed respondents had more than a thousand employees, which is about the size where companies start to have some mission-critical applications that might affect a substantial number of either internal employees or consumers when they don’t work well.

    We also wanted a smattering of fairly large customers, or I should say enterprises, only because the challenges that people at that spectrum have are very different from let’s say an enterprise with a thousand employees. So 25-percent of the surveyed respondents had more than 10K employees.

    The other thing that we did as a part of the survey was to focus on the personas that we are talking to, so that again we get good representation across the board. So we had everybody from practitioners to managers to IT execs in this survey because the interesting thing that we found out is that there are some questions where we see a fairly wide discrepancy between these personas.

    DevOps: You say that the survey shows that respondents need flexible deployment options. Did respondents say why in particular they needed flexible deployment options?

    Kalyan Ramanathan: Absolutely, and I think you’re specifically referring to the fact that the solution can be available on premise or in the cloud, and just about every enterprise that we talked to believes that this is a key requirement and it has become a key differentiator for AppDynamics in the market. Many of the companies that we talked to are, I would say, Fortune 1,000 to Fortune 2,000 companies, and in these companies not all their assets are in the cloud today, not all their applications are in the cloud today, and they also deal with a lot of sensitive data.

    So today many of them may choose to work with our on-premise solution, but at the same time I think the writing’s on the wall. I think none of us can question the fact that the cloud is indeed gaining acceptance and that some or many of these applications will indeed go to the cloud at some point. So having that option, having the flexibility that one day my applications will go to the cloud and therefore I could use a cloud-based monitoring solution, without doing a forklift upgrade or replacement of everything that I have, is indeed a good flexibility to have and that’s essentially what our customers see in the AppDynamics product.

    Final Thoughts

    Consider this exchange, dear reader, as enlightenment into what went into this study or as a backdrop for your examination of the published survey results from AppDynamics if you care to read them. You can get the full study here.

  • Jut Operations Data Hub for DevOps

    Jut Operations Data Hub for DevOps

    “You have to be able to ask questions of your data in real-time and then respond to the answers,” says Apurva Dave, VP of Marketing, Jut. Jut leverages live streaming, batch analytics, and visualization to enable enterprises to ask important queries of their operational data using the Jut operations data hub, which Jut created with DevOps goals (such as Continuous Deployment with Quality) in mind.

    Jut supports real-time and historical data queries across all log, event, and metrics data. “Our platform is built on dataflow, dealing with large scale streaming data,” says Dave. Jut is an option for organizations that don’t want to go out and build their own operations data hub software.

    Core Data Types

    Jut intends to join data such as user activity (on websites and software), event data (triggered operationally such as support ticket filings, customer transactions, and software upgrades or migrations), and unstructured data (application and system logs) together for analysis, says Dave. Jut works with structured data such as metrics data (such as application level data like the user numbers and key action numbers or system level data like CPU utilization, memory utilization, and network I/O data), explains Dave.

    Jut attempts to analyze the whole of your data to answer questions you may have like: “What resulted when I deployed a new version of my application into production?” Jut might answer based on an analysis correlating system resource utilization (“Was the app too resource intensive this time? More so than we expected?”).

    Or you may ask Jut this question: “How did my code respond to user activity / user demand? How did it perform?” Jut could ask this question based on its assimilation of both unstructured and structured data. “It’s common for enterprises to A/B test two separate versions of a site or software feature. Using Jut, you can ingest those test results and analyze them in real-time to determine what version produces the desired effect,” says Dave.

    Big Data Analytics & a Huddle about Juttle

    “We have users ingesting tens of billions of data points per month. That number is growing fast,” says Dave. Jut targets big data management for easier analysis using data infrastructure techniques that simplify / streamline, integrate, and manage big data backend infrastructure. “We also use our own dataflow language to quickly iterate and visualize lots of data,” says Dave. All this is designed to ease DevOps teams’ work in accessing and assessing operational data to discern how the software and the business that it serves are performing.

    Jut designed its Juttle dataflow programming language for data analysis and data visualization. And Juttle is dedicated to those tasks alone. “While you could try to cobble together the right packages and libraries in a general language like Python or Java to do these analytics, it would be relatively messy, time-consuming, and error prone,” says Dave.

    Juttle is an extensible language enabling analytics and visualization. “Jut’s data scientist built an anomaly detection algorithm entirely in Juttle and then published it on GitHub for anyone to use inside Jut,” says Dave.

    Juttle uses a structured approach to querying your data. “The approach looks like a series of piped UNIX commands, so anyone familiar with scripting languages such as JavaScript or query languages such as SQL should feel at home with Juttle,” says Dave. Jut responds to questions in seconds to tens of seconds (for larger data sets) based on iterative, interactive analysis.

    Investigate, Question, Consider (Try?)

    Outsourcing to Jut puts the enterprise in a position to focus on existing coding duties while proving whether Jut is the better operational analytics move for analytics performance and cost-savings (Jut offers a free trial). But as with the analytics themselves, ask a lot of questions of Jut. Do your homework before committing even to trial software (Jut has a demo and an online “playground” as well).

    Tip: Jut made sparkling comments from its Beta-development-phase customers such as NPM and Mylio available. Rather than include those for your consideration, I suggest you reach out through your connections to these and other enterprises that are experimenting with this data hub software.