devopsy.kinsta.cloud

Tag: datadog

  • Datadog Monitoring and JIRA Integration

    Datadog Monitoring and JIRA Integration

    Operations uses monitoring to provide early detection of issues with systems/servers/networks. Among those monitoring options is Datadog, a software-as-a-service (SaaS) cloud service/application monitoring tool that allows Operations to be notified of issues with their applications in near-real time. Development teams, meanwhile, use JIRA to track tickets and manage overall agile development.

    But what if your product is the very thing that Operations manages? The sweet spot for DevOps is that location where the app is internally developed, and managed by Operations, because the entire stack is under IT management.

    That’s why Datadog and JIRA are now integrated.

    Datadog can monitor an infrastructure and application on a variety of levels at a variety of locations in the infrastructure stack, from disk availability to response times to memory usage. Datadog has the ability to monitor and emit alerts when things are abnormal or out of specified ranges, or they outright fail.

    JIRA is Atlassian’s Agile project management software, with issue tracking and all of the usual agile tools in one place.

    From Datadog Monitoring to JIRA ticket.
    Configuring Monitoring’s alerting to JIRA.

    Now, Datadog can create JIRA tickets, update them as more information becomes available and populate them with relevant information from the running systems.

    Say, for example, one cloud instance runs low on memory and passes over a threshold. Datadog detects the issue and opens a ticket in JIRA that shows what system and when, as well as whatever other information Operations has defined. That ticket is now in the system like any other, and a developer can grab it and look at memory use for the portions of the overall application that run on the instance in question.

    According to the press release, these are the three automations included in this integration:

    1. Creating JIRA tickets when performance issues are detected within Datadog.
    2. Populating JIRA tickets with pertinent information about identified issues.
    3. Updating existing JIRA tickets when new information surfaces.

    I’m an analytical person, so my first thought was, “How cool,” and my second was, “But what if it spams JIRA with non-development issues?” Since Operations configures what goes to JIRA, and Datadog supports alert rollups (multiple alerts in one or two notifications), it is possible to keep such spamming under control. An overzealous Operations staff initially might configure the reporting to send too much, but as experience grows, this should become a viable way to automate bug reporting. In particular, tagging of Datadog-generated events should show quickly what alerts are creating too many tickets, and give the team opportunity to change the way tickets are created—or, if it turns out they’re created because of a serious application-impacting error, raise the priority of the fix.

    Which brings us to the updating status part. In Datadog, operations can leave a comment on an issue with an “@jira-update” tag and JIRA will get notified of the comment. That means Operations can raise awareness of an issue that may seem mundane but has larger impact than is obvious, or add detail to a JIRA ticket if needed, all from within Datadog.

    It is interesting that Datadog is also integrated with BitBucket, another Atlassian product, so it is possible that opened tickets can be marked as “in test” when an integration build includes reference to the ticket. In short, BitBucket integration means that Datadog’s monitoring is aware of commits and checkouts, and Operations staff will be able to correlate that to issues that occur. Adding in JIRA integration means that Operations sees the commits, gets the alerts and updates JIRA. The wealth of information is such that you’ll want to use their various filtering techniques just to make certain you’re getting the important stuff.

    There are some areas that are intriguing to me, such as the impact of a particularly complex set of queries/reductions on a big data platform. I haven’t tried the Hadoop integration, but it appears that this agent would be able to communicate that a given reduction is the source of the problem to the JIRA interface. Even if these complimentary integrations aren’t there yet, you can feel it coming to where “This reduction caused systemwide issues” is what is generated in JIRA. That’s pretty cool.

    Overall, the integration of Datadog and JIRA is a DevOps dream come true, tying Operations closer into Development, and allowing Development changes to be tracked closer in Operations. After all, the point is to keep the systems running as close to peak efficiency as possible, and automate as much as can be. The issues spawning from Datadog alerts into JIRA tickets also pretty much guarantees increased need for development and operations to talk, as problems that crop up by the automated system are discussed and researched.

    If you’re dedicated to a cooperative environment for Dev and Ops—even if you aren’t calling it DevOps—the Datadog/JIRA integration is worth a look.

    — Don Macvittie

  • Every (Data)dog Has its Day

    Every (Data)dog Has its Day

    Bienvenido, Datadog! You have truly made it to the big leagues, as you join a growing list of reputable companies that have suffered an apparent data breach/security incident. Many people view this as a dreaded moment, one that should always be avoided. However, it’s almost as though nowadays it’s a rite of passage for a company to be hacked at some point in its lifespan. Call them growing pains, if you will.

    Datadog is a software-as-a-service (SaaS) company that helps monitor and collect analytics for customers, which include the likes of Facebook, Spotify, Salesforce and Citrix.

    As with any data breach, Datadog is recommending users change their credentials immediately—a common practice among down-on-their-luck dogs. This recommendation applies only to stored passwords; however, those enlightened few who use Google Auth or SAML should be OK. The notice also directs admin users to revoke or change credentials stored in the Datadog system to prevent fur-ther security breaches and unauthorized access.

    Obviously, data breaches should be avoided at all costs, and can be seen as a hit to a company’s bottom line. But Datadog stands on solid ground here following the attack. Its ruff-and-tuff password storage method, bcrypt, which uses a unique salt, is one of the strongest ways to keep credentials and slow decryption efforts, according to Datadog Chief Security Officer Andrew Bechere.

    Fur-tunately for Datadog, at least right now, its security procedures and bcrypt system have prevented what could have been a major disaster. The company has informed its customers that the “known” attack vectors used by the hackers have been patched.

    It is my utmost opinion that we throw Datadog a bone here as we continue to see hacks and breaches become commonplace among tech companies. Not only has it become quite common, but Datadog cur-tailed the damage of the breach by implanting bcrypt security encryption.

    Oh, you’re a stickler for your information remaining private? I hate to break it to you, but it’s the internet. Nothing is sacred and private once it goes online. However, you can take measures and increase security to make it extremely difficult and lengthy for hackers to obtain information. Still not good enough for you? Let’s just take a look at other companies that have suffered hacks.

    Sony suf-fur-ed a tremendous hack last year that listed executive emails and personal information including addresses and salaries. Home Depot suffered a hack in September 2014 that compromised 53 million email addresses. And let’s not forget about the JP Morgan hack affecting 83 million customers, the eBay hack that compromised 145 million active users (including login credentials and physical addresses) and the 2013 hack of Target, which exposed personal and financial information of 110 million customers.

    So enough pup-arazzi for Datadog, don’t terrier yourself up about it. Every once in a while a company suf-fur-s a data breach. What can I say? The puggle is real out here.

    I’m sorry for all the puns. In related news, Datadog issued a press release, which you can find here.

  • Datadog Expands Monitoring Solution Across Stack for Microsoft Customers

    • The leading cloud-monitoring service Datadog announced their latest integrations with Microsoft Azure Resource Manager and Microsoft Azure SQL Database
    • Datadog agent allows customers to seamlessly monitor their stack, from cloud hardware down to custom metrics for their application

    NEW YORK–(BUSINESS WIRE)–Datadog, the essential monitoring service for dynamic cloud infrastructure, today announced expanded support for the Microsoft Azure cloud platform. Following the release of Microsoft Azure Resource Manager (ARM) – the latest way to deploy virtual machines on the platform – Datadog will now support Microsoft Azure SQL Database as well as cloud servers deployed through ARM. This will empower large enterprise companies running their infrastructure in the cloud to correlate metrics from these new services with application performance. The new capabilities will be demonstrated in the Datadog booth 1724 at the 2016 Microsoft Worldwide Partner Conference taking place July 10-14 in Toronto, Canada.

    “We make it a top priority to build out integrations for new services from major cloud platforms as they are released,” said Amit Agarwal, Chief Product Officer, Datadog. “By utilizing Datadog, enterprise customers are now able to effectively manage the performance of their large scale Microsoft Azure-hosted applications.”

    Datadog already boasts a number of Microsoft integrations, from monitoring the virtual machines deployed on Azure, to collecting metrics and events from native windows services such as Windows Management Instrumentation and Event Viewer.

    Datadog is compatible with the following Microsoft hardware and middleware:

    Hardware

    • Classic Virtual Machines
    • VMs deployed through Azure Resource Manager
    • SQL Database

    Middleware

    • Windows
    • Internet Information Services
    • SQL server
    • Windows Event Viewer
    • Windows Management Instrumentation

    Application Custom Metrics

    • StatsD for C#

    “Datadog is a key monitoring tool for enterprise companies embracing the cloud,” said Eduardo Enciso, Global Business Strategist, Microsoft Corp. “Their new integrations with Microsoft Azure allow our mutual customers to use the capabilities in Microsoft Azure VM and Microsoft Azure SQL Database, with full visibility across multiple Azure services and applications.”

    Datadog is available for one-click deployment through the Azure Marketplace and also from the Microsoft Azure Portal. They will be demoing their latest integrations at Microsoft’s Worldwide Partner Conference in Toronto.

    About Datadog

    Datadog is the world’s leading monitoring service for cloud-scale applications, bringing together data from servers, databases, tools and services to present a unified view of your entire stack. These capabilities are provided on a SaaS-based data analytics platform that enables Dev and Ops teams to work collaboratively to avoid downtime, resolve performance problems, and ensure that development and deployment cycles finish on time. Since launching in 2010, Datadog has been adopted by thousands of enterprises, including Twilio, Airbnb, Netflix, EA, Spotify, Warner Bros. Games and AdRoll.

    Contacts

    Mindshare PR
    Heather Fitzsimmons, 650-279-4360
    heather@mindsharepr.com

  • CloudHealth Partners with Datadog, Connecting Cloud Performance Metrics to Cloud Costs for Enterprise and Fast-Growing Technology Companies

    The New Integration Enables Customers to Rightsize Complex Cloud Environments

    BOSTON–(BUSINESS WIRE)–CloudHealth Technologies, the leader in Cloud Service Management, today announced an integration partnership with Datadog, the leading monitoring service for dynamic cloud applications. The integration of the CloudHealth and Datadog platforms offers customers end-to-end visibility across every dimension of their cloud environments, giving them recommendations and analysis to make analytically driven rightsizing decisions in the cloud.

    While the decentralized management and dynamic provisioning of cloud resources greatly improves application development agility, it creates new challenges for organizations, related to cost allocation, infrastructure rightsizing and governance. The CloudHealth-Datadog partnership addresses the major challenge enterprises face as they embrace agile processes and cloud-based infrastructures: an inability to effectively manage cloud resources at scale.

    Datadog provides a SaaS platform that monitors data across the entire infrastructure stack and presents that data in elegant and easy to understand dashboards, allowing for actionable insight. The CloudHealth platform simplifies cloud management, enabling enterprises to consolidate, evaluate, analyze, optimize and automate their clouds.

    By connecting cloud metrics to costs, the integrated platforms allow customers to manage assets in a more strategic and holistic fashion. CloudHealth Technologies and Datadog bring together granular performance data, end-to-end visibility and powerful analytics to help customers continually make the best decisions about provisioning and rightsizing in the cloud. The CloudHealth-Datadog partnership provides customers with the following benefits:

    • Rightsizing analysis: Customers can use data from both solutions to receive recommendations about where to adjust resources that are over- or under-provisioned, saving them time and money.
    • Visualization of key assets: Using CloudHealth, customers can zero in on the cloud data they care about – looking by department, line of business, or any relevant Perspective. Using Datadog tags, they can get a centralized view of all their assets in one place alongside other tools and services.
    • Advanced performance metrics per asset: Customers can maneuver between the tools to really understand their environments. They can jump into CloudHealth to track trends or scan through performance data or jump over to the Datadog console to explore individual assets in more detail.

    “To optimize your cloud resources, you need to have tools that help you visualize your data, understand it and act on it,” said Joe Kinsella, CTO and founder of CloudHealth Technologies. “Putting Datadog’s best-in-class cloud monitoring assets together with CloudHealth’s Cloud Service Management capabilities gives customers the ability to truly solve the issue of cloud complexity.”

    “The cloud cost allocation and optimization is a significant challenge for many of our enterprise customers who spin up tens of thousands of cloud instances across various business units,” said Amit Agarwal, Chief Product Officer at Datadog. “We look forward to partnering with CloudHealth to provide customers who use both products a solution that helps customers quickly allocate and minimize costs, while ensuring that their critical applications are adequately provisioned at all times.”

    Many joint customers are already using CloudHealth and Datadog together, including Elastera, a provider of managed cloud hosting environments for ecommerce sites. The company moved to the cloud to innovate and get to market faster and adopted CloudHealth and Datadog to get accurate rightsizing recommendations for its AWS infrastructure.

    “CloudHealth takes information collected from Datadog and connects the metrics to the money. It shows us if an instance is under-utilized and how much we could save by making a change,” said Zachary Stevens, co-founder and chief architect, Elastera. “Using this powerful set of data analytics, we are able to downgrade over-provisioned assets, saving us time and money.”

    About CloudHealth Technologies
    CloudHealth Technologies gives organizations a new, more efficient way to visualize, optimize, and automate their cloud environments. The CloudHealth Cloud Service Management platform integrates with services across the datacenter and cloud, giving enterprises, startups, and service providers access to a centralized hub of robust data analytics. Using a policy-driven approach aligned to business processes, customers leverage CloudHealth to drive cost savings, improve agility, enhance security, and reduce complexity. With CloudHealth, customers are able to innovate faster without sacrificing governance or control.

    The world’s cutting-edge companies rely on CloudHealth to meet their cloud management needs, including Amtrak, Dow Jones, Acquia, and Sumo Logic, among others. Based in Boston, MA, the company is backed by Sapphire Ventures, Scale Venture Partners, .406 Ventures and Sigma Prime Ventures. For more information, visit www.cloudhealthtech.com or follow us on Twitter @cloudhealthtech and LinkedIn.

    About Datadog
    Datadog is the world’s leading monitoring service for cloud-scale applications, bringing together data from servers, databases, tools and services to present a unified view of your entire stack. These capabilities are provided on a SaaS-based data analytics platform that enables Dev and Ops teams to work collaboratively to avoid downtime, resolve performance problems, and ensure that development and deployment cycles finish on time. Since launching in 2010, Datadog has been adopted by thousands of enterprises, including Twilio, Airbnb, Netflix, EA, Spotify, Warner Bros. Games and AdRoll.

    Learn more about the integration on the CloudHealth Tech blog and the Datadog blog!

  • Datadog Adds Hadoop and Spark Integration to Leading Cloud-Scale Monitoring Platform

     Users can now monitor the four most widely-used technologies in the Hadoop ecosystem: HDFS, MapReduce, YARN, and Spark

    NEW YORK – May 24, 2016 – Datadog, the leading SaaS-based monitoring platform for cloud applications, today announced support for Hadoop, bringing its unified view of applications to the Hadoop ecosystem. Hadoop users can now benefit from Datadog’s rich dashboards, full stack visibility and correlation, sophisticated and targeted alerts, collaborative tools and integrations, and easy setup with no maintenance. Integrations can be turned on immediately, adding to the long list of technologies DevOps teams can monitor easily and collaboratively with Datadog.

    By adding the power of Datadog to Hadoop, users can now see hundreds of Hadoop metrics alongside their hosts’ system-level metrics, correlating what is happening within Hadoop with what is happening throughout their stack. Users can also avoid problems by setting alerts when critical jobs don’t finish on time, on outliers or any other problematic scenarios.

    This product update supports:

    HDFS: Using the HDFS integration, users can monitor the number of data nodes and blocks, disk space remaining on each host and cluster, as well as namenode load and lock queue length.

    • MapReduce: The MapReduce integration includes metrics for map and reduce jobs pending, succeeded and failed, bytes read by job or in total, as well as input/output records.
    • YARN: The YARN integration gives visibility to nodes, applications, cluster cores and cluster memory.
    • Spark: With the Spark integration, users can see driver and executor, RDDs, tasks, job stages and more.

    Since launching in 2010, Datadog has been adopted by thousands of enterprises, including Twilio, Airbnb, Netflix, EA, Spotify, Warner Bros. Games and AdRoll. For additional information on Datadog, please visit www.datadoghq.com.

    Supporting Resources

    Company Blog

    Twitter: @datadoghq

    About Datadog

    Datadog is a monitoring service that brings together data from servers, databases, applications, tools, and services to present a unified view of the applications that run at scale in the cloud. These capabilities are provided on a SaaS-based data analytics platform that enables Dev and Ops teams to work collaboratively to avoid downtime, resolve performance problems, and ensure that development and deployment cycles finish on time.

    #

    Media Contact

    Mindshare PR

    Heather Fitzsimmons, 650-279-4360

    heather@mindsharepr.com

  • Migrating to the Cloud? Some practical advice

    Migrating to the Cloud? Some practical advice

    Oh if the world was only Greenfields. If everything was new and we didn’t have to worry about legacy systems, what a wonderful world it would be. But that is not the way of it unfortunately. More times than not using the cloud for our applications and infrastructure means migrating some existing infrastructure from an on premises install up to the cloud. How do you do that successfully?

    You have to break some eggs to make an omelet, but migrating to the cloud isn’t like whipping up breakfast. We can’t afford down time, broken links or broken apps. As more and more apps are moved to the cloud, a solid process needs to be put in place. I asked a few people I know to help me put together a checklist of things to consider and do in moving from on prem to the cloud or more likely deploying a hybrid environment with both on prem and cloud.

    I asked my friend JP Morgenthal who knows a thing or two about cloud migration and DevOps and I dug in a bit with Alex Rosemblatt from Datadog. Datadog actually has quite a bit of experience in this area. Their product/service monitors both cloud and on prem servers side by side, so they deal with this very situation quite a bit. Taking what both of these gentlemen had to say, here is a checklist that should help you:

    1. Evaluate application for functional limitations to running in the cloud. This could include operating system dependencies, storage performance, network bandwidth/performance. Some of the key requirements to note:
      CPU
      Memory
      Storage
      Attached storage vs block shares
      IOPS
      DR
      Performance
      Underlying hardware

    2. Choose your method of migration: Lift-n-drop or re-platform. Lift-n-drop is simply porting the application to the cloud without modifying the architecture and providing all the application components. Re-platforming takes into account using cloud services where possible in place of existing application components.

    3. Pick an IAAS vendor, some criteria to use:
      Available functionality – auto-scaling, additional services etc
      Instance type/AMI availability
      Regional zone distribution
      Support SLAs
      API availability
      Performance metrics/log access

    4. Put in place a migration project management plan that includes:
      Cutover date selection
      Testing project plan
      Acceptable performance benchmarks
      Cutover day ops
      Internal communication and additional training
      Ops team
      Dev team (s)
      Changes in app dev to account for new functionality?
      QA team
      Management
      Non-tech org teams affected

    5. If hybrid architecture (and it should probably be hybrid), select how the existing data center will integrate with the cloud service. This could be Layer 2 networking through telecommunications provider or Layer 3 through Internet. Now would also be a good time to think about how you will monitor both on prem and in the cloud. Again Datadog does this side by side.

    6. Establish the network scaffolding for the application. Develop the allowed pathways for various application components to communicate. This will simplify the deployment of application components later. Test that the right ports are open for sending and receiving.

    7. Deploy the application in a single subnet with full Internet access. Remove any barriers to ensuring the application will operate on the cloud service provider’s resources. This is simply a test environment to make sure the baseline components will operate and communicate with each other.

    8. Deploy the application into the appropriate subnets to ensure servers are not available via the public Internet.

    9. Port the data from the current application environment

    10. Perform a complete regression test of the system

    11. Add additional testing to see how the system will perform if cloud services become unavailable

    This is obviously not an exhaustive list, but should give you a start in your planning. Eventually you are going to have to make a cloud migration if you have’nt already. Better to start planning for it now.