devopsy.kinsta.cloud

Tag: Atlas

  • Get Your DevOps Right With These 7 Technologies

    Get Your DevOps Right With These 7 Technologies

    Editors’ Note: As with all of our contributed articles, the views expressed in this article are of the author, not devopsy.kinsta.cloud. Due to its vendor-agnostic stance, devopsy.kinsta.cloud does not endorse any particular technology.

    There was a time when the operations team and the developers regularly were at each other’s throats. The developers were never satisfied with production code updates and the server administrators chafed at developers piling up the serve requests and circumventing the processes.

    However, the battle lines have been somewhat erased since the advent of DevOps. IT teams adopting DevOps can move rapidly from concept to production to enhance the business impact, quality and velocity of the application delivery. Many organizations are shifting to DevOps because of its high efficiency and the number of people taking DevOps training are also increasing due to numerous job opportunities in this field.

    DevOps Technologies to Know

    A number of technologies have emerged to help in the shift to DevOps, providing services ranging from the configuration management to application portability. Following is a list of seven technologies worth noting.

    Splunk

    Splunk is used for detecting and fixing issues across the life cycle of an application in real time. It allows the developers to see data from the production environments, eliminating the need to access production machines. Splunk enables DevOps processes including continuous integration and deployment.

    The Splunk platform is used for DevOps to:

    • Provide end-to-end visibility across each and every component of the DevOps tool chain.
    • Utilize the objective metrics to make sure the code meets the quality SLAs and is operational.
    • Correlate the business metrics with changes in code to obtain new business insights.

    ScriptRock GuardRail

    ScriptRock Guardrail is designed to rapidly pinpoint the differences in configurations on desktops and servers, comparing the states between systems and over time. ScriptRock Guardrail shows what has changed, helping simplify diagnostics and decreasing downtime.

    Developers who work with agile methodologies are accustomed to a test-first way of working with the unit tests, making sure that everything works as planned. As continuous delivery becomes more prevalent, the DevOps and Ops teams need to take a similar approach. This is where the ScriptRock GuardRail comes into its own, providing you the tools to construct and build infrastructure level unit tests and view the results visually, thus making it very easy to rapidly drill into the real problem and get your services running.

    ScriptRock GuardRail focuses on addressing the key prerequisites to any DevOps, and, more precisely, initiative and automation. They are described below.

    • Visibility: The GuardRail platform delivers visual configurations showing differences between like devices, configuration drift monitoring and automated visualization and scanning of network device and server configurations.
    • Accountability: Users can curate and filter configurations to generate policies that can be utilized to ratify the configurations on a continuous or ad hoc basis.
    • Automation: All the policies can be exported to Windows PowerShell DSC, Docker, Ansible, Puppet and Chef.

    ScriptRock GuardRail works with any automation tool and can be delivered as single-tenant SaaS, multitenant SaaS or a virtual appliance on-premises.

    SaltStack

    The SaltStack Enterprise Software helps DevOps organizations by organizing the systematic motion of code into production by maintaining the complicated infrastructures refined for optimal application delivery and business service. While DevOps is not a tool, it requires a tool-centric approach to enable continuous delivery. SaltStack helps to configure as well as quickly deploy the dynamic applications and general-purpose infrastructure they run on. DevOps is gaining wide popularity around the world because of the tools like SaltStack that enhances its efficiency.

    SaltStack is a cloud infrastructure tool and delivers the systems management for application configuration, cloud building, server provisioning and data automation. It is an event-driven tool and can automate the tasks within DevOps workflow and makes the operations highly approachable for the developers. SaltStack provides a common language and common way to manage the servers and thus helps to bridge the gap between Devs and Ops.

    Puppet

    Automation is the foundation for many DevOps practices. It is the key to continuous delivery and faster deployments. Automation imposes consistency across the production, testing and development environments. It makes sure users can rapidly orchestrate the changes throughout the infrastructure: container and cloud deployments, virtual machines and bare-metal servers.

    No matter where it runs, Puppet provides a standard way of operating as well as delivering a software. Users define what they want their infrastructure and apps to look like utilizing an easy-to-read common language. From there, they can enforce, test and share the changes they want to make across their data center. Reporting and visibility is available at every step of the way to help prove compliance and make decisions.

    The Puppet Enterprise enables users to test changes before deploying them to production. With Puppet Enterprise, the infrastructure is defined as code, which enables users to adopt the best practices of agile development including automated deployment and testing, and deliver the software more reliably. This is because the configurations are consistently promoted, shared and tested across all environments, from Dev laptops through test to production.

    Atlas

    Atlas was the first commercial product released by HashiCorp, provider of open-source tools that enable the efficient deployment of software-defined applications and data centers. Atlas provides visibility into the infrastructure, including virtual machines, containers and servers, in addition to service discovery and configuration management. Building on the popular open-source projects of HashiCorp such as Terraform, Consul, Serf, Packer and Vagrant, the closed-source proprietary Atlas enables DevOps across different cloud services, which include OpenStack, Azure, Google Compute Engine and AWS, and provides a dashboard for maintaining, deploying and developing applications.

    Many engineers are shifting to Atlas today for their organizations’ customer engagement platforms. Atlas brings it in one place and enables the companies to handle end-to-end starting from configuration management to container deployments across various cloud environments. Every open-source tool is fine alone, but the capability to handle and orchestrate them as a whole is extremely powerful.

    Chef

    Chef is a cloud infrastructure and systems framework that automates the management, deploying and building of the infrastructure via repeatable, short scripts known as “recipes.” But the actual strength of Chef lies in its utilization of the pluggable configuration modules(also known as cookbooks), about 2,000 of which are available through Chef community. Facebook, the high-profile user of Chef, recently open-sourced few of its own Chef cookbooks, which include its Grocery Delivery and Taste Tester testing framework, which watches a source code repository such as Git and keeps the local Chef server in sync.

    The complex tasks are automated by Chef that are otherwise resource- and time-intensive. But, more importantly, Chef allows users to focus their efforts on improving and innovating the quality of their services. It also opens the door to more efficiency and collaboration across the organization.

    Docker

    Through its containerization technology, Docker brings portability to the applications wherein the applications run in self-contained units that can be moved across the platforms. It consists of Docker Hub, a cloud service for workflow automation and application-sharing and Docker Engine, which is a lightweight packaging and runtime tool.

    Docker has revolutionized the containerization of applications enabling DevOps to package any kind of application in a lightweight environment. The packaging is done in a way as easy as installing a mobile app. Docker is considered as the most popular DevOps tool. By using docker, the developers can containerize their apps and can run them anywhere. Built on top of Linux containers, docker is a powerful ecosystem where DevOps can use many apps (available on Docker Hub) to the maximum advantage.

    About the Author / Savaram Ravindra

    Ravindra was born and raised in Hyderabad, India, popularly known as the ‘City of Pearls’. He is presently working as a Content Writer at Appmajix Technologies. His previous professional experience includes Programmer Analyst at Cognizant Technology Solutions. He holds a Masters degree in Nanotechnology from VIT University. He enjoys spending time with his friends. He can be contacted at savaramravindra4@gmail.com. Connect with him also on Twitter and LinkedIn.

  • MongoDB Atlas: One Giant Leap, But More Steps Needed

    MongoDB Atlas: One Giant Leap, But More Steps Needed

    Atlas, holding up your databases in the cloud. I’m not sure if that’s what MongoDB was going for, but I personally like the analogy.

    At MongoDB World’16, MongoDB announced Atlas, a database-as-a-service offering (in the cloud) of the company’s popular open-source NoSQL database. Rather than dealing with the complexities of setting up, provisioning and configuring a MongoDB cluster yourself, Atlas allows companies to leave all those details for the masters of MongoDB to manage, freeing up application developers to focus on more important things (such as application development). Atlas currently is available on Amazon AWS, but there are plans to expand it to other public cloud platforms such as Microsoft Azure and Google Cloud Platform.

    Depending on the needs of your application, Atlas allows you to scale your cluster up or down, both vertically (with more powerful servers) and horizontally (with more shards in your cluster). You can also provision more disk space as your data grows. Simply tell Atlas and it will automatically update your cluster in the cloud to match your specifications. The documentation is unclear whether Atlas leverages AWS’ autoscaling features, but it appears that scaling must be done manually.

    Bringing Up a MongoDB Cluster

    When you first log in to Atlas, you’re met with a pretty simple user interface (UI). The very first thing you can do is to start up a new cluster in the cloud.

    2Before deploying a cluster, you need to know the AWS region to place the cluster, AWS instance size, amount of disk space needed, whether to use encrypted storage, whether to enable backups, the speed of the underlying storage (i.e. number of IOPS) and the number of shards and replica set members.

    You first specify the type of AWS instance you want to run your cluster on (ranging from M10 to M100) and how much disk space you want to provision, anywhere from 10GB to 16TB. All instances can be configured with encrypted storage with no additional cost. I first started by selecting the default settings from the M10 instance, which also happens to be the cheapest.

    3Instance sizes, specs, and configurable options. Details including cost are consolidated on the right.

    After specifying your instance, you can then configure the number of nodes in your cluster. For replica sets, you can have either three, five or seven members. Additionally, you can specify whether the cluster should be sharded and the number of shards in the cluster. Note that sharded clusters requires an AWS instance of M50 or higher, which is one of the more expensive instances, so the jump from a basic unsharded cluster to sharded cluster will be fairly expensive!

    4

    5The cheapest sharded cluster is 56 times more expensive than the cheapest unsharded cluster. You probably don’t want to be using Atlas for any sharded testing clusters if you can help it.

    After you’re satisfied with your cluster configuration, you can just click “Confirm & Deploy.” My cluster was spun up within a couple minutes, but your mileage may vary. After adding the IP addresses from which you’ll access the Atlas instances to a security whitelist, you can begin using your new mongo cluster. Something small that I really appreciated about Atlas is that it gives you the exact command and hostnames you should use to connect to your cluster (both through the mongo shell as well as through MongoDB’s many drivers). It makes connecting to your cluster incredibly simple. Just copy and paste.

    6Atlas gives you high-level stats of your cluster, including IOPS, read and write performance, disk usage, etc. You can also get more detailed metrics for each node.

    7Connecting to your cluster made easy! Just download an updated mongos binary and then copy and paste.

    After creating your cluster, it is fairly simple to scale your clusters horizontally by adding more shards. For me, the whole process of upgrading my 1×3 to a 2×3 cluster took around 10 minutes (again, your mileage may vary). Pretty slick! In this process, Atlas will automatically provision more servers and configure your cluster with new config servers, mongos and shards. Notably, once you decide to shard a cluster, you are unable to go back down to an unsharded instance, so make sure that you’re absolutely sure when you decide to scale up to a sharded cluster—you won’t be able to go back!

    Overall, outside of a few unsupported cases, the process of setting up a cluster and manually scaling it was quite smooth. While what has been achieved thus far in reducing cluster configuration to a few tunable options is definitely a step in the right direction in simplifying cluster setup for the user, Atlas can do much more. Compare with other existing cloud database services, such as Amazon’s DynamoDB or Microsoft’s DocumentDB, where all you need to specify is the IOPS requirement of your application (application IOPS, not storage IOPS as it is in Atlas). In these competing cloud databases, issues such as replication, number of shards, and high availability are hidden away from the user. Granted, DocumentDB has been out for a year already and DynamoDB many years before that, so it may just be a matter of time before Atlas catches up. Regardless, there is a lot more Atlas can do to be truly serverless and even simpler to use—a solid first step, but lot more needs to be developed to compete with cloud databases and to win enterprise customers.

    Under the Covers: Backup and Recovery

    Now since I work on something similar here at Datos IO, I was quite interested in Atlas’ backup and recovery options. At a first glance, it looks similar to the backup and recovery services provided by MongoDB’s earlier products such as CloudManager and OpsManager. After playing around with the feature, I’m more convinced that the backup technology Atlas uses is essentially the same as previous offerings, only that it’s applied to Atlas instances.

    Backups are free for the first 1GB and $2.50 per GB per month for every GB after that. Backups can be scheduled at set intervals but with a minimum interval of six hours. While I was initially put off by the minimum snapshot interval (after all, a LOT can happen in six hours!), after some investigation, Atlas also allows you to schedule “checkpoints” in between snapshots. Checkpoints can be used as restore points, but they take longer to restore because it requires applying all deltas since the last snapshot. It’s important to note that this feature is only available for sharded clusters. If you want to protect data on an unsharded cluster, you’ll be stuck with a minimum snapshot interval of six hours.

    Backup can be set when by toggling a flag during cluster initialization. During restore time, you can select a specific snapshot to restore. If you need a finer granularity of restore, Atlas also provides point-in-time recovery, which is pretty neat. You can specify a time between two snapshots to restore at a minute-level granularity. If you need an even finer granularity, you can specify the specific oplog timestamp and interval you want to restore.

    8Existing cluster snapshots and the types of restores you can do.

    As I performed a couple of backups and restores, I had a couple reservations about the feature. Most of my qualms arose from the fact that Atlas treats the entire cluster as its ‘unit’ of backup. What I mean by this is that backups created by Atlas are snapshots of the entire cluster. So there is no granularity at all, and this is problematic in a few ways.

    First, since Atlas takes snapshots of the entire cluster, you can’t control what data you want to back up. Say, for example, only a small portion of your database contains critical sensitive information that needs to be backed up. With Atlas, to back up your critical data, you’ll have to back up all your noncritical data as well! Considering that the cost of backup increases with the size of your data, you’ll end up unnecessarily paying a lot for backups of data you don’t really care about.

    Moreover, your options when recovering your backups becomes constrained by the fact that backups are at the cluster-level. By nature of being cluster-level snapshots, backups must restore back the entire cluster; you can’t restore back a specific database or collection. This can be problematic when, say, only one of your collections in your production cluster becomes corrupted. In that case, Atlas takes a very heavy-handed approach of bringing down the entire cluster and resetting every collection back to the point of the snapshot, as opposed to only restoring the problematic collection.

    Lastly, Atlas only allows you to restore snapshots to clusters with the same number of shards as the snapshot. For example, if you have a cluster with three shards, you can only restore to other clusters with three shards. One potential reason you may want to restore data to a cluster with a different topology is for testing purposes: Perhaps you want to measure application performance as you increase the number of shards in your cluster, or perhaps you just need a small, cheap cluster for basic tests. If you want to restore production data to a test cluster in Atlas, you’ll either have to spin up a cluster with the same topology as the production cluster (which is good for MongoDB, but sad for your wallet), or you’ll have to manually download from Atlas and then manually restore that data back to Atlas into a cluster of different topology. Both cases work, but are far from being ideal.

    Concluding Thoughts

    Atlas is a pretty good move by MongoDB to make it even easier to build up applications on top of its database by abstracting out a lot of the more mundane details of cluster configuration to the people who know MongoDB best. For me, Atlas was a simple and fast way to create and scale up MongoDB clusters. While a good step in the right direction, Atlas can do more to make it a truly serverless approach like other cloud databases in the market right now.