devopsy.kinsta.cloud

Tag: auto-scaling

  • DevOps Proves No Coaster Ride for Roller

    DevOps Proves No Coaster Ride for Roller

    Keeping amusement parks running and the fun flowing is Roller‘s main job. The company provides software for the leisure and entertainment industries, notably theme parks, amusement parks and trampoline parks. Its core functions are ticketing, POS and CRM. Clients include Coney Island Luna Park, Aussie World and Twinlakes Park.

    When Roller decided to expand and redevelop its operations in 2015, DevOps was the main attraction. CTO Andrew Brodie led the company’s push, having worked on Roller’s original implementation in his previous role as co-founder and technical director of digital product design studio Bravo!

    Automation of Roller’s processes was a major goal. Even small organizations should adopt good practices, Brodie says, and that includes automating everything that can be automated (within reason) to the point of being able to click one button to deploy a system.

    Before DevOps

    In its early days, Roller used Microsoft tools including Web Deploy to deliver a monolithic application that met the requirements of a minimum viable product. Roller initially ran on Ninefold, a now-defunct Australian IaaS platform.

    When the company attracted additional funding in 2015, it was able to expand its development team with people who had various skills including DevOps. Today, it comprises around 50 people—12 developers and various user experience, quality assurance and project management specialists.

    The newfound expertise enabled the redevelopment of the system as a series of applications and microservices and eliminating most frequently performed manual operations.

    Tools

    Roller’s deployment process had become a bottleneck, so the company invested time in adopting the most appropriate tools. Several came from HashiCorp, including Packer (for creating machine images), Terraform (cross-provider infrastructure management), Consul (service discovery, configuration and orchestration) and Vault (centralized storage and control of distributed secrets such as keys).

    After Ninefold’s demise, Roller became heavily involved with AWS, which “had everything we needed” in the way of APIs and toolsets “to code out our infrastructure,” Brodie says. But even though “we’re really happy with Amazon,” the company has avoided using exclusively AWS tools to retain the ability to move to other platforms such as Azure.

    Implementing load balancing, auto scaling groups and so on as code took about a month’s effort. “It was a great move for us,” he says.

    Other tools used by Roller include the Chocolatey package manager, and Microsoft’s PowerShell and its Desired State Configuration feature. After a machine image is tested, that’s exactly what moves into production, providing confidence that it will work as expected.

    Roller uses blue-green deployment with autoscaling groups to make it easy to roll back if necessary. Deployment or rollback takes less than 10 minutes. When individual problems arise, Roller’s practice is to kill the unhealthy server and bring up a new one.

    The company maintains separate deployment pipelines for each application and service, using Atlassian’s Bitbucket distributed version control system and its Pipelines feature to automate the deployment of checked-in code.

    Roller can deploy code within an hour if necessary, but currently works to a weekly release schedule. The goal is to move to multiple releases in a day.

    Resources

    Major Australian jobsite Seek served as an inspiration for Roller’s DevOps path. “We’ve always looked at the bigger players,” Brodie notes. In addition, Meetups (“there’s something for everyone”) and conferences (he specifically mentioned DDD Melbourne and NDC Sydney) have been invaluable in being able to talk with other companies about their experiences. “We’re keen to stay ahead of the latest practices,” he says, so the company encourages its people to attend such events.

    Other resources he finds useful include Stack Overflow and Plural. Plus, he recommends “The DevOps Handbook” by Gene Kim, Jez Humble, Patrick Debois and John Willis, which he describes as “a fantastic resource,” providing practical advice particularly around issues at the organizational level.

    Tips

    When asked for advice for those starting a new business or major project, Brodie recommends pragmatism:

    • Consider DevOps from the start “but don’t overcook it.” In other words, begin with the basics and evolve from there.
    • Aim for continuous improvement—people will keep generating new ideas, but picking the right time to implement them is important.
    • Look for a mentor, and talk to people about the tools they’re using.
    • Take advantage of outside organizations’ capabilities. For example, Rackspace can help architect an application for AWS.

    — Stephen Withers

  • Tips to Optimize the Cloud for Scaling

    Tips to Optimize the Cloud for Scaling

    When I first started migrating companies to the cloud (about seven years ago), companies would ask me, “How come the app isn’t scaling? How come users are still complaining about the slowness?” I would explain that the cloud enables auto-scaling, but you still have to make sure your app can scale in the cloud. In people’s minds, auto-scaling and the cloud was synonyms.

    I don’t get asked that question anymore (maybe people realized the cloud is not as magical as it seems). But now, people think the cloud is synonymous with speed, agility and cost-effectiveness. Sure, cloud enables all that. But you still need to make sure you optimize the cloud to facilitate that. I see many companies in the opposite situation: their accounts full of resources with names such as test1, test2 and finaltest2, or they put together a SWAT team to reduce an AWS bill, or security is an afterthought. So many engineers are making changes, it’s hard to know who is doing what.

    Before you run into this situation (maybe you’re already there), you have to create tools and processes to optimize your cloud for scale.

    Establish Strong Policies

    The magnitude of changes in cloud infrastructure is high. Unless you establish strong policies, managing all the resources across multiple cloud accounts get out of control pretty quickly. The most effective way we have seen is using tags for resources. For example, use tags that identify whether the resource is from a certain department are added to each resource. Typically, it works very well when the tagging process is embedded into the automation. If there are new resources that don’t comply, automation should terminate these resources. You should enforce policies like these sooner than later.

    And just because these resources were provisioned with proper tagging, it doesn’t mean they are provisioned for its full capacity. You should build automation to find underutilized instances and establish a process for downsizing. To ensure everyone complies with these policies, you should also establish change management process.

    Change Management and CMDB

    The objective of the change management process is to ensure all the infrastructure changes cause minimum disruption and meet internal guidelines. Many say that ITIL change management process is not compatible for fast environments such as cloud. I agree that the traditional way of doing change management by tracking changes manually just doesn’t work, due to the magnitude of changes. But in the cloud, you can leverage automation to streamline the change management process; for example, you can use automation to filter normal changes versus manual changes. Once all the AWS changes are reviewed and approved, in addition to bringing visibility to all the changes for security and reliability change management can simultaneously help reduce waste.

    Most companies heavily leverage auto-scaling, where new resources can be launched based on specified metrics. However, this behavior makes approving changes difficult. If new resources are appearing and disappearing, how do we track what truly changed in the infrastructure? To gain a true picture, you need an infrastructure delta, which shows resource updates for the still-running resources for a given time period. You can also use automation to build a dynamic change management database, and use auto-discovery to keep CMDB in sync.

    Shared Responsibility

    DevOps calls for breaking down the walls between operation and development team to deliver code faster. If you release software fast, you can rapidly add value to your customer. The cloud enables this culture because developers no longer have to wait for IT to provision resources. What doesn’t work well, though, is when no one takes responsibility for optimizing cost and maintaining security. If engineers can launch resources, they are also responsible for cost and security—it’s a shared responsibility. You should provide visibility to the cost for each department, based on tagging we talked about earlier, so they know the cloud is not free 🙂 Anyone can launch resources as long as they understand the security best practices for each service they are using.

    Conclusion

    Lean thinking calls for reducing waste. What’s waste? Anything that doesn’t add value to the customer. You should work fearlessly to reduce waste. You move to the cloud because you can innovate fast and it’s cost-effective, so you can quickly add value to your customer. You should continue to optimize your cloud infrastructure to ensure it’s meeting its original intent.

    About the Author / JT Giri

    JT is the creator of nOps – AWS cloud security tool. He spent last 10 years helping companies migrate to cloud and build automated infrastructures. nOps is a next generation security and collaboration solution for the cloud. nOps applies change management practices to deliver integrated security and reduce cloud waste.

  • Deep Art of Scaling for Cloud Environments, Part 2

    Deep Art of Scaling for Cloud Environments, Part 2

    In Part One of our two-part series on scaling for cloud environments, we looked at some of important factors for scaling and best practices for code storage. Let’s continue the discussion.

    Auto Scaling for Next-Generation Applications

    There are a few very good-looking cloud hosting service providers that provide the features mentioned in Part One: AWS, Google Cloud Platform (GCP) and Microsoft Azure. If you are going to auto scale, it’s best to use one of these cloud hosting services. There are other cloud hosting service providers in my knowledge that follow similar approaches, but they are not widely used or are platform-as-a-service (PaaS) ecosystems so we will not cover them.

    Also, it’s important to remember that in auto scaling, all app nodes are changing and created dynamically, with a new host, new IP, etc., so your synchronization mechanism needs to determine how to update the code with the new nodes that may change or be deleted later.

    Options to Auto Scale and Not Die Trying

    Lsyncd

    Lsyncd allows you synchronize nodes across the cluster that are spinning up/down new instances, Work using this script, which monitors the auto scaling group for new nodes and updates the Lsyncd master with new auto scaled slaves. Discovering which IPs or instances are within the auto scaling group is the preamble auto scaling faces, but this script will help resolve the discovery.

    GlusterFS

    Another approach is to use GlusterFS, which ensures high availability and syncs between your web nodes. You need a minimum of two instances in case one fails. The other instance can jump in and still be tolerant for failures. Once the failed instance is back online, it’s added again to the cluster. GlusterFS works similarly to master/master or master/slave. I won’t cover how to configure GlusterFS, but I’m sure this approach was a very good option before AWS EFS was released. If you are not using AWS, this is an awesome option.

    On the other hand, if you are using AWS environments with EFS, consider registering your web node to the master node using “AWS User data,” so every time a new EC2 instance is created by the ASG, it will register this new volume to the master and keep data synced with the rest of the nodes.

    Pre-Baked AWS AMIs (Old Approach)

    Given the fact that your code won’t change or barely changes, baking an AWS AMI and updating the auto scaling launch configuration will do the trick—this is an older but still-used approach. When you have a defined code release schedule and a defined development workflow, you could bake an AMI for every new code release and, once ready, recycle the nodes to use the new AMI with the latest code update. There are plenty of tutorials on this method.

    Admin Server (mainly used for WordPress, Drupal, Magento and CMS-based apps)

    Another wild approach is to use an admin or master server out of your auto scaling group. This is used commonly to sync code from the master to the scaling group, but I believe this a waste of server resources that can be used inside the auto scaling group. I would recommend this approach for WordPress or Magento sites; in addition to this, consider proxying the wp-admin or administrator URL to the admin server, or update the CMS directly to the admin server. As with NFS, AWS EFS will facilitate everything with WordPress, Magento, joomla and CMS-based apps. Click here for an example of what I’m talking about.

    Implement AWS EFS

    Finally, the easiest approach is AWS Elastic File System – EFS. Amazon Web Services released EFS a few months ago as a very promising technology—a distributed file system similar to NFS but improved—think of it as a sophisticated NAS solution. During AWS Re:invent, the company said EFS is based on NFSv4.1 but with better network performance, and is a pNFS workaround. I have waited for such a component in the arsenal of AWS for months, though early access was only available for the Virginia region. Today, though, it is supported across all AWS regions – Yay!

    EFS can be mounted as a local mount point within fstab, similar to NFS, and you can share your code across your AWS auto scaling group: Set your EFS volume in/etc/fstab to load automatically in each reboot. Therefore, If you are in AWS, use EFS; it will facilitate everything—really. To accomplish this approach, make sure to update your user data in each EC2 instance with the EFS mount point, so every time a new instance is created it automatically will have the EFS volume.

    Note 1: All EFS mount points should be in the same region; if an EC2 instance tries to mount from another EFS region, it won’t be allowed.

    More Details can be found here.

    Google Cloud Platform

    Well, what do we have for Google Cloud Platform? I suggest GlusterFS, Lsyncd, rsync or Google Cloud Storage—although it shares a lot of features with Amazon S3, it is faster. Google states that its cloud storage is faster than Amazon S3 due to its infinite edge points across the world and, yes, it is correct. The easiest way to share the code across your auto scaling group is to use Google Cloud storage and GSutils. Additionally, you could use GSutils and call the bucket in each Google Instance—just make sure the visitor/user does not change the code or static content in the instances. Also, Gsutils helps sync the bucket data to your instance data to stay in sync. If your code changes dynamically with your traffic, you need to use gluster or Lsyncd, given that it is very complicated to deploy since Google auto scaling is always recycling instances/IPs and needs some workarounds.

    Don’t Forget Scaling Databases!

    Another topic that we didn’t touch was scaling a database, for AWS RDS, DynamoDB, Postgresql, Mysql and Nosql databases. All these modern databases are easy to scale; some of them are PaaS such as DynamoDB or RDS. For Postgresql, MySQL exists only for manual scaling, which is adding new database nodes to the master/slave cluster. In AWS RDS, with some clicks you can get a slave “replica” and you are scaling already. But I won’t go further about it as it’s out of scope for this piece.

    Scaling Microservices and Serverless Apps

    Definitely, these topics need another talk. There will be another time to explore this new architecture approach of splitting your application into pieces. Microservices is changing the IT paradigm, but to survive for this new paradigm, you need to understand first scaling for cloud before evolving to microservices with Docker, Consul, Compose, RancherOS, AWS ECS, Kubernetes and Serverless apps architectures.

    Conclusion

    diagram

    As you can see, scaling an application can be challenging when you are dealing with a custom app or CMS-based application. With this article,  you have the options and their pros and cons in tackling this constant preamble of how to scale with custom apps. Everybody uses their own scaling process and seriously, they never fit with other applications’ needs. Now that AWS EFS is out there, I’d recommend it every time you can, but in case you are not using AWS, implement Lsyncd or GlusterFS, but don’t forget the other methods that also can help accomplish your scaling goals.

    About the Author / Alfonso Valdes

    alfonsovaldesAlfonso Valdes is the CEO and Founder of ClickIT Smart Technologies, a IT agency focused in the architecture and implementation of Modern Cloud Applications highly resilient in Linux and Open Source technologies. He combines more than 8 years of IT experience in the field of scalable architectures, distributed systems and developing cloud-native applications, and has worked for different U.S companies such as: Sprint Nextel, Texas Instruments and Nike USA. Nowadays he is encouraging enterprises to join the cloud. Connect with him on LinkedIn.

  • Deep Art of Scaling for Cloud Environments, Part 1

    Deep Art of Scaling for Cloud Environments, Part 1

    The number of resources to scale an application are abundant, but every application has its own challenges to scale successfully, and the solution to this problem doesn’t come handy. This two-part article focuses on the best approaches to build a scalable application, software-as-a-service (SaaS) application, mobile app, or WordPress-, Drupal- or Magento-based website. The scope of the approach extends to both auto scaling and manual scaling.

    In my experience, there are two types of applications: ones that use an admin and Admin panel to modify the code, layout and functionality (WordPress, Joomla, Drupal, etc.) and modern, next-generation applications that are stateless and follow the 12-factor app approach.

    But first, we have to decide whether to use auto scaling or manual scaling. It is an important decision to make in the very beginning, as it changes the whole approach and complexity of developing your app. We will be discussing both, but in essence of auto scaling, you will need to consider how to maintain the code synchronized across the web nodes and how to maintain new web nodes that constantly are being created/deleted in the auto scaling group. (With stateless apps this is not an issue, as the images or static content can go to Amazon S3, CDN or isolated services. Later on will explain a bit about these technologies.) Working with manual scaling will facilitate your work—you just need your code synced across all the web nodes. All of the approaches mentioned below apply to manual scaling.

    But on the other hand, auto scaling with an admin application is a headache for some common reasons: Every time a front-end developer  changes something in the admin, code will be inconsistent across multiple web nodes along with the Git versioning.  Because of this issue, I don’t  recommend using Git for such applications—there are always discrepancies between the last commit/version in Git and the last update in the admin CMS, which makes auto scaling with admin even more complicated.

    Let’s cover some approaches to solve this problem. Using an NFS distributed file system will resolve everything for a CMS app, but as everybody knows, this approach pulls down the performance.

    What’s Important for Scaling:

    • Designing a deployment workflow that is easy for your team, your manager and developers—that’s why you are here, to improve development productivity by defining a good mechanism to deploy code to your cluster or auto scaling group. Here, you can think of Webhooks, rsync cronjob, bash scripts, Jenkins, CircleCI, DeployBot, manual pulls or AWS CodeDeploy.
    • Isolating and centralizing all services. Databases, caching session, integrate s3 for static content, implement CDN, worker servers, cronjobs, Load Balancer, app server, web server, etc., to name a few.
    • Dynamically synchronizing code in a live application with many web nodes behind a load balancer or AWS auto scaling Group.
    • Ability to use the 12 factor approach for next generation applications, or stateless apps. You may be using this approach already but you don’t know it!
    • Bootstrapping your application with a stateless approach—this will facilitate all your work when you implement a scalable infrastructure. There are many ways to bootstrap your base application using bash scripts, docker files, vagrant file, AWS code deploy, git, user data file from AWS and baked AWS AMI, among others.
    • Using immutable servers in your environment. Make sure your applications and servers are in a pre-configured image state. Every moment a new server/instance is created in an auto scaling mechanism, the instance should be identically cloned of any other instance in production without any further configuration. As you know, AWS, Docker, Google Cloud, and cloud-based providers are much more easier for immutability. This really helps to deliver software faster and decrease time to market.
    • For CMS based apps, stay out of git for now, unless you have a very good code change control with your CMS admin and a very good discipline to push your changes with every dashboard/back-end change in the CMS.
    • A load balancer for auto scaling, such as AWS ELB, Google Cloud LB, Nginx or HAProxy.

    Advised Practices for Code Storage

    Understanding how your application works in terms of how visitors dynamically change your application data is very important. It’s imperative to understand how the session is saved, either locally or in Redis/Memcached. If it’s saved locally, you need to add this data to your synchronization mechanism. Another storage change is the user image uploads for WordPress or Magento-like CMS-based websites; use of plug-ins such as Wp super cache or w3 total cache makes image upload configuration easy. Plus, the option to integrate with a CDN, Amazon s3 or a cloud storage pushes the boundaries even further.

    But if you have a modern application, you need to understand and guide your development team how to change local image uploads path to a cloud storage or CDN. Make sure all your dynamic data is saved in a SQL or NoSQL database, and all cached database data (db queries, objects, sessions, keys, etc.) is saved to redis/memcached, Taking care of these factors will facilitate your scaling journey.

    Why not use NFS?

    Well, we hear a lot around the internet that the easiest way to achieve scalability is integrating NFS. Though it is the simplest way to look for and implement a fix, reducing the work for a service provider/developer  by almost 50 percent, it’s a real sacrifice in terms of application performance. Autoscaling an environment with a slow response speaks against the whole cause of the setup! It is very irritating to a client whose application speed is 1 second to 2 seconds when switching to a scalable environment with NFS slows the application by at least 50 percent.

    It’s also very disappointing when your small EC2 instance is much faster than your auto scalable environment with NFS. Additionally, using NFS has a point of failure, unless you have a failover in the cloud that is rarely used. It is just not worthwhile to have a complete standby environment waiting for a failure, and it’s better to use this environment within your cluster and have more resources to consume your traffic, or even better use this redundant environment to balance between different regions or data centers.

    Why not use Amazon S3 to Share Code?

    What about using Amazon S3, Google Cloud Files or any cloud storage to share code?

    Implementing Amazon S3 to share code between nodes will slow down your site similar to NFS. There’s not much of a difference accessing the S3 URL to mounting the S3 to a Linux directory, so don’t use Amazon S3 as the document root or code base folder. Keep the use of S3 limited to static content that needs to be fetched over time. Yes, there are websites that can survive and work decently using Amazon S3 for static web pages, but from my experience, I don’t recommend it for websites that support an application responsible for driving revenue to the company.

    Stay tuned for Part Two of our conversation on scaling for cloud environments.

    About the Author / Alfonso Valdes

    alfonsovaldesAlfonso Valdes is the CEO and Founder of ClickIT Smart Technologies, a IT agency focused in the architecture and implementation of Modern Cloud Applications highly resilient in Linux and Open Source technologies. He combines more than 8 years of IT experience in the field of scalable architectures, distributed systems and developing cloud-native applications, and has worked for different U.S companies such as: Sprint Nextel, Texas Instruments and Nike USA. Nowadays he is encouraging enterprises to join the cloud. Connect with him on LinkedIn.

  • 3 Tips to Avoid Painting Your DevOps into a Scalability Corner

    3 Tips to Avoid Painting Your DevOps into a Scalability Corner

    When everything is firing on all cylinders, DevOps can be awesome. Teams work together seamlessly. Routine tasks are automated. Continuous delivery is continuously integrated, continuously tested, continuously deployed and continuously monitored. All is well, and right with the world—unless your DevOps needs begin exceeding the capacity of the platforms and tools you depend on. Scalability is a crucial element of DevOps success.

    Wikipedia defines “scalability” as the capability of a system, network or process to handle a growing amount of work, or its potential to be enlarged to accommodate that growth.

    The need for scalability is one of the primary reasons that DevOps and cloud computing are so integrally linked. While it’s possible to do DevOps on some level using physical hardware or virtual servers running on hardware in a local data center, it can quickly become impractical—if not impossible. Cloud computing using platforms such as AWS, Rackspace or Microsoft Azure enables an organization to increase server capacity exponentially at the push of a button.

    “Many of the issues and challenges experienced before the advent of cloud are no longer issues: Engineers now working on cloud implementations remember working at companies that feared the Slashdot effect—a massive influx of traffic that would cause servers to fail,” explains Jason McKay, VP of Engineering for Logicworks, in a blog post.

    Tackling Scalability

    There are a number of factors impacting scalability in your environment, which could impact the success of your DevOps. Here are three things to keep in mind to ensure your environment is ready to scale:

    1. Make sure your tools and apps are scalable

    The ability to scale capacity on demand—automatically—is one of the chief advantages of cloud computing, and one why cloud computing is so important to DevOps success. There is more to scalability, however, than just server and network capacity. You must also ensure that the tools you rely on can seamlessly scale with the infrastructure, and that the apps you’re delivering through your DevOps environment are designed to take full advantage of the increased capacity as well.

    2. Continuous optimization

    Some tools and apps will continue to run as the underlying infrastructure scales up or down, but may not run optimally. Make sure that the tools you use and the apps you deploy are able to continuously adjust to changes in server, bandwidth and storage capacity. The tools and apps should be able to take full advantage of the resources available to ensure the best performance possible at all times.

    3. Don’t forget about storage

    Most apps have a data component and that means there is data storage that must be scaled as well—either as a function of the broader network infrastructure scaling or just in response to increased demand for data storage.

    Dan Leary, VP of Products, Solutions and Alliances at Nimble Storage, believes that scalability is emerging as a differentiating factor between storage solutions. “Due to the high performance offered by many vendors of flash-optimized storage solutions, performance will no longer be the primary differentiating factor when considering one vendor over another. Instead, a much greater focus will be placed on the array’s reliability, data integrity, scalability, manageability and in-built data-protection capabilities. We are now getting to the point where performance in storage has been democratized and will change the way people purchase storage.”

    There is one more thing to remember when considering scalability: cost. While it may be easy to automate scalability on cloud platforms, it’s not free, and the DevOps tools you depend on may include licensing fees that could be impacted as your usage scales, so keep that in mind.