devopsy.kinsta.cloud

Tag: AWS

  • How Organizations Benefit from AWS App Patterns

    How Organizations Benefit from AWS App Patterns

    As a startup, an organization thrives by breaking rules, challenging the status quo and standing for all that is against the norm. The same rules don’t apply once the same organization grows, matures, becomes popular, gains a long list of clients and expands to a global footprint. With scale and growth comes the need for reliability, consistency, being able to deliver the same level of service to all customers and keeping promises made to customers. At this stage, application patterns become indispensable to an organization’s day-to-day functioning.

    Support of the Application Now and Long-Term

    Mid- to large-size organizations typically deal with numerous internal-only applications and external vendor applications. These applications are changed and updated regularly. To support the applications’ needs, and effectively manage them, organizations need a change management system in place.

    A service such as AWS Config helps to audit AWS services that power applications so there’s increased visibility. It tracks changes made by users and applications, and presents a clear picture of how these changes impact AWS resources.

    Enabling Shift-Left Without Additional Overhead

    DevOps bring together Dev, QA and Ops into smaller, cross-functional teams. The goal is to break down silos between teams, facilitate collaboration and drive better results faster. Accordingly, shift-left is a tactic used to encourage QA and Ops teams to play a more pivotal role in the initial Dev stages of the pipeline.

    The benefits are many. QA brings their experience of testing the application to its limits and informs about what’s feasible to implement. Ops chimes in with concerns and things to watch for once deployed. While shift-left is great from a team standpoint, it can meet with hurdles and can fail if communication patterns are random, and ownership of each step is not clearly defined.

    Application patterns encourage systematizing this collaboration between teams. Patterns help to define the type of infrastructure to be used, their configuration and the kind of tools that fit in with the team’s requirements. Application patterns guide decisions and help to make the right choices, and sometimes compromises, at every step.

    Supporting the Advancements in Open-Source Without Fear of Tool Attrition

    There is an explosion of open source tools today. If DevOps teams are attracted to the next shiny new open source tool, they’ll soon find themselves overwhelmed with tools they don’t use and that slow them down.

    Open source tools evolve quickly, and their popularity rises and falls with the times. One look at the open source monitoring space will show you that older tools such as Nagios and Zabbix are now becoming less relevant, and modern container-aware monitoring tools such as Prometheus are taking their place.

    AWS supports the use of the top open source tools; for example, Elasticsearch. The AWS Elasticsearch service integrates seamlessly with Kibana for visualization and delivers the ELK stack without the maintenance.

    Application patterns enable you to define a strategy for tool adoption. You may find the best open source tool in the world, but what’s the point if you can’t use it in your toolchain, or if it slows down the pace of operations? The popular DevOps mantra of people, processes and then tools holds true here.

    Value Stream Management

    Value stream mapping is about visualizing the flow of value from an organization to its customers. In simple words, it’s about taking stock of what you have to offer to customers and monitoring how your offerings change over time.

    Value flows outward to customers, as well as internally between employees and systems. For example, in the context of DevOps, it helps to map the value of Ops to Dev and QA. As these teams are each other’s internal customers, knowing what the value stream is can help remove bottlenecks and drive greater efficiency.

    Building a Team and Culture First

    AWS is a great place to build team culture as it is expansive and includes services for pretty much anything you’ll ever need. You can find tools such as CodePipeline to quicken development pipelines, to make sure everyone on the team is informed of important changes using CloudWatch and SNS queues, and to automate common tasks with Lambda.

    Rather than jumping feet first into AWS and then figuring out how it all works, it pays to first define application patterns and team culture that will govern the way you use AWS. Patterns and culture are closely related. If culture is about the people, application patterns are about the infrastructure and technology stack being used by people.

    Conclusion

    Whether you’re yet to move to AWS or are already on board but not getting the most out of it, application patterns help evolve the way you manage applications in AWS. Application patterns help scale applications over the long term, they enable shift-left for DevOps teams without the friction of overhead. Application patterns make it possible to adopt the right open source tools and find success with them. By using application patterns you can visualize the entire process and drive great efficiency, and finally, they support great team culture within DevOps teams.

    DevOps teams that run their operations predominantly on AWS need not fly blind. Application patterns bring greater visibility, control, collaboration and ultimately value.

    This sponsored article was written on behalf of Eplexity.

    — Twain Taylor

  • The Accelerating Cloud Wars

    The Accelerating Cloud Wars

    Enterprises have embraced cloud computing at a phenomenal clip, according to newly-released report, A for Adoption, from cloud access security broker provider Bitglass.

    Over the past five years, the report found cloud penetration grew from 24% (in 2014) to 86% of organizations today. Last year, that adoption rate was 81%. Yet, while cloud has helped enterprises to reduce IT capital expenditures and become more agile in meeting business demands, these organizations need to be cognizant of the impact the shift to the cloud has on securing user access—something the report hints as an area where organizations lack.

    For instance, the Bitglass report found, of the 86% of organizations that adopted cloud, only 34% of those have implemented single sign-on (SSO). In addition to tracking SSO usage, Bitglass uncovered adoption rates for a host of different cloud-based tools, including Microsoft Office 365, Google’s G Suite, Amazon Web Services (AWS), Slack, Salesforce, Box and ServiceNow.

    The annual report highlighted the rapid increase in the adoption of a number of services, such as Salesforce and Slack, which respectively increased by 55% and 44%.

    The survey also found that Google’s G Suite, with a 33% adoption rate, is still struggling to regain headway on Microsoft Office 365 and that productivity suite’s 79% adoption rate. At 82%, financial services organizations are the fourth largest adopters of Office 365, while 79% of healthcare organizations have done so.

    Those enterprises that have deployed AWS increased by 43% in one year, reaching 20% adoption across all industries this year. Not surprisingly, the technology industry is the leading implementer of AWS at a 32% rate and is also the leading industry using Slack at 79%.

    One of the greatest challenges when it comes to effectively managing and securing access to the cloud is access management. Interestingly, only 34% of enterprises have implemented SSO for cloud access. However, the amount of SSO adoption varies significantly from industry to industry.

    For instance, the adoption of SSO in financial services grew nearly 41% year over year, while in healthcare that growth reached 30%. However, the technology industry hasn’t embraced SSO much more, having reached a 33% adoption rate. Interestingly, education led SSO adoption at 61%, with the government firmly in second with 46%.

    The adoption of SSO also varied by organizational size. As it turned out, larger organizations are the leading adopters of SSO. In fact, 64% of large firms said they utilize SSO, which is nearly 50% more than what medium-sized organizations reported. That’s more than twice the adoption of small organizations.

    The Bitglass report is based on the company’s scanned analysis of cloud to identify which cloud services were in use across a sample of more than 138,000 companies. According to the company, the analysis included productivity suites such as Office 365, G Suite and messaging apps such as Slack, as well as file-sharing apps such as Box.

    — George V. Hulme

  • What Are AWS Application Patterns?

    What Are AWS Application Patterns?

    The cloud-native approach is not just about compute power and cloud services. A “set it and forget it” attitude won’t work with cloud native; there are far too many moving pieces and there is just way too much complexity. Instead, this approach requires constant testing, optimization and improvement. This is epitomized in the two key phrases of the DevOps methodology: continuous integration and continuous delivery.

    To make this a reality, however, it takes a strategic approach to defining application patterns that will govern the way in which cloud-native applications are built and shipped, rather than a one-off effort. Since AWS is the leading cloud provider, we should discuss which application patterns are relevant to the AWS platform. There are many aspects to consider when it comes to AWS application patterns, and we will cover them here.

    Operational Excellence

    The best starting point for application patterns is a commitment to operational excellence. Typically, the term “Ops” is used to identify the IT team, but in this case, operational excellence rests on the shoulders of Dev, QA and Ops teams. In cross-functional DevOps teams, every member should strive for operational excellence at every step–even in the smallest of tasks.

    In their white paper on operational excellence, AWS encourages its users to manage operations as code. This helps avoid human error, enables automation of complex processes and allows for quick and easy rollback when errors occur. Failure, according to AWS, is an opportunity for learning. They even advise performing “pre-mortems” that try to anticipate potential points of failure before a deployment.

    AWS Trusted Advisor is a useful tool that enables this type of accountability and shared responsibility within a team. It checks and reviews your new and existing AWS workloads looking for opportunities to save costs, improve performance, tighten security or follow best practices for scaling operations.

    Security

    Security is a pattern that is non-negotiable. The term “DevOps” has inspired another variation called DevSecOps that signifies this fact. Security operations take center stage when talking about DevOps.

    The interesting thing about DevSecOps is it actually results in more freedom, flexibility and trust across teams when done right. This is unlike traditional approaches to security which created silos, stifled creativity and slowed down innovation. Security is now a key piece of the puzzle, not an afterthought. AWS has many machine learning-based security solutions such as Macie which automatically identifies and secures personally identifiable information (PII).

    Along with this, configuring role-based access controls using AWS IAM is essential to have greater control over who or which applications access data and for how long. Access Control Lists govern access to data stored in AWS S3 buckets.

    The news is full of instances of data breaches due to weak security policies and access controls. Capital One is a recent example of a data breach that affected 100 million of its users. While security in the cloud is more complex than before, AWS has the necessary tools and systems in place. It’s up to organizations to implement a security strategy that prevents misuse of their data.

    Reliability

    Today, there is a low tolerance for downtime because of the potential loss of revenue. With the stakes so high, reliability is a pattern any application custodian would have high on their list of priorities. AWS has multiple reporting and health monitoring tools to keep you constantly on the pulse of your applications.

    Architectural patterns like microservices enable better resilience by decomposing monolithic applications into more manageable chunks called services. This “divide and conquer” approach leads to higher availability, which is critical for SaaS applications.

    Patterns like chaos engineering seek to intentionally kill random instances and services and force the DevOps team to figure out workarounds and backups for every scenario.

    Application Performance

    With the explosion of data, applications are always dealing with lots of data to transfer, store and process. An application’s performance depends on three factors: the backend data storage layer, the middleware components and the frontend.

    The backend data layer is the most critical component when it comes to performance. Choosing the right storage formats, database types, storage devices and organization patterns all contribute to the speed of data access. AWS has an amazing array of storage services, including S3, RedShift, Aurora and Glacier. AWS gives its users every type of storage solution, all of which should be considered to improve data management.

    AWS is devising new ways to query large datasets without bearing the load on your existing systems. RedShift Spectrum is one example of this. It is a great way to “minimize the need to scale Redshift with a new node, which can be expensive.” If you’d still rather not have to manage any RedShift clusters on your own AWS Athena is a great serverless alternative

    That gives you all the power you need to query large datasets in S3 without having to maintain any clusters of your own.

    This pay-per-use model is making its ways across the entire AWS stack with services such as ECS and Fargate offering completely serverless experiences when operating containers at scale. What started out with AWS Lambda is catching on fast, and many AWS services like the ones mentioned above enable serverless experiences with a focus on application performance.

    Cost Optimization

    When moving to the cloud, pricing is completely different. Rather than paying for perpetual licenses, you now pay a small fee for every small resource that you use. Though small on its own, you’ll soon realize that cloud costs can shoot up easily if you’re not paying close attention to your spending.

    Cost optimization is necessary as you scale operations in the cloud. Cost optimization includes looking for more efficient services, whether that’s within the same vendor platform or a different platform. At a minimum it may require a lift-and-shift approach to transition from expensive old infrastructure to inexpensive new alternatives. For example, shifting workloads from VMs to containers to serverless within the same platform can show great cost savings, since containers and serverless are generally more cost-effective than VMs.

    It pays to leverage the expertise of experts in this area. They should ideally be people who have spent years using AWS, understand Well-Architected best practices and have found ways to cut costs and derive more value from the same budget.

    Application Patterns Enable Extensibility

    Microservice applications adopt a pluggable architecture. This enables them to be easily integrated with external applications. Typically, API-based communication patterns are most common. AWS API Gateway is a service that manages API design for AWS applications. In some cases, an SDK is the better choice–for example, when you need more control over the developer experience and integrations. It’s important to know when to choose either option.

    Application Patterns Enable Portability

    Portability is the ability to move workloads across instances without interrupting their functioning. As instances change, the workloads which they support shouldn’t be affected. This is only possible if there is a clear separation layer between the workloads and the infrastructure that runs them.

    Application Patterns Enable Shift-Left

    The DevOps methodology is known to encourage a shift-left approach to software delivery. This means that not only Dev, but also QA and Ops contribute to the design and development stage of an application. This fosters deeper collaboration across teams and breaks down silos.

    A shift-right approach holds true as well. Shift-right means developers are responsible for the code they write. They are equal owners with operations teams for the resilience of the apps they build and also the bugs they introduce into the system. This practice of putting ownership and responsibility where it belongs is very healthy for a DevOps team.

    AWS facilitates pipeline management using tools such as CodePipeline, CodeCommit and CodeDeploy. They each perform similar but complementary roles and help build a CI/CD pipeline. They make it easy to get started with CI/CD within the AWS platform.

    For organizations further down the line in their cloud journey, AWS’ pipeline management tools may seem too simplistic for something like GitOps. Instead, they’d want to look outside AWS for tools such as Jenkins X or Weave Flux which are designed with GitOps automation in mind. Whether it’s the simple or advanced route, a pipeline brings great benefits over a manual development process.

    Conclusion

    Though tools enable the implementation of helpful application patterns, on their own, they are ineffective. Application patterns need a deliberate strategy and an approach for a start. Only after a strategy is in place, can you move to deciding which tools to use. With a strong strategy in place, any of the tools mentioned here can be swapped out for other better tools and the results will be the same. It’s important to not be distracted by the next shiny tool but keep a keen focus on the strategy for implementing AWS application patterns.

    As we’ve seen here, application patterns are diverse, but they all build upon each other. Without even one of these pillars, organizations would not be able to realize the goal of cloud-native applications. On the other hand, if they give careful thought to each of these application patterns, organizations would deliver cutting-edge applications that can only get better with the passing of time.

    This sponsored article was written on behalf of Eplexity.

    — Twain Taylor

  • How to Transition to Become a DevOps Engineer

    How to Transition to Become a DevOps Engineer

    You’ve either heard the buzzword or you’ve seen the extensive list of rewarding aspects that come with being a DevOps engineer. Whatever the reason you’re here, this guide should help with taking the right steps toward becoming a DevOps engineer, whether this is your first job after graduating or an opportunity to reskill leveraging your prior industry experience.

    Immerse Yourself

    Learn the fundamentals, practices and methodologies. Understand the why behind DevOps before jumping into the tools. I believe the key goal of any DevOps engineer is to increase speed and maintain or improve quality across the entire software development life cycle, in order to provide maximum business value. Read articles, watch some YouTube videos and go to one of your local Meetup groups or conferences–it’s useful to become a part of the welcoming DevOps community where you’ll learn from the mistakes and successes of those that have been here before you.

    Your Background

    There are likely two key times where you’ll be transitioning into the role of DevOps engineer; either reskilling during your career or as your first job after graduating. You may have prior experience in technology, such as working as a software developer, systems engineer, systems administrator, network operations engineer or database administrator, all of which will bring broad insights and useful experience for your future role as a DevOps engineer. If you’re starting your career, maybe you’ve finished your degree in computer science or any other STEM field which will give you some of the basic stepping stones you’ll need in this transition.

    I think it’s important to breakdown the broad spectrum of what DevOps engineer roles entail. Most enterprise companies today will use DevOps engineers in three different positions.

    • DevOps Engineers, with a Dev Bias: Work in a software development role, building software applications. The key point is they leverage CI/CD, shared repositories, cloud and containers as part of their everyday work, but they are not necessarily responsible for building or implementing tooling. They will understand infrastructure and in a mature environment, will be able to push their own code into production.
    • DevOps Engineers, with an Ops Bias: Could be compared to systems engineers or systems administrators. They understand software development but do not spend the core parts of their day building software applications. Instead, they are more likely to be supporting software development teams to automate manual processes and increase efficiencies across the human and technology systems. This could mean breaking down legacy code and using less cumbersome automation scripts to run the same commands or it could mean installing, configuring or maintaining infrastructure and tooling. They ensure the right tools are installed and available for any teams that need them. They enable teams with learning how to leverage CI/CD and other DevOps practices.
    • SREs (Site Reliability Engineers): Think of site reliability engineering as having software engineers solving operations and infrastructure problems. SREs focus on creating scalable, highly available and reliable software systems.

    In the ideal world, DevOps engineers will understand all the above, which is the norm at some mature technology companies. However, if you knock on the door of top tier banks or many Fortune 500 companies you’ll realize oftentimes these roles do usually have biases towards dev or ops.

    Operating Systems

    Whichever background you’re coming from, you should start with the fundamental technologies you’ll need to use and understand as a DevOps engineer. Linux is most likely the operating system you’ll be using daily, although some of you might be using Windows. This is where everything runs, and fundamental knowledge is important. You can install Linux at home where you’ll be able to break as much as you want and continue learning along the way.

    Scripting

    Next up you’ll need to pick a scripting language. There are many to choose from, such as Python, Go, Java, Bash, PowerShell, Ruby and C/C++. I suggest starting with Python, it’s one of the most popular for a reason, it’s easy to learn and interpret. Python follows the fundamentals of object-oriented programming and can be used for web development, software development, creating desktop GUI apps and even business applications.

    Cloud

    After Linux and Python, my next pick would be cloud, where you could choose from Amazon Web Services, Azure or Google Cloud Platform. I’d start with AWS, as they have an extensive collection of free learning tools available to  take you down any track from learning about AWS as a developer to operations and even the business facing components. In fact, you might even be overwhelmed by how much is on offer when you first get setup with AWS. Maybe start with EC2, S3 and VPC and see where you want to go from there.

    Programming languages

    For those of you with a passion for software development. You should keep on improving your programming skills. Some good languages commonly used in this role include Python, Go, Java, Bash, PowerShell, Ruby and C/C++. You should also become familiar with tools such as Jenkins and Git/GitHub which you’ll be using frequently as part of the CI/CD process. Infrastructure is no longer left to the operations guys so you’ll need to have some exposure to a cloud platform such as AWS, Azure or GCP.

    Containers

    Finally, start learning about containerizing code using tools such as Docker and orchestration platforms such as Kubernetes. All the above have extensive online learning resources for free and most cities will have local Meetup groups where you can turn up and learn from experienced people in a friendly environment.

    For those of you with less experience in development, you can still get involved using your passion for automation, increasing efficiency, collaborating with others and a never-ending desire to improve on your own work. I would still suggest learning the tooling outlined above, but with less emphasis on the coding/scripting languages. It will be useful to learn about Infrastructure as a Service, Platform as a Service, cloud platforms and Linux. You will likely be setting up the tools and learning how to build systems that are resilient and fault-tolerant while writing your own code.

    Reskilling

    If you’re transitioning into a DevOps role during your career, you should start within your company to explore opportunities. Can you reskill by working with an existing team? There might be an option for you to shadow other team members, ask for advice and reskill without even leaving your current job. This might not always be possible and some of you will need to move into another company.

    If you can learn some of the practices, tools and technologies listed above, you’ll be in a good position to demonstrate relevant knowledge during interviews. The key is to be honest and to not set yourself up for failure. Most hiring managers understand you don’t know all the answers but if you can show what you’ve been learning and explain you’re open to learning more, you should be in with a good chance.

    Starting Your Career

    You can apply to direct open opportunities at companies who are hiring junior DevOps engineers. You will often find these companies say they’re looking for more experience and you should apply back in future when you’ve gained some commercial experience. It’s the typical, frustrating scenario of “we want more experience” but nobody is willing to give you the first chance.

    It’s not all gloomy though, there are a few companies focused on training and upskilling graduates directly out of the university. MThree, where I currently work, delivers exactly that. We hire fresh graduates and train them for eight weeks in our Alumni Academy. Training covers everything from theory through to hands-on projects with Python, Jenkins, Git, Ansible, Docker, Kubernetes and AWS. By the end of the training you’ll have some solid exposure to the entire software development life cycle and a good understanding of how this applies in a Fortune 500 environment. The best part is once qualified, our teams then leverage their extensive list of clients to provide job opportunities for junior DevOps engineers. MThree pays these associates on a full-time salary with benefits for the first 18-24 months before they join the client as direct employees at the end of the agreement. This is a great way to bridge the gap from university into your technology career.

    In summary, there are many ways to transition to become a DevOps engineer. This is a very rewarding career route which will likely keep you engaged and challenged as well as increasing your earning potential.

    — Conor Delanbanque

  • Top 5 AWS Security Mistakes: Leaky S3 Buckets

    Top 5 AWS Security Mistakes: Leaky S3 Buckets

    As we get ready to discuss our list of the Top 5 AWS Security Mistakes in the upcoming devopsy.kinsta.cloud webinar, we wanted to provide a preview of the type and depth of information we’ll be discussing. Since the most-talked about, and likely the most vulnerable, aspect of AWS security is inevitably those leaky S3 buckets, it’s a given we’ll tackle S3 first.

    In this article, we’ll dig into S3 buckets, explain how they become vulnerable, cover eight different ways they can inherently become open to the public and how to find and protect any public facing S3 data.

    The S3 Bucket

    S3 is one of the oldest services in AWS—so old that parts of it still support XML-based policies instead of the JSON you see everywhere else. S3 also has a lot of features that aren’t as commonly known or used anymore, such as allowing someone else to host put objects in your bucket yet still maintain ownership of them.

    Think of an S3 bucket as a server that can have subdirectories and files. Because S3 buckets can have a different root url (e.g. “dops.s3.amazonaws.com”) and have their own life cycle and settings, we tend to think of buckets as servers, not directories. Also, directories in S3 allow you to organize objects, but don’t have any distinct settings of their own, other than a name.

    AWS has a wide range of use cases it needs to support, which is why we have all these mechanisms. Sharing publicly, within an account, between accounts, hosting websites and so on, creates complexity. The good news is S3 always defaults to secure and private. The bad news is AWS allows human beings to use it (and therefore weaken security) and can be confusing to manage. For example, S3 supports both read and write permissions (and list, for buckets). It’s possible to have no public read but public write permissions, which could lead to people placing bad files in your directories.

    8 Ways AWS Data Becomes Public

    There are at least eight different ways an AWS S3 bucket can inadvertently become open to the public and be exposed to a data breach.

    • Bucket ACLs (Access Control Lists): This is an XML document that defines the first layer of access. By default, only the account owner has access, but this can be opened up to other AWS accounts or the public at large. Amazon recommends against using these at all, but it’s also the easiest way to make something public quickly, so we see it all the time.
    • Bucket Policies: These are super-flexible JSON policies that allow you to set things such as IP-based and other conditional permissions on a bucket. While this should be the primary way for managing public access, ACLs are the first tab and about three clicks to make something public. It’s this flexibility that leads to mistakes such as opening up to a wider range of IP addresses than intended or using a negative to block access to some IPs but inadvertently granting access to everyone else.
    • IAM Policies: These are the normal IAM permissions you use to control access throughout your AWS account. You can’t make a bucket public with them, since they only govern AWS users, but you can open up access to the bucket by authorizing another AWS service access, and then that service exposes the content.
    • Object ACLs: These are the primary object-level controls. It’s just like a bucket ACL and uses XML to allow access from other accounts or the public. Objects do not necessarily inherit bucket ACLs. You can totally make an object public even if the bucket is private.
    • Explicit IAM or Bucket Policy Statements: IAM policies and bucket policies can have explicit statements referencing the object, which will override the object ACL (if you own the object), since they are evaluated first.
    • Pre-Signed URLs: These object-level policies must be created using code (not the console) and provide temporary access for anyone with the URL. It’s used, for example, to share files for a few minutes or an hour for someone using your app to download a media file.
    • CloudFront Origin Access Identity: CloudFront is the AWS content delivery network and can serve as the front end to S3. You can create something called a CloudFront origin access identity to write an IAM policy that allows CloudFront to access the S3 content. If the CloudFront allows public access, it can access the S3 content and that won’t show in a bucket policy or ACL.
    • Cross Origin Resource Sharing (CORS) Policies: CORS is required if you use S3 in a website and don’t want the browser to break due to same site security settings. CORS in S3 won’t override an ACL or bucket policy but could mask public access in limited situations where the data is exposed in the web code through the authorized site.

    All put together, AWS evaluates IAM permissions first. Within those, the only one to make a bucket public over the web is the CloudFront Origin Access Identity. AWS then turns to the bucket ACLs and policies for any explicit deny statements. Then it looks at the object ACLs for public access. If you make an object public and the bucket policy doesn’t have an explicit deny, it is still public, but otherwise a good bucket policy will block everything. Besides, a lot of the problems are open bucket ACLs, not object ACLs.

    4 Ways to Find Your Public S3 Buckets and Protect Your Data

    What’s the best way to find public buckets? It’s relatively easy.

    1. Log into the console, click on S3 and look for the Public tag. AWS uses some advanced back-end math to evaluate all the bucket policies to figure out if something is public, which catches most of the fringe cases, but it does not show if the bucket is private and objects in it are public.
    2. Check CloudFront to see if you use any origin access identities. It’s easier to look here than in all your bucket policies (usually). This saves you from looking through object ACLs as well.
    3. Check any CORS policies, if you use them. This is the least common situation and I even hesitated to complicate this post by mentioning them.
    4. The bad news … there is no easy way to find public objects. Even finding them programmatically can be difficult if you have a large number. Also, changing a bucket ACL doesn’t cascade to the object ACLs so you need to run through and fix them one by one. However, if the bucket is locked down the attacker would need to know the full URL/name of the object to find it. It’s not perfect, but it’s better than nothing.

    That’s it. Although, if you know of any other fringe cases, please don’t hesitate to let us know. While conducting assessments, we find well over 90% of the problems are just basic bucket ACL and policy mistakes. Object ACLs are also an issue, but slightly lower risk since you need the exact URL to find them if your bucket is otherwise secured properly.

    If you are curious about the other 4 Top AWS Security Mistakes, sign up for the Aug 14 webinar.

    — Mike Rothman

  • AWS Advances DevOps Agenda at Summit Event

    AWS Advances DevOps Agenda at Summit Event

    Amazon Web Services (AWS) has added services to its cloud platform designed to automate the creation of virtual private clouds (VPCs) and made available additional tools to build applications and integrate external data.

    Speaking at the AWS Summit event, CTO Werner Vogels said these latest services are part of an ongoing effort to automate DevOps processes to the point at which developers will need to write only the code for business logic that runs on top of the AWS public cloud.

    One of the latest tools the company is making available in pursuit of that goal is AWS Cloud Development Kit (CDK), a software development framework for defining cloud infrastructure in code as constructs and provisioning it through AWS CloudFormation. AWS CDK currently supports TypeScript, JavaScript and Python with support for C#/.NET and Java available in preview. That capability will eliminate the need to write low-level code using a different tool just to spin up a VPC on AWS, Vogels said.

    AWS CDK also allows organizations to design and compose their own custom components for managing infrastructure alongside their application code.

    The cloud services provider also announced AWS EventBridge, a programmable event bus for integrating data on AWS with data residing in software-as-a-service (SaaS) applications on any platform based on a serverless computing framework, and the general availability of AWS Toolkit for Visual Studio Code, a plug-in for the Microsoft integrated development environment (IDE) that makes it easier to build, debug and deploy code on AWS.

    Finally, the company announced that Amazon CloudWatch Container Insights is now available in preview. This offering provides access to automated dashboards summarizing the performance and health of their Amazon Elastic Container Service (ECS) and Fargate clusters by tasks. Those offerings come on the heels of AWS Fluent Bit, a tool for container logging being made generally available earlier this week.

    Vogels stressed that the way applications are built is now fundamentally different in the age of the cloud. Developers are making greater use of containers and serverless computing frameworks. The challenge with containers is that in the absence of a cloud service provider to manage the environment, the complexity of the underlying middleware and infrastructure conspires to reduce the amount of time developers spend on writing actual application code, he said.

    To address that issue, AWS is investing more in container and Kubernetes services as well as in on-demand serverless computing services such as AWS EventBridge that don’t require organizations to pay for idle processes, Vogels explained.

    In general, Vogels said AWS continues to advocate for the elimination of separate IT operations and cybersecurity teams in favor of a more unified approach leveraging automated services provided by AWS that ensure, for example, the entire continuous integration/continuous deployment (CI/CD) platform is secure. However, while the number of organizations embracing best DevOps and DevSecOps processes continues to increase steadily, it’s much less clear how many of them will standardize those processes on a single cloud platform at a time when reliance on multiple clouds also continues to increase steadily.

    — Mike Vizard

  • DevOps Chat: Global Intelligence for AWS GuardDuty, With Sumo Logic

    DevOps Chat: Global Intelligence for AWS GuardDuty, With Sumo Logic

    Wouldn’t it be helpful to know if other cloud users are seeing the same or similar attacks that you are? Security intelligence about cloud applications beyond just those you own and operate as an enterprise opens up a new dimension in attack visibility against even large sets of cloud apps.

    During AWS re:Inforce 2019, Sumo Logic announced it is extending its machine analytics and intelligence platform to include AWS GuardDuty. Dubbed Global Intelligence Service for Amazon GuardDuty, the new service is more than just a data aggregation and reporting play. The new service provides additional context around GuardDuty data by reporting attack information across multiple Sumo Logic customers using AWS GuardDuty. Essentially, it’s a “crowdsourcing” approach to reporting threat intelligence across the cloud.

    In this episode of DevOps Chat, David Andrejewski, senior engineering manager at Sumo Logic, joins us to talk about this new, more expansive threat intelligence service. More information about Global Intelligence Service for Amazon GuardDuty is available in the press release and website.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: Hi, everybody, this is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by David Andrzejewski, Senior Engineering Manager at Sumo Logic, and our topic is a real meaty one—Global Intelligence Service for Amazon Guard Duty. David, welcome to DevOps Chat.

    David Andrzejewski: Thanks so much. Thanks for having me on.

    Ashley: Well, we really appreciate you being on the podcast today. Would you start by introducing yourself, just tell us a little bit about what yourself and what you do at Sumo Logic, and maybe just a brief overview of Sumo Logic the company?

    Andrzejewski: Yeah, fantastic. So, Sumo Logic, what we offer is a cloud-based machine data analytics platform, which is a bit of a mouthful, but what that specifically means is a cloud-based, multi-tenant service where you can send your logs and metrics and data to our service for helping you monitor, troubleshoot, and secure your applications, platform, environment, infrastructure for your own application or your IT infrastructure.

    So, we offer this as a kinda cloud-based web app service. The idea here is, you’re sending us all of that data and we can kind of manage the scale, reliability, and analysis for you and kinda help you focus on running your business and kind of running your app and everything like that.

    Specifically at Sumo Logic, I manage the Advanced Analytics Team, and we’re focused in particular on ways that we can do especially kind of useful or interesting things to help customers get more out of their data. So, some of the features in the past we’ve worked on are log reduce, which kind of applies clustering techniques to your logs to kind of give you a more comprehensible snapshot as well as outlier detection and things like this. But our current focus right now is this kind of Global Intelligence Service that we’re really excited to be announcing at AWS Reinforce.

    Ashley: Great. Well, we are a DevOps and also a security podcast—talk a little bit about how Sumo Logic fits into the DevOpsSec world.

    Andrzejewski: Yep, absolutely. So, ultimately, in any of these situations, you really need some kind of—you need the ground truth actual data to really kind of know what’s happening in your environment, both from what kinds of experience are your customers getting, how is the, kind of the health of your machines or what resources are available, what is the performance of various services, you know, events are being kind of logged and audited and then you’re kind of pulling data together from all up and down the stack into one place.

    And that’s kind of where we come in as sort of one kind of one-stop shop for sending all of these very kind of heterogeneous logs into one service such that the teams that are responsible for up time from kind of a DevOps SRE perspective and security from all sorts of aspects of that, which increasingly is, like you said, kind of this DevSecOps. The developers themselves are often increasingly deeply involved in all of that, where they can kind of pull together the data from their own kind of custom application logs, the infrastructure, the actual machines, the operating system, the firewall, the databases, the third party services and kind of have it all in one place so that when you can sort of help detect when things do go wrong and then kind of identify, dig in, troubleshoot what’s going wrong and then once you fix it, you know, verify that it actually is corrected.

    Ashley: Mm-hmm.

    Andrzejewski: So, you would interact with the service by sort of operating queries against your data, creating dashboards and visualizations and using these dashboards or potentially setting up alerts to notify members of your team that some condition has been triggered that requires immediate attention or action to generate sort of scheduled kind of reports on these kinda things and in general, kinda analyze and interact with the data, the raw data being kind of sent out by your software, your machines and everything like that to really understand what’s going on in your app.

    Ashley: And are you oftentimes the system in the SOC, or do you connect into a different monitoring system, alerting system in the SOC?

    Andrzejewski: So, this is kind of a—as far as the SOC aspect, some customers are kind of using it for tracking these different kind of logging events, others are sort of, we have a lot of kind of integrations with other services to kind of consume data kind of emitted by those services. So, for example, AWS Guard Duty, you know, alerts like that.

    So, it all kinda depends on, there’s sort of a wide variety of ways that customers have kinda made use of Sumo Logic. So, yeah, in some cases, I think that that’s a reasonable use case that the [Cross talk]—

    Ashley: It’s probably, I imagine, where they’re starting from if they have systems all ready for you to integrate in, or maybe you become the main single pane of glass, if you will.

    Andrzejewski: Yeah, so, in the security use case, I think—again, it depends on kind of which sort of person within the organization might be most interested in that. Developers often, you know, given that this is a place, a single sort of solution that you can collect the logs for both these kind of monitoring SRE troubleshooting as well as some of the security use cases.

    Ashley: Mm-hmm.

    Andrzejewski: That ends up being kind of an appealing property that you can kind of have one service to collect the data, analyze the data for various use cases, right?

    Ashley: Okay, great. Well, this is the week of AWS Reinforce, and I feel like we’ve kind of laid out enough teasers about your announcement that you’re making this week. [Laughter] Tell us about Global Intelligence Service for Amazon Guard Duty.

    Andrzejewski: Yeah, yeah, we’re really excited about this. So, the basic idea here is that we’re going to add some additional context around the Guard Duty alerts where we can essentially take your alerts and kind of recontextualize them in terms of how frequently we’re kind of seeing them or they’re being sent or observed across others.

    So, if something is a very kind of rare threat that very few other organizations are kind of getting, seeing in their environment, then that might be something that you would want to prioritize more highly, right, if you are—you know, there’s going to be some, in any kind of alerting system like Guard Duty or others, there’s often going to be a very kind of high just baseline of sort of very generic, general threats that everybody kind of sees at some base rate all the time, and this is kind of one way that you can sort of kind of do some noise reduction on those and really kind of prioritize and focus on the ones that are a little bit more idiosyncratic or unique to your environment and more, perhaps more interesting and more relevant for kind of immediate investigation and triage.

    And this is kind of being exposed to the customers via an app, which is kind of, in the Sumo Logic scenario, it’s kind of some pre-defined analyses and dashboards against your Guard Duty data. So, your customer would sort of consume this service by installing the new Guard Duty with Global Intelligence Services app, and they would immediately get access to these new visualizations, analyses, and dashboards that kind of place their own Guard Duty alerts in this modified context, this more—you know, bringing in a little bit broader information about how often these are being seen in other environments, what are the general most common threats, threat purposes, various threat types seen across sort of the general customer population at Sumo Logic.

    And this will sort of kind of serve these multiple purposes for the user—on one hand, helping you kind of prioritize rare events, on the other hand, kind of giving you sort of this bird’s eye view survey of what’s generally out there, right? And that can help potentially in prioritizing or allocating efforts at improving your own security posture or efforts in, for example, you see that a particular kind of threat is especially prevalent, you know, elsewhere, that might be something that you say, “Okay, well, what do we have to be doing to make sure that we’re, our bases are covered with respect to that ourselves,” right?

    So, ultimately, we want to kind of show you your AWS Guard Duty threat data, but in this kind of enhanced context that’s hopefully going to be more meaningful and more relevant to you and help you make better decisions and kind of run your business more securely.

    Ashley: Okay, so, the benchmarking aspect of this, is that something that happens in real time? You know, we’re looking at the console, we’re getting alerted to say, “This attack is happening, and by the way, it’s happening on 20 percent of the other AWS customers” or however your present it, or is it more a retrospective looking at past data of what’s occurred over the last month, last quarter, last year? Or is it both?

    Andrzejewski: Yeah, so, this is something that we’re kind of, in some sense, as we are having conversations with customers, seeing what’s going to be most valuable for their use cases. Right now, the real time—is it real time, like, up to the second? You know, is that really something that’s going to be valuable, or is that going to be kind of too noisy of a signal?

    Ashley: Mm-hmm.

    Andrzejewski: So, right now, it’s derived from sort of recent history in some sense, and yet, potentially, that’s an interesting question of whether more very long term stuff, like, month over month is going to be useful for people, but you know, is it—right now, we’re not kind of focused on the use case of, you know, up to kind of real time real time.

    Ashley: Mm-hmm.

    Andrzejewski: Really, your data will still be real time, real time, but the question of whether that’s—you know, does the context need to evolve that quickly, or is that just going to kind of introduce more noise?

    Ashley: Right.

    Andrzejewski: Part of the advantage of kind of using a little bit of a recent historical window is looking at—you know, it gives you a little bit broader pool of data to kind of contextualize your real time activity against, right? So, again, it’s all about sort of your data and kind of seeing how—you know, how it kind of looks in context of this broader view and evolving the broader view in real time. It’s an interesting thought.

    Ashley: Well, that makes sense, too, because you may be having a false positive or something. Not everything that is being highlighted by Guard Duty might be as relevant as other things, so if you have a chance to filter some of that information out to say, “This is what’s relevant” versus, if everything’s real time, you could introduce some noise, maybe some things that aren’t as helpful, too.

    Andrzejewski: Yep. Yeah, exactly. So, it’s—that’s definitely a potential tradeoff, there.

    Ashley: So, what would you learn—because now, there must be some kind of an opt-in where you’re looking at events that are happening across different customers, AWS users—what would you learn by knowing that a whole other population of people are seeing this same kind of attack versus if you were only looking at your data, if you had known that you would do something differently?

    What would be a scenario like that where this would be super helpful?

    Andrzejewski: Right, so, the—you know, one kind of particular case sort of going the other way of a rare event might be that, you know, again, so, some of these threats are going to be very kind of common, and it ends up that that’s not something quite as potentially actionable or something that you need to jump on right away, but there’s still kind of part of your Guard Duty feed and they’re going to kind of potentially be a little overwhelming when you’re looking at everything. Whereas this more kind of like rare threat type, this—by kind of knowing, “Okay, this is a little bit out of the ordinary in that lots of customers are not seeing this,” that kinda helps it pop to the top of your queue and maybe you are going to allocate your very limited time and attention to focus on that.

    It might be a little more challenging to prioritize or see that kind of rare event pop, because you don’t necessarily know that it’s rare, you just sort of see it amongst all of the other kind of more common alerts and more common threats.

    Ashley: Right. This is a rare event, something that’s unusual, you should take a look at this versus something that’s happening normally across multiple AWS accounts.

    Andrzejewski: Exactly. That’s exactly right. And so, that potentially becomes a valuable tool to kind of—like I said, ultimately, teams everywhere are ultimately struggling with the scarce resource of your time and attention.

    Ashley: Mm-hmm.

    Andrzejewski: And anything that can kind of bring in some additional context information to help you allocate that time and attention more efficiently, more effectively is potentially, hopefully going to give some good value for our customers.

    Ashley: Well, many resources are precious resources, and certainly security folks are definitely in that precious category. So, being able to assign them to something you know is something we need to look at versus kinda chasing things that we really don’t need to be paying attention to would be super valuable.

    Andrzejewski: Absolutely.

    Ashley: So, obviously, Sumo Logic does a lot of other things in your products. You have operational analytics, business analytics, those kinds of things. Does this Global Intelligence Service tie into those in some way, to give you some additional broader insights, or this focused really—just really heavily on the security aspect of what’s happening in Guard Duty?

    Andrzejewski: No, absolutely—that’s a really good point. So, right now, kind of, very sort of aligned with Global Intelligence is some of the stuff that we’ve been doing a few years now around the modern app report and similar where kind of giving these sort of horizontal views of technology choice trends and various kind of software choices, infrastructure, cloud provider choices, kind of adoption of various tools and technologies across the industry, and that definitely kind of extends into the, kind of the DevOps SRE space. And again, hopefully the idea there is that these tools kinda give you this sort of broader context about actual hard data where various—you know, where the market in general is on their journey to cloud adoption or their migration to Docker or Kubernetes or other kind of related technologies.

    Ashley: Mm-hmm.

    Andrzejewski: So, that’s one place where there’s already something in place at Sumo Logic that’s kind of bringing that broader sort of horizontal context to bear to kind of help teams kinda make more effective, more informed decisions and, again, prioritize and allocate their attention and efforts in kind of using that extra information to kind of know, actually, from the hard data where others are actually at.

    Ashley: Mm-hmm.

    Andrzejewski: So, right now, this particular announcement at AWS Reinforce is focused on Global Intelligence Services for AWS Guard Duty in particular which, as you said, is absolutely a very kind of security focused use case.

    Ashley: Mm-hmm.

    Andrzejewski: But in general, the general sort of Global Intelligence Services, you know, kind of platform and approach is something that we’re also actively thinking about how can we help in the operational analytics use cases? Where are there places where that broader context is going to help our customers make better decisions or solve their problems more quickly?

    So, right now, we have this Guard Duty announcement, definitely focused on security, but in general, we see Global Intelligence Services as a very powerful kind of horizontal enabler across the SRE DevOps kind of side of the house as well.

    Ashley: Mm-hmm. Yeah, it seems like an area that would be ripe for machine learning—yeah, the statistical analysis that you could do across multiple areas of the analytics that you’re doing. You know, that’s what data’s all about, right? What insights and learnings can you get from that that brings additional value benefit to the business?

    Andrzejewski: Definitely, yep.

    Ashley: Well, good. Is this product available now? Is it coming out some time down the road? How soon can we get our hands on this?

    Andrzejewski: So, as of the kind of announcement, it should be available to Sumo Logic customers at the enterprise level, and again, it would be a matter of installing the app and kind of start—obviously, in this case, you would need Guard Duty as well. I can provide some links to go out with this, if that’s helpful—

    Ashley: Sure.

    Andrzejewski: – that would have kind of links to the specific resources that would be involved in getting started and getting set up with this, but yeah.

    Ashley: Okay, great. It sounds like a pretty easy thing to do to get started. If you’re already running Sumo Logic, installing the app is not a heavy lift, it’s pretty straightforward.

    Andrzejewski: Absolutely. And the other aspect, of course, having Guard Duty in place as well. Which, again, is—in the links I’ll send, should be, is a pretty painless process.

    Ashley: Excellent! I also want to mention, too, that Sumo Logic is located in Booth 714 at the Reinforce conference. So, definitely recommend folks stop by and maybe see a demo of this, get to talk with folks about the new announcement.

    Well, our podcasts have no problem flying by quickly, because we talked about great things and I appreciate you doing that with us. David, I’d like to thank you, David Andrzejewski, Senior Engineering Manager at Sumo Logic, for joining us.

    Andrzejewski: Well, thank you so much for having me. It’s been a really great conversation—super, super interesting to talk about.

    Ashley: Great. It’s good stuff, and I wish you and Sumo Logic the best at the show this week.

    I’d like to also thank you—you, our listeners—for joining us. This is Mitch Ashley with devopsy.kinsta.cloud, and you’ve listened to another DevOps Chat. Be careful out there.

    — Mitchell Ashley

  • DevOps Chat: ‘War, Peace & IT’ With Marc Schwartz

    DevOps Chat: ‘War, Peace & IT’ With Marc Schwartz

    Marc Schwartz is one of those rare individuals who is not only a great IT exec, but he also can analyze what works and doesn’t work and write it up in a way that helps others. His “A Seat at the Table” is still one of my favorite DevOps books.

    Marc’s new book is called, “War, Peace & IT.” You can download an excerpt of it here: dl.orangedox.com/WarPeaceITExcerpt

    In this DevOps Chat, we speak with Marc about the book and some of the lessons behind it. Have a listen and check out the book ASAP.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, it’s Alan Shimel for devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. I’m really delighted to have as my guest on DevOps Chat today none other than Marc Schwartz. Marc, if you don’t know, is an AWS executive, multiple author on some of the greatest books on DevOps you’re gonna read—”A Seat at the Table” is still one of my all-time favorites—and I think as he’s described in the latest blurb on IT Revolution, an iconoclastic CIO.

    Marc, welcome to DevOps Chat.

    Marc Schwartz: [Laughter] Thanks. I’m glad to be here.

    Shimel: So, Marc, what is an iconoclastic CIO?

    Schwartz: [Laughter] I like to question assumptions. You know, I find that, very often, people are frustrated. There are obstacles to what they’re trying to accomplish, and the problem is that they’re constrained in all sorts of ways, and if you don’t question the assumptions and knock away some of those constraints, they’re gonna continue to be frustrated—so, that’s what I’m all about.

    Shimel: Yep. I agree. So, I don’t want to spend a lot of time, but Marc, we mentioned you’re from AWS, but I didn’t really mention a role. Just so our audience, who may not be familiar—what currently is your role with AWS?

    Schwartz: They call me an enterprise strategist, and what means is, I’m part of a small team of ex-IT leaders, many of us ex-CIOs or CTOs of large enterprises who did digital transformation, moved to the cloud, and what we do now is, we work with executives from large enterprises to help them overcome some of the challenges that they face as they’re trying to do a transformation.

    So, often, it’s things like cultural change, or organizational change, or organizational structure, financing models, people skills, things like that. And we speak at conferences, we write books and blogs and we work with individual customers.

    Shimel: Very cool. No, let’s jump in, Marc. You’ve got a new book, just recently came out. It’s available on Amazon, by the way, in case anyone is interested, as well as Barnes and Noble, 800-CEO-READ, Indiebound, BAM and any other place where you can find decent books. It’s called “War and Peace and IT: Business Leadership, Technology, and Success in the Digital Age.”

    Marc, share with our audience—what’s the book about?

    Schwartz: Well, the book is essentially an answer to a challenge that people gave me after they read “A Seat at the Table.” Because, in “A Seat at the Table,” it was written primarily for IT leaders, and throughout that book, I made the argument that we needed to get rid of this business versus IT distinction, you know? The way we talk about the business and IT as if it’s two separate things, the way that we see IT as taking requirements from a business and executing on those requirements, but that way of thinking is no longer effective. We need to find a way to bring IT into the business itself.

    And people responded very well to that message in “A Seat at the Table,” but they said, “This doesn’t work unless somebody tells the business folks the same thing, you know? [Laughter] We can’t just change the relationship ourselves. So, somebody should write a book for the business executives to explain to them what you’re talking about.” And I realized that that same book, if I could write it, would also help a lot of the executives that I was talking to as part of my job who were facing impediments to their transformations.

    So, I wrote a book that essentially said, “One of the big impediments to your transformation is that the relationship between the business and IT is not the right relationship, and here’s how you can change it, and here’s why it will make a big difference for you.”

    Shimel: Excellent. So, you know, I think you laid it out pretty well there, Marc, for people. You know, I’m just back—we were talking offline and I told you I was at a KubeCon + CloudNativeCon in Barcelona last week and it was another great event, 7,500 people.

    And what really kinda hit me square between the eyes was the fact that though this is a fairly technical event, right? We’re talking about cloud native and a lot of open source software programs and ops procedures, the business angle of it, the amount of business people who are there trying to get smart about what’s the latest trends, what’s the wave, if you will, in IT, and how could they leverage things like serverless or Kubernetes and some of these newer things to make their business better, not necessarily because it’s cool or, you know, shiny trinket syndrome or anything like that.

    But really, how does—when does rubber meet the road with this technology? And, you know, I can’t help—I mean, I’m heartened by it, Mark, but I still don’t know if business executives are all bought into that.

    Schwartz: Yeah.

    Shimel: Your thoughts?

    Schwartz: Well, I don’t think it’s about the cloud, per se, or DevOps or whatever it is. What really is important is how the cloud and DevOps and new ways of doing IT empower businesses to do new things that they couldn’t do before. And I think that’s what the non-IT business executives need to learn about what’s possible now that was not possible before.

    And the reason why they have to, why it’s an urgent matter is because their competitors are, you know? Whatever tools you have available to you, your competitors also have, and if you’re not using them, and your competitor is, then you’re at a business disadvantage. And I think mostly, it comes down to a question of speed, one way or another. So, using the cloud well, using DevOps well, lets you move at a much faster pace to get IT capabilities to change your business.

    And that pace is important not just for the obvious reasons that you need time to market, you need to get first mover advantages, you need to respond quickly when you competitors disrupt you, but also for more subtle reasons, because we now use speed as a way to test out ideas quickly and see if they’re gonna work, and manage our risk by testing out ideas before we fully commit to them, and we use speed to get feedback from customers and users and alter our product offerings based on what we’re able to learn, and we use speed to mitigate risk to improve our security posture, to improve our availability of systems.

    So, speed becomes a really important factor, and the cloud, DevOps, you know, their technical implementations, but they happen to help companies achieve speed, and that’s the important message, I think, that executives need to take away.

    [su_pullquote align=”right” class=”do_blockquote”]I don’t think it’s about the cloud, per se, or DevOps or whatever it is. What really is important is how the cloud and DevOps and new ways of doing IT empower businesses to do new things that they couldn’t do before.[/su_pullquote]
    Shimel: I agree. But, you know, there’s an old saying—speed can kill. Great speed kills, too, right?

    Schwartz: [Laughter] Well, if you’re using the right model for speed, it doesn’t kill, and the reason for that is that the models that use speed usually also use automated controls or guardrails. So, what you want, and the idea is, you want to be able to move really quickly, but within the boundaries of guardrails that enforce compliance, enforce security, enforce quality and it’s possible to do both. We always thought, in the past, that there was this trade-off between speed and stability or speed and quality, and it turns out that, in this DevOps world, in this cloud world, speed—increased speed actually increases quality, resilience, availability, security and all of those things. It’s a question of learning how to use speed for those purposes.

    Shimel: Yeah. And that’s really—I mean, tongue in cheek, speed kills. What I was really trying to get at is, speed for speed’s sake alone is, you know, speed has to be wielded—with great power comes great responsibility, right?

    Schwartz: [Laughter] Yes. You have to achieve the speed for competitive reasons and for business model reasons, as I was saying before. So, it’s not really an option, but you do, at the same time, have to learn how to use that speed as a way to reduce risks and improve the quality of what you’re offering.

    Shimel: Exactly, exactly. So, what’s the reception been, Marc? What kind of feedback have you been getting from people?

    Schwartz: Well, the book was just published a couple of weeks ago, really. Where are we? Yeah, May 14th was the publication date, so a couple of weeks. And people seem to be enjoying it. I was a little bit playful in writing the book, and I think there are some aspects of it that are entertaining people, and that makes me feel very good.

    But from the feedback that I’ve been getting, people are seeing it as the successor to that previous book, just as I had hoped. You know, this is the way of looking at a similar message, but very much from the point of view of non-IT business executives. You know, how should the CFO be thinking about the new world, the new digital world, how should a CEO be thinking about it, how should the board of directors be thinking about it? And I’m very happy that people see it as valuable on that front, and a lot of people are saying, “Well, I’m gonna have my entire executive team read this, and it’s gonna stimulate some great discussions,” so I’m very happy about that.

    Shimel: That’s fantastic, great stuff. Let me bring in another aspect of it and get your thoughts on it, Marc. You know, we’ve heard a lot over the last, I’ve heard it maybe over the last year or 18 months—DevXOps, right? We have DevSecOps, DevBusOps, DevHROps—DevWhateverOps, right? DevXOps. And I think there’s a little bit of a reverse equation, here. Business leaders have to be more involved in IT and have to learn how to harness IT to further their businesses’ goals and delight customers, as you say, right?

    At the same time, we can take lessons from IT and things we’ve learned in IT with things like DevOps but not only DevOps. But IT has had, in a relatively short period of time, had a rather torturous road of self-improvement and of learning new things and evolving to things like DevOps that we see. What can businesses do—not harnessing IT, per se, but maybe saving some idiot taxes?

    Schwartz: [Laughter]

    Shimel: You know, from mistakes and lessons learned from IT and apply them to general business?

    Schwartz: Yeah. I think—I also joke about DevOps, DevSecOps, DevSecBizOps, DevSecFinBizOps—you know? You can keep adding to that.

    The real point is cross functional teams, right? DevOps meant we’re gonna have a cross functional team that has dev skills and ops skills, and at some point we realize that that set of skills is not quite enough, they also need to have security skills and you can keep adding to that infinitely, because the bottom line is that functional silos don’t work very well in a fast moving environment like we have in the digital world today. Functional silos slow down your response, they slow down your ability to innovate and grasp new opportunities and respond to change and things like that.

    So, that’s really what was behind the DevOps, DevXOps concept. And I think businesses find this, in general, that when you have functional silos and they operate as silos, you introduce waiting time when you pass things from one silo to another. You reduce innovation because each silo thinks of itself as having its own responsibility instead of joint responsibility for results.

    So, the lesson that I think everybody can take away from what’s happened in IT is that, to respond quickly in an environment where you need to respond quickly, you have to do away with silos and instead focus on cross-functional teams that [Background noise] objective. At least, that’s my biggest takeaway.

    Shimel: Sure, sure. [Background noise] I’m sorry, my machine’s—uh oh, here we go. You know, I’m on my, I’m not gonna give you the brand of the surface I’m on. There’s only one company that makes surfaces.

    Schwartz: [Laughter]

    Shimel: It’s on a roll of reminders.

    Schwartz: Excellent sound effects.

    Shimel: It doesn’t shut up, here. Look at this, this is—

    Schwartz: That’s okay, we could use some background music.

    Shimel: Yeah, I guess so. It’s happening—I don’t know if I like the rhythm, but anyway. [Laughter] So, Marc, a good question is—look, you’re globetrotting around the world evangelizing your AWS role and everything. And yet, you know, over the last, let’s say, three years, four years, you know, this is the third book, actually, isn’t it?

    Schwartz: Yeah. It’s the third one, and I’ve started to work on a fourth.

    Shimel: Can you give us a sneak preview of the fourth, or it’s too premature yet?

    Schwartz: Well, it is too premature. It can go in lots of different directions, but I can tell you what I’m thinking a lot about right now, which is bureaucracy and how to overcome it on the one hand, but even more than that, I think you can harness it in many ways, if you are trying to become a digital organization.

    And, you know, I think that fulfills my role as an iconoclast. I think bureaucracy is one of those words that people have a very strong negative reaction to. But, of course, in my role before AWS as a government CIO, I, of course, witnessed, experienced a lot of bureaucracy and developed strategies for coping with it that I think would be useful to a wide audience. So, I think I’ll probably write something that has to do with that.

    Shimel: Okay, pretty cool. And you know what? We mentioned “A Seat at the Table” and, you know, there’s nothing like your first book, but I feel amiss if we shouldn’t also mention the title of the second book, right, and that is “The Art of Business Value,” right?

    Schwartz: Yeah. “The Art of Business Value” was sort of my initial train of thought. You know, I said to myself, we’re talking a lot about business value and how important it is to guide what you do in the IT world by delivering business value and how you should make prioritization decisions by business value, and it occurred to me that nobody was really talking about what that meant, you know? What is business value, exactly?

    And when I looked into it, I realized that people were not really interpreting it, in many cases, in the best way, and that if you thought a little bit more deeply about that, it would change the way that you practiced IT.

    So, that was what drove that first book, and in that book, I introduced the question of what does a CIO have to do with figuring out what business value is? How does a CIO make sure that he or she is delivering business value? And I realized that that was a good topic for a second book, actually. [Laughter] Something that I really needed to dive more deeply into. So, that’s what generated “A Seat at the Table.”

    And then, as I said before, once I had written “A Seat at the Table,” it was sort of clear to me what the next thing had to be. It had to be the book that I hope will speak to non-IT business leaders. So, that explains the sequence of the three books in my head.

    Shimel: Very cool. So, Marc, is there a tour going on with the book? Are you out speaking—I mean, are you gonna be, for instance, at DOES: London, coming up?

    Schwartz: I will be there. I speak at a lot of conferences in my role with AWS, and so, I’m on a permanent tour, essentially. [Laughter] I speak all over the world, and of course, I write blog posts and I’ll be writing some more books. And it all ties together with my role at AWS, where I’m part of a team that’s trying to lead enterprise executives into the future. You know, here’s the way that we see things developing, here’s how you can take advantage of what’s going on with the cloud and with DevOps.

    And so, I don’t have a speaking tour specifically about my book, but a lot of the topics that the book touches are also the topics I talk about for AWS.

    Shimel: Great. Great, great, great. Marc, we’re about out of time. We mentioned you can get the book on Amazon. Up on devopsy.kinsta.cloud, I think we have a link to a, I don’t know if it’s the first chapter or an excerpt of the book. We’ll put that in the show notes.

    Look, man—first of all, congratulations on what sounds like another great book. I’m actually, I’m gonna be on the road a whole bunch in June, so, I’m gonna have a chance to catch up on my reading on planes and stuff, so I’ll get a chance to read this one. And then maybe we can circle back, maybe at the end of June in London, and we can talk a little bit more in depth about it.

    Schwartz: Uh huh. Sounds great.

    Shimel: Great. Hey, Marc Schwartz—iconoclast, CIO, or iconoclastic CIO as well as multiple award winning author, a real DevOps expert. His newest book, “War, Peace & IT,” you can find out more, get an excerpt from it on devopsy.kinsta.cloud, you can get it on Amazon or any number of good quality book sites.

    Hey, Marc, congratulations, and thanks for being our guest on DevOps Chat.

    Schwartz: Thank you very much.

    Shimel: Alright. Enjoy your day and we’ll be in touch.

    Schwartz: Thanks.

    Shimel: Alrighty. Bye bye. This is Alan Shimel for DevOps Chat and devopsy.kinsta.cloud. Have a great day.

    — Alan Shimel

  • RDX Eases Oracle Database Migration to PostgreSQL on AWS

    RDX Eases Oracle Database Migration to PostgreSQL on AWS

    RDX this week launched a set of tools designed to make it easier to convert applications running on an Oracle database to Amazon Aurora, an instance of an open source PostgreSQL database running on the Amazon Web Service (AWS) cloud that is compatible with Oracle databases.

    Announced at the AWS Summit event in Washington D.C., the clckwrk Refactoring Service for Oracle promises to reduce the amount of manual effort associated with refactoring applications for the Amazon Aurora platform by automating as much as 60% of the database schema conversion process.

    James Ball, chief strategy officer for RDX, said in addition to reducing costs, many organizations want to shift applications to open source databases in the cloud to further their DevOps ambitions. Each time an application needs to scale, an organization will incur additional Oracle licensing fees. In contrast, an open source database enables DevOps teams to scale database instances as required automatically, he said.

    AWS has been making a concerted effort to convince customers to give up proprietary databases in favor of open source databases running on its cloud platform. That competitive pressure is a driving force behind new alliances that previously would have been considered unimaginable. Oracle, for example, recently partnered with Microsoft to promote the integration of the Oracle Cloud platform with the Microsoft Azure cloud. Oracle would prefer its database run on its cloud, but through the alliance is signaling its willingness to work with Microsoft to deploy Oracle database on Azure, a cloud platform that is the leading industry alternative to AWS. Microsoft, of course, has its proprietary database to adopt in the form of Microsoft SQL Server.

    Ball said RDX not only provides tools to accelerate the conversion to Amazon Aurora, but it also provides managed services for customers that want to outsource the management of Amazon Aurora to RDX.

    It takes most organizations several months to migrate a legacy application on to a public cloud. AWS has a vested interested in helping accelerate that process whenever possible. But many organizations that have embraced best DevOps practices find their efforts are hampered by reliance on proprietary databases. In fact, one of the reasons so many developers tend to favor open source databases is to avoid all the headaches associated with getting approvals for new database licenses.

    It’s unclear to what degree the rise of DevOps in the enterprise will ultimately shift database allegiances. Oracle is trying to maintain as much control as possible by making its databases available as a managed service that eliminates some of the purchasing overhead associated with spinning up new instances. However, the cost of an Oracle database license is still buried in the cost of the managed service.

    Not every organization is looking to transition away from a proprietary database, but there is now enough momentum around the “open source first” movement to make it worthwhile to build tools that automate the refactoring of legacy databases not just to reduce costs, but also to make IT a lot more agile.

    — Mike Vizard

  • Logz.io Unveils Azure Open Source ELK Monitoring Solution

    Logz.io Unveils Azure Open Source ELK Monitoring Solution

    Needless to say, DevOps as a practice is largely fueled by open source. The growth and popularity of open source tools such as Elasticsearch, Kubernetes and others are driven largely by the increased flexibility, innovation and familiarity of options. While this isn’t news for developers (65% of which are active open source contributors), leading cloud providers are taking notice, as they try to increase their appeal to DevOps engineers to finally win the cloud wars.

    Currently, AWS is the leading cloud provider. However, with Azure gaining more and more momentum, all three major cloud providers are scrambling to offer their customers the best open source options and empower them to use the tools they know and love in a manner that is convenient and native to their cloud environments.

    Going Big on Open Source

    Today, Logz.io and Microsoft Azure announced that our ELK-as-a-service will now be offered on Microsoft Azure. Logz.io offers an intelligent machine data analytics platform built on ELK and Grafana, two of the most popular open source monitoring and troubleshooting tools. Our platform enables engineers to easily deploy, run and scale ELK without the hassle and pain of maintaining and managing the stack themselves.

    Now, with Logz.io for Azure, users can use open source ELK as a highly integrated experience in Azure. The new offering includes seamless integrations to various Azure services, out-of-the-box Azure dashboards, built-in alerting and crowdsourced machine learning capabilities.

    The integration between Logz.io and Azure is not only significant in the context of the cloud wars, but in the culture of Microsoft itself. While the company was previously associated with a more proprietary structure, they have made tremendous strides in recent years toward empowering open source by opening its .NET framework and Azure SDK, as well as their acquisition of GitHub.

    According to Julia Liuson, corporate vice president of Microsoft’s developer division: “At Microsoft, we see open source as a major driver of innovation within the engineering community. Since we strive to empower our customers and facilitate their success, it is important to us to meet them where they are by making it easy to access and use open source tools. Our partnership with Logz.io will strengthen Microsoft Azure ties with the developer community, enabling teams to easily build and monitor their products using ELK.”

    AWS vs. Google Cloud

    Microsoft’s investment in ELK  comes just in time to provide an answer to its competitors, AWS and Google Cloud. Just two months ago, AWS announced Open Distro for Elasticsearch, which in their own words “is a value-added distribution of Elasticsearch that is 100% open source (Apache 2.0 license) and supported by AWS.” The new offering includes managed Elasticsearch and Kibana along with features such as built-in alerting, security and more. This was AWS’ most recent appeal to the engineering community, offering one of the most popular monitoring tools as a managed service on AWS.

    Following this release, Google Cloud announced its own open source offering featuring integrations with leading companies such as Elastic, MongoDB, Confluent and more. According to Google, the purpose of  these integrations was to “provide users with a seamless user experience and the ability to easily leverage these open-source technologies in Google’s cloud.”

    Is Open Source the Future?

    If what’s happening between cloud vendors is any indication of future trends in the engineering space, it seems imperative that all organizations targeting DevOps teams must consider how they’ll fuel open source and empower its vast user community. At Logz.io, we are proud to be part of this initiative.

    Overall, it is great to see the strides large organizations are taking in furthering the interests of the developer community and enabling them to work using the tools, methodologies and strategies that best suit their workflows and ideals.

    — Daniel Berman