devopsy.kinsta.cloud

Tag: AWS

DevOps and ITaaS: Two Sides of the Same Coin
It seems IT practitioners haven’t realized yet the scale of digital disruption. Mired in the old IT thinking, they firmly believe that sophisticated tools are all businesses need to survive the ongoing digital disruption waves. Most of them warn, “Digital transformation is changing the way we do business, the way we deliver IT services must change, businesses need to adapt the way they work.” The problem is, there’s nothing new in what they offer to help businesses work the “right” way.

Indeed, the same old IT thinking’s practices are still applied with the illusion that deployment automation and configuration management tools on their own will make businesses agile, responsive to market opportunities, and therefore in line with the digital economy’s competitive challenges. That belief misses the point for two reasons: First, it’s an incorrect understanding of what business agility is, and second, it’s a misunderstanding of the global competitive environment’s transformation.

Turning your IT organization lean and agile is the competitive differentiator your business lines desperately have been calling for. Authentic DevOps implementation is the way to go. That’s the price to pay to keep up with the pace imposed by new leading-edge companies such as Google, Apple, Uber and the like.

The changing competitive environment forces businesses to focus on innovation, market responsiveness and timely revenue

A digital economy threatens the existence of all businesses. Yes, you read it right, their existence is at stake. And that fact is underestimated by IT vendors and consulting firms.

As illustrated, we’re shifting from business environments where established businesses (incumbents) control a large percentage of market segments to competitive contexts where, attracted by the amazing digital opportunities and the adoption of cloud which significantly reduces infrastructure expenses, a growing number of startups disrupt entire industries:

The changing competitive environment (Credit: Philippe Abdoulaye ITaaS Now LLC)

The proliferation of startups created by highly qualified unemployed seniors and creative entrepreneurs, coupled with the propagation of highly innovative digital services (e.g., Internet of Things [IoT] applications) substituting traditional services, have been disrupting the vast majority of industries.

Several examples substantiate that fact: the taxi industry shaken by the emergence of technology-driven companies such as Uber, the banking industry facing new breed of competitors including PayPal, Google and Apple; and the healthcare industry increasingly disrupted by companies such as GlowCap.

To grow and prosper in the digital era, your scale, brand equity or market capitalization aren’t enough anymore. To succeed, your business lines must be on top of the digital economy’s competitive standards: competitive innovation (the ability to continuously improve customer experience), market responsiveness (the ability to deliver added-value services to the right market segments at the right moment) and timely revenue (the ability to continuously increase customer value).

The digital economy has taken IT to an inflection point—deploying sophisticated tools isn’t enough; being lean and agile is the way to go

The obligation for business lines to focus on innovation, market responsiveness and timely revenue is unprecedented for techies; it forces them to think beyond tools and address issues as unusual as organizational and operational effectiveness.
Indeed, innovation, market responsiveness and timely revenue assume greater cross-functional collaboration, IT focus on business priorities, organizational flexibility and accelerated time to market. Make no mistake, you’ll never achieve them by only deploying tools, but by also making your IT operating model flexible through proactively eliminating the organizational and operational dysfunctions that prevent cross-functional collaboration, fast problem-solving and decision-making.

That’s what is meant by making the business agile.

The new IT thinking is summarized in that simple arithmetic: Digital Competitiveness = Lean and Agile Operating Model + Effective and Efficient Tools. Without the lean and agile operating model, innovation and timely revenue are wishful thinking. Without effective and efficient tools, innovation and market responsiveness are wishful thinking as well.

The organizational and technological dimensions are complementary. Adam Jacob, Chef’s CTO, confirms that fact in a very informative article, “The Secret of DevOps: It’s Always Been About People, Not Technology.” This is where DevOps can help businesses succeed.

DevOps is the IT asset businesses need to succeed in the digital economy

Contrary to a now (unfortunately) established belief, DevOps isn’t a technology; Chef, Puppet, Jenkins, OpsWorks, alike aren’t DevOps, they’re only pieces of it. It’s important to be clear about it.

As I’ve been implementing it, DevOps is a system of principles, practices, roles and responsibilities, governance structure and tools purposely arranged to establish cross-functional collaboration, enable accelerated applications delivery and, in turn, make the business responsive to market opportunities:

DevOps Model (Credit: Philippe Abdoulaye ITaaS Now LLC)

As illustrated, DevOps as a business capability is made up of three components: the Rules of the Game, the IT Operating Model and the Continuous Delivery Pipeline (CDP). Let’s explore them.

DevOps Rules of the Game create the conditions for lean and agile IT environments

Rules of the Game are a fundamental dimension of DevOps: They enable the establishment of lean and agile environments. I refer to them as the DevOps non-technological piece.

In practical terms, they’re a set of principles, practices, roles and responsibilities and tools institutionalized to ensure accelerated applications delivery.

They serve as catalysts to specific organizational and operational effective practices. Let’s take the principle, “Facilitate with system thinking (involvement of all relevant stakeholders)”—it forces application development stakeholders to adopt a collaborative mindset, which in turn breaks down organizational silos to eliminate the impediments to operational flexibility and institutionalize cross-functional collaboration.

They’re the poor cousin of today’s DevOps practices, which ignore them in favor of more technical and technology issues. That’s a terrible mistake that reduces DevOps influence vs. the digital economy’s challenges.

The agile IT Operating Model establishes the lean and agile work environment

The IT Operating Model reflects the interactions that take place across your application development value stream. In practical terms it represents the work environment derived from the application of DevOps’ rules of the game.

I always remind my clients that DevOps’ rules of the game aren’t theories, they’re the foundations of a tangible strategic capability: the lean and agile IT environment. They demand a thorough transformation of your application development work space to put together business, application development, and operations:

The lean and agile IT work space for the digital era illustrated (Credit: O.C. Tanner)

The picture illustrates what’s meant by lean and agile IT environment, it substantiates Adam Jacob’s point that “Fundamentally DevOps is about taking the behaviors and beliefs that draw us together as people, combining them with a deep understanding of our customers’ needs, and using that knowledge to ship better products to our customers.”

The IT operating model is the poor cousin of today’s DevOps practices which ignore it in favor of more technical and technology issues. Once again, it’s a terrible mistake that reduces DevOps influence vs. the digital economy’s challenges.

Make no mistake, technology matters, the CDP helps to meet digital technologies challenges

The importance of the rules of the game and the resulting IT operating model don’t suggest at all that technology doesn’t matter. Make no mistake about it, the CDP is essential; I refer to it as the DevOps technological piece. Through deployment automation and configuration management tools, it significantly accelerates application development, testing, resources provisioning and deployment.

However, one thing to be aware of is, digital economy raises technological challenges that transcend speedy delivery issues. They include:
- The importance of IoT
- The pivotal role of predictive analytics
- The increasing importance of APIs
As a matter of fact, IoT devices generate massive data and making use of it to generate revenue requires big data technologies to integrate, automate and analyze them.

On the other hand, this ocean of IoT data has generated an API business; the idea is, to monetize their data and generate revenue, businesses must provide their customers and partners with APIs that easily give access to their systems.

Implementing the CDP as part of PaaS cloud services can help to prosper in the digital economy; it allows businesses to benefit the cheap and on-demand computing, storage and network resources provided by the cloud.

Takeaway

In hindsight, with a business-oriented perspective, the ITaaS delivery model and DevOps follow the same goal: Provide business lines with the lean and agile IT environments they need to be flexible enough and responsive to meet the competitive challenges of digital economy:

The ITaaS Delivery Model (Credit: Philippe Abdoulaye ITaaS Now LLC)

Like DevOps, the ITaaS delivery model (…) takes advantage of an agile IT operating model to help businesses succeed in the digital era. It implements consumer-provider relationships across the IT value stream to ensure organizational flexibility and operational agility.

CIOs must realize that what matters to the business is to continuously generate revenue through improving customer experience and increasing customer value. Deploying applications at the speed of light for the sake of speed doesn’t make sense from a business perspective; as long as expected business benefits aren’t achieved, speed is a useless gadget.

Implement DevOps as what it is: a three-dimension IT capability including rules of the game, IT operating model and, of course, continuous delivery pipeline.
April 6, 2016
6 Key Considerations When Migrating to AWS

Every day, more and more organizations take the plunge and migrate to the Amazon Web Services public cloud. What they leave behind is the cumbersomeness of the traditional data center, but despite the cloud’s benefits in scalability, agility and efficiency, they discover a new set of challenges that need to be overcome.

Gartner estimates that more than 50 percent of enterprises will have adopted a hybrid cloud approach by 2017. Transitioning from a traditional on-premise IT infrastructures to a public cloud can be daunting, however, and success requires a different mindset and range of skills. This shift can be particularly challenging for traditional enterprises because they have a lot at stake. Here are the five main challenges in moving to the AWS cloud.

Cloud Finances

Different organizations have different financial approaches, and their choices of IT infrastructure reflect this fact. For some, the on-premise approach of making a large, upfront capital expenditure to purchase infrastructure and then capitalizing the investment over time may be the favorable option because they prefer to keep complete control over their IT environment. However, for others, a heavy initial expense is not ideal, so a cloud approach with only ongoing, operational costs is more fitting. This option may be particularly suitable for organizations with fluctuating needs on a monthly basis, as an on-premises data center will not offer them the flexibility they require.

Regardless of the approach, it is important to compare the respective costs before deciding which one is most suitable. The best option may be to combine both on-premise and cloud to create a hybrid cloud environment. This will allow for steady workloads to be kept onsite while bursts in demand can be processed by an on-demand, public cloud.

Security, Availability

The idea of handing over all of your data to a public cloud provider can be a daunting because of the obvious security and availability concerns. However, public cloud providers must adhere to strict compliance protocols and can implement and maintain much higher security levels than on-premise installations because they have more available resources.

In fact, enterprises have avoided migration to the cloud for a long time due to security concerns. However, this is slowly changing, as IT leaders now realize that running on the public cloud actually can bring the advantage of greater security because AWS has industry-compliant cloud infrastructure (HIPAA and PCI-DSS standards, for example).

Nonetheless, AWS customers must understand their responsibility in protecting user data and implementing a service level agreement (SLA). For more information, take a look at the AWS “sharing the responsibility” model.

Rehost (Lift and Shift) vs. Rebuild (Re-architect)

Migration to the cloud requires some amount of preparation before in-house applications can be moved. Gartner describes five different methods of application migration, and two of them are specifically relevant when discussing IaaS—rehosting and rebuilding.

Rehosting (also known as the “lift-and-shift” approach) involves taking an application and re-deploying it to a different hardware environment without making changes to the application’s architecture. Although this is a quick-fix solution that will have your application up and running on the cloud quickly, rehosted applications probably will not be able to utilize the main advantages of the cloud (such as horizontal scalability) because they were built on top of custom vertically integrated hardware. Rehosting allows organizations to get away from high on-premises infrastructure costs, but the approach is suited mainly to applications with easily defined patterns.

Resource-intensive apps (such as those used for image rendering or big data analysis) are more suited to the rebuild (or “re-architect”) approach, in which the original code is scrapped and replaced with new code. These types of applications naturally rely on commodity hardware and have the ability to scale, so keeping them in closed, on-premise silos could mean you end up with performance issues and high costs.

Rebuilding is naturally a time-consuming and costly process. The solution, therefore, lies somewhere between the two approaches, with the main deciding factors being cost and security of the workloads.

Get it Right from the Start

At the beginning of the process, be careful in selecting which apps or services you move to the cloud. If the first few are successful, that will set the tone for the whole migration. If some of the early projects fail, enthusiasm for the migration will dwindle and make it more difficult to complete.

Be smart by choosing projects that are most likely to be successful, such as non-critical applications or services that can be migrated easily. Make sure your environment is transparent from the start by using modern monitoring and logging systems that can collect large amounts of log data and then parse it to find events and provide insightful information.

In addition, knowing your cloud environment will significantly increase your success rate. Don’t get too ambitious and try to do it all at once. Follow each small success with another, and your team will ensure a smooth migration.

Beware of Staff Apprehension

Any dramatic change within an IT environment will face staff apprehension. You might expect the shift to be met with resistance from corporate management, but the real doubts will come from your IT team. After working with on-premise infrastructure for years, administrators will not welcome any changes they think could jeopardize their jobs. Help your team embrace this shift by making sure they receive suitable training to prepare them for a new set of challenges.

Log Analysis and Metric Collection

Organizations moving to AWS suddenly find themselves managing a highly scalable and highly dynamic environment that requires a new type of log analytics and metric collection. The need for centralization of data is crucial in dynamic environments because often you find yourself trying to troubleshoot a problem on servers that no longer exist.

During the most recent re:Invent, AWS CTO Werner Vogels emphasized the need for centralized log collection as necessary for success in AWS migration:

IT organizations suddenly gain access to thousands of servers running more applications than ever that also wind up generating more log files than most IT organizations can possibly cope with. … In order to be successful in the cloud, IT organizations need to relentlessly measure how both systems and applications are performing. The problem is that log collection and processing is painful.

All of the individual components Vogels described each have their own outputs of important log data that need to be analyzed. To ensure a successful AWS migration, logs from your applications, infrastructure and S3 Access buckets—and much more—all must be monitored to ensure all of the parts of your new AWS environment are operating in a way that is secure, highly available and scalable. (To help IT organizations, we have published a free e-book on AWS log analytics with the open source ELK Stack.)

AWS: A Final Note

AWS is the preferable public (IaaS) cloud choice for enterprises today, and it looks set to stay this way. To achieve a successful transition during a migration of workloads and products to AWS, the process needs to be carefully planned and implemented in a step-by-step fashion that will prove the benefits of the move to all of the stakeholders.

March 15, 2016
Agile Infrastructure: Because your Infrastructure Deserves Tests, Too
Editor’s Note: This post was updated Aug. 4, 2018, to reflect an updated slide presentation at the end of the article.

Infrastructure as Code is a truly powerful concept whose significance is best described with the words of Adam Jacob, CTO at Chef, in Web Operations: Keeping the Data On Time: “Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources.” And so it’s not for nothing that treating infrastructure as code is regarded a key principle for reducing cycle time and maximizing flow in Continuous Delivery and DevOps, respectively. Even if your plans are less bold as you read this article, wouldn’t you agree that consequent validation of what you automate on the infrastructure level — long before it goes into production — would be extremely powerful?

So, bear with me for a while and please don’t rush head over heels into automation by simply picking one of the established infrastructure automation tools such as Chef or Puppet, getting everyone trained in Ruby, and blindly start to “automate all the things”. If that’s how you do it, then you are most probably doing it wrong. In You’re Doing DevOps Wrong. Automation in the Enterprise., author Alan Sharp-Paul suggests that, before doing any automation, you first have to understand precisely the requirements and how to validate them to gauge success and maintain quality of your automations over time:
- Are the configuration files in place? Do they contain the right settings?
- Is this port open? Is that port closed?
Why Agile Infrastructure?

Honestly, you and I wouldn’t expect anyone to come up with reliable software merely by choosing an arbitrary programming language and a nifty IDE without having some sort of engineering process in place, right? Likewise, the (automated) provisioning of a solid production environment demands a no less methodical approach. Luckily, Infrastructure as Code enables us to treat the provisioning of infrastructure as well as automating deployments and configuration as an Agile engineering discipline. Although the concepts of Agile Infrastructure are not entirely new, testing against live infrastructure has only recently been facilitated with the consolidation of tools such as Ansible, Chef, Puppet, Docker, Vagrant and Serverspec under the hood of Test Kitchen (see more on that below). Here are some of the key implications of applying Agile software development practices to the infrastructure level:

Alignment of Devs and Ops

Agile software development teams are driven by customer demand and aim to produce outcome in short, regular intervals. Enabling IT Operations and Infrastructure departments to work in small iterations and react to changes quickly allows them to align with and effectively become driven by software development. The outcome of these joint efforts will be both working and deployable software at the end of each iteration.

Fail Fast: Feedback Loops

Executing infrastructure tests as part of your Continuous Delivery build pipeline (before an application is deployed and long-running tests are executed) establishes another important feedback loop that enables you to “fail fast” and get the right people involved. The underlying principle is simple: by expressing your intentions twice — once in the code and once in a test — you’ll know you’ve caught a bug whenever these assumptions don’t match.

Regression Safety

Believe it or not: whether you are using Ansible, Chef, Puppet or any other — you’re running on code! Their respective issue lists are full of unresolved issues of various severities with new issues coming in regularly. Having a sufficient safety net of tests for your infrastructure in place allows you to quickly identify bugs as they creep in (from either side) and protects you against unwanted regressions. After all, a bug in your infrastructure can be very hard to track down and can be much more severe than a bug in your application.

Refactoring

Code refactoring becomes a breeze when you have tests in place that immediately alert you when you broke the desired state of your infrastructure (and all you wanted was to make your code simpler and more maintainable). This is invaluable when you want to recompose your existing infrastructure from 3rd-party components offered on public hubs such as Ansible Galaxy, Chef Supermarket or Puppet Forge, and especially when you are migrating away from your existing automation stack.

The Red, Green, Refactor Cycle of Test-Driven Development

Test-Driven Development (TDD) is a popular software development process that involves writing code in short, repeating development cycles with the goal to enforce good design and enhance confidence. While it is certainly no silver bullet and has been controversially discussed in the past, the Red, Green, Refactor Cycle, which is at the heart of TDD and whose principles I want to quickly summarize here, hopefully gives you some inspiration on how to incrementally build up your infrastructure:
1. Red: Understand the requirements and the state you want to implement. Then think about how to capture this state in a test. Implement the test, then run your entire test suite and see the added test fail — and if it doesn’t, something is really wrong.
2. Green: Implement just enough functionality to make the test pass. Don’t care much about readability, simplicity and design for now — just make it work. Run your tests again and watch all tests pass. If they don’t, either fix your test or your implementation, or take a step back (revert to a point in your code that’s known to work) and start over with a smaller increment.
3. Refactor: Now that your tests pass, improve your implementation and make sure, by running your tests again, that you didn’t break anything. Once you are done refactoring, start the cycle over again.
Keep in mind that each repetition of the cycle refers to a small increment. In terms of infrastructure, such an increment could be the installation of a user, a package or the placement of a configuration setting. When you arrive in a spot where you broke something, you can easily revert to a point in your code that’s known to work. Leveraging such practices really makes the difference between Infrastructure Automation and Infrastructure as Code, as described in Infrastructure as Code – Automation Is Not Enough by Keith Morris.

Test Kitchen, Terraform, AWS and awspec

Now, how to get there? Test Kitchen is an easily extensible test harness that allows you to test your code written in Ansible, Chef, Puppet (and others) on various cloud providers, virtualization providers and operating systems with tests written in a variety of test frameworks.

In the following presentation, I give hands-on examples for you to quickly dive into the topic of using Test Kitchen with awspec as a framework to test Terraform deployments on AWS:
November 13, 2015
Avoiding Common AWS Security Risks

According to IBM’s Cyber Security Intelligence Index, 95% of all security incidents involve human error. That’s beyond significant, and when rapidly deploying multiple iterations in a DevOps shop, that means there are lots of opportunities to fail.

Our own independent analysis based on customer configuration data shows that security control mechanisms Amazon Web Services (AWS) makes available to its customers are not always being used consistently due to human error, creating unnecessary security exposures while operating workloads in cloud.

In this article, we’ve highlighted the five of the most common AWS security risks detected in the first half of 2015, and share simple steps you can follow to address them.

EBS volumes are missing optional EBS volume encryption…

EBS volume encryption uses AES-256 and is a convenient mechanism for meeting data-at-rest encryption compliance mandates. An unencrypted EBS volume cannot be converted to an encrypted volume after creation. It is a setting that must be applied on creation of an EBS volume.

EBS volumes contain no snapshots less than two weeks old…

Snapshots are a low cost way to recover EBS volumes and they can be made while a system is online. We have seen a number of occasions where clients have benefited from snapshots in IR scenarios by enabling them to forensically analyze systems that had been compromised from recent snapshots in virtual private cloud sandboxes.

Unused EC2 Security Groups…

Keeping your EC2 security groups clean eliminates the risk that an unauthorized security group policy will be used by mistake to open attack surface. Often clients have had relatively new AWS users mistakenly launch EC2 instances using insecure security groups that lead to incidents. Practice good security hygiene and remove your unused EC2 security groups.

Unused or unmaintained ELB Security Groups…

Keeping your ELB security groups clean eliminates the risk that an unauthorized security group policy will be used by mistake to open attack surface. In large multi-tier architectures, ELBs frontend fleets of backend systems, and ELBs have security groups that should use authorized security group policy. Practice good security hygiene and remove unused ELB security groups.

Global permissions to access TCP or UDP ports in a security group that is attached to an active instance/ELB…

Restrict access to known static IP addresses or CIDR ranges within your control. Many clients have experienced scenarios where they found themselves conducting incident response efforts due to unauthorized EC2 instances that were launched using insecure globally accessible security group policies. Monitoring active security groups enables clients to shorten time to detection and time to remediation of firewall related vulnerabilities and enables clients to minimize attack surface.

About the Author/Justin Lundy

Justin Lundy, CIO & CTO, co-founded Evident.io after leading Adobe’s Cloud Security efforts to identify, implement, and manage security controls and strategies during rapid adoption of the cloud at scale. He has spent more than a decade in the infosec industry, and is a subject-matter expert in multiple domains. Justin previously defended the castle at Adobe, MTV/Viacom, Sun, and CA.

October 22, 2015
AWS re:Invent – New Releases Drive Cloud Security Innovation

The recent AWS re:Invent conference in Las Vegas saw the release of several new AWS services for security. Some of these announcements such as AWS Web Application Firewall and AWS Inspector seemed to take many in the security press and blogosphere by surprise.

Cloud-native security companies like Evident.io however, weren’t surprised by these announcements. In fact, AWS has been actively partnering with their security ecosystem to create opportunities for partners to innovate on top of these new base services.

Both incumbent and emerging partners need to remember that a platform is not a product, it is a constantly growing and evolving entity that will continue to adapt as the cloud ecosystem continues to change.

If you want eternal stability, consider investing in a pet rock. If you want to be part of continuously improved technology ecosystems, then strap in because AWS is the rocket ship of platforms in this space.

AWS, no matter what they innovate, will encroach on similar value positions of its ecosystem as a matter of existence. However, this is the best situation possible for customers of the platform.

Customers win out because they receive constant technology innovation and improvement as part of their platform experience, which is something that was never present in the datacenter ecosystem.

Let me paint some color to what these services mean for the ecosystem:

AWS Web Application Firewall (WAF)

AWS WAF enables customers to create filtering rules that whitelist or blacklist application calls coming inbound from the internet towards a customer’s deployed CloudFront CDN resources.

For example, a customer could write a WAF Rule that filters requests to their web application based on source IP address matching or pattern-matches against the request string.

If I was running an e-commerce store that only sold products to the US, I would ensure that my WAF rules contained blocking request patterns for IP Address ranges from China, Syria, Sudan, North Korea, and other embargoed countries (and huge sources of malicious traffic).

While this appears to be a direct attack on some cloud security providers, it really creates a new market path and opportunity for them to deliver their WAF intelligence to the customer.

Let’s be honest — there’s nothing differentiating about building a big x86 appliance and having it run the rules intelligence for your infrastructure.

AWS has removed that barrier of virtualizing the appliance and created a direct path for these companies to deploy their core value proposition — curated and powerful rulesets that protect customers from historical, existing, and future attacks.

Expect to see branded rulesets available through some kind of marketplace or other delivery mechanism here, as that is what customers need.

AWS Inspector

Inspector, in our eyes, was the most important announcement in recent AWS history. We’ve finally seen the true convergence of cloud-based security begin to emerge: Continuous API-centric security intelligence accessibility that is pervasively built into the platform’s diverse feature-set.

With the advent of Inspector, security providers can directly interface to the hosts and applications to provide deep, thorough security visibility for customers without deploying additional third-party agents.

Let’s face it, customers were already overburdened with agents before they ever got to their security stack, and the value delivered by security agents has been questionable at best in cloud environments.

Expect security platforms to integrate singular-views of workload-wide security intelligence in order to provide customers a complete picture of their security posture, risk profile, and attack surface.

Config Rules

Config Rules was, perhaps, the least exciting of AWS announcements in the security space. Unfortunately for Config-based service interactions, the limitations of the service are still apparent a year after initial launch.

Config Rules offers no exception, where interesting capabilities for customers to define custom rules that help them understand what changes impact security in their environment are hampered by limited service support, a cost model that is too unpredictable for small customers and of minimal security value beyond the “ping” that happens when rules fire.

These challenges are compounded by the barrier to entry of writing well-structured and contemplated security rules by a broad audience of non-security experts, which delivers an almost false sense of protection.

Contrasted with CloudTrail, which gives valuable audit information that makes sense at an operational level, Config Rules anticipates the customer will have strong security experience and operational knowledge to leverage the service.

But some customers in every user base are satisfied with what we call “good enough” security in the industry, which defines a scope of offerings that barely make the cut to be called security technology.

The platforms and vendors slinging “good enough” security tend to be at the international buffet, where you get a below-average pizza, a mediocre taco salad, and a poor hamburger.

Nothing you get leaves you with a feeling of being satisfied, because not enough effort was put into the final product to actually change the way you think about the experience.

But mature, focused products will always deliver a superior customer experience, and in security that should never be considered optional.

A vendor who divides their attention between cost management, security, optimization, etc. will never provide the same quality or attention to protecting your infrastructure as a focused innovator in the space can offer.

Users can expect a waiting period for service support for new AWS services, meaning that your security controls are dependent on Config’s roadmap.

But security savvy customers, are not keen on waiting for security to grow up around them – they need it now, they need it on 100% of their cloud infrastructure, and they need it to be actionable.

This is why many will continue to choose the leaders in each security layer over the basic security features tied into the platform itself.

Expect Config Rules to grow over time – but if Config is any indicator, it will be a long time before Config Rules provides real value for customers on the platform.

The real value here is that innovators in the security space will continue to fill the gaps left by services in the platform and innovate customer value atop the technology that does make the cut.

Wrap-Up

Well, that was a journey – and we only covered three services! AWS re:Invent 2015 was a whirlwind of sessions, 19k participants, and awesome discussions on every topic under the sun.

It’s hard to believe we have to wait another year to see this massive gathering happen again, but that leaves a lot of time for innovation, new customer success stories, and to sort through the giant piles of swag that everyone picked up in that madhouse of an expo hall.

Author: Tim Prendergast co-founded Evident.io to help others avoid the pain he endured when helping Adobe adopt the cloud at a massive level. After years of building, operating, and securing services in AWS, he set out to make security approachable and repeatable for companies of all sizes. Tim led technology teams at Adobe, Ingenuity, Ticketmaster, and McAfee.

October 16, 2015
Alert Logic Cloud Insight brings cloud-based security to your cloud infrastructure

One of the primary driving forces behind DevOps is its fluidity. There is a domino-effect that cascades from the developers who create the apps to the IT admins who deploy and administer the apps to the individuals who download and use the apps and back again. Traditional security can’t keep pace so Alert Logic created Cloud Insight—a cloud-native vulnerability and configuration management solution for cloud infrastructures running on Amazon Web Services (AWS).

I’ve written previously about the continuous continuousness of DevOps. Anything and everything that used to be manual tasks on a static infrastructure is now automated to the Nth degree. “Continuous development and continuous testing lead to continuous deployment and continuous delivery, which requires continuous support. Continuous monitoring produces continuous integration and continuous change.”

All of that also means that it’s virtually impossible to keep up with the current configuration or accurately assess vulnerabilities in a constantly-changing environment. If your entire cloud infrastructure can change at the push of a button you also need to have security controls in place that are just as automated and capable of flowing with the changes and providing continuous, real-time security. Traditional security solutions simply won’t work.

Alert Logic agrees, which is why it developed Cloud Insight. Gray Hall, CEO of Alert Logic, explained in a press release announcing Cloud Insight, “Managing security while maintaining the agility of the cloud is fundamentally different than in any other environment. With Alert Logic Cloud Insight we are delivering an innovative solution that gives businesses an added layer of visibility for workloads running on AWS—actionable intelligence to identify and eliminate risk across instances and applications while not slowing the business down.”

Cloud Insight automatically tracks changes—additions, moves, deletions—within the AWS platform. It continuously identifies threats and vulnerabilities and potential exposure to risk within the dynamic environment. Alert Logic uses AWS APIs to integrate with the cloud infrastructure to detect any changes and inspect the new, change, or modified resources for vulnerabilities or configuration issues.

Cloud Insight generates a prioritized list of remediation actions so the organization knows in real-time what’s broken and what the exposure to risk is and it can focus on addressing the issues and maintaining a strong security posture rather than spinning its wheels just trying to keep up with vulnerability assessments in a fluid environment.

Terry Wise, VP of Worldwide Partner Ecosystem for AWS, said “Security-as-a-Service solutions built on AWS and designed to complement AWS-native security features enable customers to maintain the highest levels of protection while achieving the agility, elasticity, and cost savings of the AWS Cloud.”

Alert Logic Cloud Insight is pre-approved to run on AWS. You can use Cloud Insight to conduct vulnerability scans and penetration testing within AWS without any additional permission. Cloud Insight also includes customer-facing APIs that can be used to integrate with other DevOps tools, and it can be integrated into your existing DevOps pipeline.

Pricing for Alert Logic Cloud Insight is similar to the AWS pricing model. It functions on a pay-as-you-go basis based on the number of hosts or protected instances for a given day. Customers are billed $1 (USD) per instance per day. You can request a free trial of Alert Logic Cloud Insight to determine if it’s a good solution for your organization before taking the plunge.

August 13, 2015
Librato Seeks to Unlock DevOps Application Monitoring

As wider adoption of DevOps methodologies continues to foster demand for new tooling to help streamline evolving workflows, at least one vendor is attempting to address the growing need for related monitoring capabilities.

Librato, a San Francisco-based provider of Amazon AWS applications performance monitoring (APM) services, claims that it is finding favor among the rapidly expanding DevOps community driven by its ability to inject critical visibility into the engineering lifecycle.

Acquired by Austin, Texas-based SolarWinds in January for a lofty $40 million and renamed Librato SolarWinds Cloud, the company is touting its recent adoption by well-known providers such as Slack for monitoring of DevOps-centric applications performance and health metrics.

“At its core, DevOps is destroying conventional wisdom around ritualized processes and the need to perform them, mostly in the domain of scale,” said Dave Josephsen, Developer Evangelist at Librato. “By attempting to include monitoring as a key element of the development lifecycle, this has created a whole new range of challenges, and opportunities.”

By providing a centralized Web interface and open API to help create visibility into critical applications engineering requirements, Librato maintains that it is creating the tooling necessary to empower integrated development and deployment processes, such as continuous change control.

“There are a lot of disparate tools for monitoring individual engineering tasks, but there hasn’t been any practical way to bubble up that data for efficient monitoring,” Josephsen said. “Within the context of DevOps, change control is a ritual that we’ve abandoned as we’ve embraced test driven development and automated away the human element, so there’s a distinct need for monitoring of feedback and metrics.”

For example, cloud-based productivity messaging specialists Slack are using the Librato “StatsD” lightweight daemon process to stream business and app-health metrics regarding PHP applications into the monitoring platform. Slack also leverages Librato to chart data regarding connect, query and fetch latencies for its server clusters, and to monitor internal metrics about overall system health.

“Having those graphs side-by-side lets us clearly see and compare the effects of network latency on one of our new services, and that’s very cool,” Richard Crowley, director of operations at Slack, wrote in a recent testimonial posted to the Librato Solarwinds blog.

Industry experts agree that broader adoption of DevOps strategies will continue to drive demand for relevant monitoring tools, with solutions such as Librato seeking to deliver “modern stack architecture for looking at the modern stack.”

“I think that APM is always interesting because it’s always new,” said James Governor, industry analyst with RedMonk. “The pace with which we’re building new architectures to deliver services to end users is increasing all the time, and with every new wave of technology, we tend to get new and associated models of APM tooling.”

Librato is taking advantage of the need for a system that brings together the elements necessary for cloud native monitoring, and in an agentless manner different from other competing tools such as New Relic, the analyst noted.

However, whether or not the platform is able to break down traditional engineering siloes to enable more efficient DevOps management, as Librato officials claim, remains to be seen, he said.

“Each wave of technology has promised to do this, to remove the inherent finger pointing, and each has always made the same claim to break down organizational firewalls,” Governor said. “DevOps hasn’t boiled the ocean in this manner yet, but there are some unique opportunities; unlike previous waves, where automation was intended to replace humans, this is now about augmenting people, and we are seeing that teams are choosing new tools from which developers can truly derive value.”

August 10, 2015
DevOps fast food: Chef on the AWS marketplace,

News from the AWS Summit in NYC today is that Chef is now available as a one click install on AWS via the AWS marketplace. While the ability to use Chef on AWS is not new (it is the leading configuration management tool on AWS), being on the marketplace makes both install and billing a one click deal. This should make it really simple to get Chef installed and running for anyone looking to leverage the many benefits Chef brings to configuration management and automation. Literally with one click install what could be simpler?

As part of this new arrangement not only will Chef on the AWS marketplace be available, but it will also be a featured offering. Chef and AWS already enjoyed a cozy relationship (they are both based in Seattle and there is a cross-pollination of personnel). This deal takes the relationship to a higher level. Both Chef and AWS are dedicating significant resources to marketing, sales and support of this initiative.

I know from speaking to AWS in the past and other marketplace vendors that AWS will often let partners list on the marketplace, but they don’t often then dedicate resources to roll out, marketing and support. Further, as far as I know Chef is the only configuration management/automation solution that is available via the marketplace like this. I don’t know if this will remain the case or for how long, but for now it is somewhat of an exclusive relationship.

I had a chance to catch up with Ken Cheney, VP of Business Development and Jay Wampold, VP of Marketing at Chef about this expanded relationship the other day. They both emphasized that this is much more than just listing Chef in the AWS marketplace. Some estimates I have seen have Chef deployed on something like 70% of AWS deployments. Ken and Jay believe that this number will continue to remain high and may even go higher as a result of the marketplace presence and the push from both Chef and AWS.

On top of their previously released expanded relationship with Microsoft via Azure, it seems Chef is putting a stranglehold on two of the largest cloud platform providers in the world. That doesn’t mean that other configuration managers don’t have presence on AWS or Azure, but Chef seems poised to be in a dominant position on these platforms.

As we see more enterprises continue to migrate more and more infrastructure to the cloud this presence will become more and more important. For now you can get Chef fast on the AWS marketplace.

July 9, 2015
The Last One That Transformed the Future
The headline made her yelp for joy. A year into the future of D-Chip, on a crisp August morning, Phoebe opened the newspaper and swiped to the technology section on her iPad. What she saw was riveting: D-Chip reborn as a star in wearable computing.

She immediately thought of the article Marc Andreessen wrote on how software transformed the enterprise. It’s true of our company’s story, she mused. The credit for making the headline a reality was due to the stressful events of the last months. Who said stress couldn’t be a game changer? As she picked absently on a plate of fruit and relished the article for the second time, she recalled the events of the past year.

Flashback a year ago at D-Chip

Remember how Rachel pitched her enthusiastic idea to Chandler? She wanted to engage six engineers in six months to build an in-house solution. A solution that would integrate D-Chip and Enogo’s systems by letting them deploy in both OpenStack and AWS and manage wearable chip code from a single deployment platform.

Chandler agreed in theory with the need for a single management platform but didn’t see the sense of doing it in-house. Why side-track from your business goals and spend months building a solution yourself when there are specialists solving this exact problem for many companies facing the same issues? After much debate, they came to a consensus. They decided to evaluate solutions from the best-of-breed products out there and transform their software development lifecycle to reflect the best agile practices. Thus was born the iTransform project.

The iTransform project

The name itself was a reminder that transforming was as much a matter of people internalizing and accepting a new way of doing things as much as it was about aligning and unifying D-Chip and Enogo systems. The first order of priority was to understand the pain-points that assailed the company. After a lot of meetings and detailed analysis, they identified key use cases from internal stakeholders, namely the engineering and IT operation teams:
- Engineering personas, developers, QA, architects, all said the same things. They didn’t have access to the latest tech because it took weeks or months for approval processes to travel up the chain. Then they waited on builds and environments from IT operation teams. Most of the time, the environments in the private datacenter OpenStack cloud didn’t work properly, and deployments never succeeded the first time. It’s no wonder they took weeks or months to release anything decent.
  
  Self-service and continuous delivery were, therefore, the most important criteria for developers, QA, and architects. Without waiting for IT admins or operations teams, they wanted to spin up build environments in test, staging, even production on their choice of pre-configured infrastructure. They wanted to consume in an on-demand fashion the latest technologies on multiple clouds. Today they were using AWS cloud and OpenStack, but tomorrow they wanted any cloud that matched their workload configuration needs best.
- IT operations and system admins brought up their set of challenges. Their primary goal was to configure environments for repeatability; safeguard builds from failures and structure stable workflows. Not an easy job come to think of it. On top of this demands from the engineering teams dictated that they learn and implement new technologies in a very short time. Technologies ranged from configuration management tools, cloud platforms, storage, networking, load balancing services, monitoring services, and more. Besides when engineers wrote configuration templates as Enogo did for AWS, it was hard to figure out who owned the deployment stack. In the case of problems, it resulted in pointless blame games. So they desired a clear level of separation between code, deployment automation, and infrastructure. Technologies and cloud platforms could come and go, but IT operations remained responsible to maintain smooth workflows, stable builds, and error-free deployments.
  
  A service catalog also met the needs of the IT operations teams. They wanted to build and manage specific platforms, runtimes and version configuration changes. Collaboration was a key factor. Many IT operations team members and engineers were experts in some but not all technologies and tools. But together in a collaborative platform, experts could contribute at their skill level and share knowledge with others. Automating configuration changes in bulk saved a ton of time. Therefore, API support for automation was high on their wish-list.
- The IT managers were important though indirect stakeholders. They often approved and handled audits, compliance, and budgetary aspects. Policies and cost management were paramount to them. IT managers needed control and oversight to implement compliant deployment policies that gave the right level of access to the right set of infrastructure on pre-approved cloud providers. They needed to set and monitor quotas and track IT spend on cloud deployments against approved budget.
Transformed enterprise

After three months of evaluating proof-of-concept solutions with vendors, they zeroed in on a solution that met all of their uses cases and helped set up their current builds. Because of this, the teams were able to ship code ten times faster. It was only a matter of time before D-Chip stood out as a market leader. It was clear to the company executives what was driving this transformation.

They retired the semiconductor business and switched their business focus entirely to SDK builds of software for wearable computing. The software services for wearable chips was now way more valuable than any piece of hardware. Their customers in wearable computing now consumed D-Chip technology as a service in the cloud. A faster and efficient software development lifecycle enabled them to carve a new market niche and become the de facto leader.

The transforming story of D-Chip is a reminder for us all. When there are problems in software development, by aligning goals and helping people in different roles work together and participate in the development lifecycle, you increase the pace of innovation dramatically. It can be so powerful that it can transform the company’s core products or services.

Disruptive change doesn’t happen one step at a time. It happens with mavericks and innovators like Rachel and Chandler leading radical changes in a focused group effort. You want innovation to transform your business? Get a dedicated group of mavericks together and make it happen.

Just like mankind advanced their understanding through each cycle of civilization from the stone age up to the Renaissance and the Industrial Revolution, the future is heralded by the IT age of our times.
April 6, 2015
The One Where All the Deployments Stall

In the first episode of the DevOps story, DChip CEO Phoebe gave Chandler an ultimatum to move out of AWS into their datacenter on OpenStack. Does Chandler succeed? Read on to find out…

Fast forward two weeks since that fateful night. Chandler is preparing for yet another leadership call. It’s deja vu, with the same story all over again. Only this time he knew the problems all too well. So he sat bleary-eyed and prepared what to say.

Time to face the music

The meeting involved the usual suspects. Phoebe, Monica dialed in from the US along with the sales and marketing VPs and Janice the VP of Finance.

Strangely, or maybe it was the late hour of the night, Chandler felt calm as he spoke:

“I want to bring everyone up to speed on the progress to shut down AWS and move to OpenStack in our datacenter. We have bad news. Enogo engineers were not able to migrate workloads to OpenStack smoothly. The problem is such that they have not managed to release a single update, and the release schedule has started to slip.”

Both Rachel and Joey eyed him sharply. He wasn’t going to back down from the truth now, so he ventured: “Engineering sprints are delayed because of this. The real problem is the OpenStack environment. It’s not mature. Most IT operations engineers are still learning to configure it, and there are many environment issues. I understand, everyone is going to hit panic mode. Which is why, I say, we keep AWS Cloud to return to business as usual.”

Soon as Chandler finished, the finger-pointing began.

“Enogo engineers used CloudFormation templates in AWS for dev/test/staging deployments. Those templates, however, don’t work in OpenStack because Heat is not set up properly,” Rachel said. Adding, “We sent the templates to the Ops team, but they never got back to us.”

“Well, when the templates are not written correctly by the developers,” countered Monica, who headed the OpenStack implementation, “we have to rewrite them from scratch in Heat. That’s not going to happen overnight, is it?”

All the while, Phoebe, who’d hardly said a word knew that Chandler was right. The 3-month transition plan to OpenStack was too aggressive and just not practical. If more and more developers choose AWS cloud, why not join forces if we are to help them succeed?

As a leader, you have to know when you’re wrong and correct course. So she did. “Chandler is right. We should go back to AWS to avoid schedule slippages and get the release back on track. I will discuss a longer term plan with Chandler.”

We need a solution

After everyone except Chandler had dropped off the video call, Phoebe said: “Look, we f*****d up. We can’t go through another f***k up and face this kind of deployment shutdown. We need a solution and fast. What do you say Chandler?

“I can’t agree more. To minimize business risks and integrate both company systems, we need a hybrid solution that supports deploying to both AWS and OpenStack clouds. It’s not enough to simply solve for the template problem of today. It should scale for future projects. We need a solution that offers a single point of management for both clouds. The Enogo team should be able to deploy with the same speed and ease of AWS in OpenStack and not rely on IT Ops to hand-hold them.

“Bingo,” said Phoebe at last smiling.

The next morning as Chandler stirred his coffee trying to awaken his still asleep brain, he read this email from Rachel:
“Hey Chandler, a couple of engineers have already written a bunch of scripts to automate deployments in AWS. I say, we devote a few more engineers and build an org-wide solution that will enable us to build and deploy in both clouds. What do you think?”

How timely, thought Chandler. So Rachel, the Enogo engineering manager wants to build the solution in-house with her team of engineers. She wants five engineers and six months to do it. This project will no doubt widen her impact and visibility in the company. Good for her!

Chandler sent a brief, “Why don’t we discuss in person tomorrow,” reply. In walked Rachel the following afternoon. The minute she saw Chandler’s face, her enthusiasm took a dive. He doesn’t look amused, she thought. His steely gaze sent a chill up her spine.

Wait to see what happens in the next installment of the DevOps story coming soon from ElasticBox.

March 14, 2015