devopsy.kinsta.cloud

Tag: cloud

  • IBM Survey: Rise of Multi-Cloud Computing to Force DevOps Issue

    IBM Survey: Rise of Multi-Cloud Computing to Force DevOps Issue

    A survey of 1,106 business and technology executives published this week by the IBM Institute for Business Value finds 85 percent of companies are already operating in multi-cloud environments, with 98 percent forecasting they will be using multiple hybrid clouds within three years. But the survey finds that only 39 percent of the respondents have implemented some level of DevOps processes to help streamline the management of multiple cloud computing platforms.

    Blaine Dolph, vice president of Global Business Services for IBM, said it’s clear that DevOps will play a critical role in enabling IT organizations to centralize the management of multiple clouds. But the IBM survey suggests that the management of multiple clouds within organizations is not nearly as efficient as it should be.

    Only 41 percent of survey respondents said they currently have a multi-cloud management strategy in place, and just 38 percent have the procedures and tools in place to operate a multi-cloud environment. Only 30 percent of enterprises have a multi-cloud orchestrator or other multi-cloud management platform to choreograph workloads, while fewer than 40 percent of organizations have cloud configuration management tools that provide information about resource configuration and relationships between resources, the survey found.

    All told, the IBM survey is only able to identify 20 percent of respondents as “multi-cloud maestros” capable of employing multiple clouds to smoothly and routinely deliver one or more business functions. The survey found almost two-thirds of multi- cloud maestros report they employ cloud management platforms.

    The survey also found that multi-cloud maestros are advanced in terms of adoption of microservices and containers. A full 61 percent of multi-cloud maestros predicted that within three years at least 80 percent of new applications will be developed using containers. Currently, 50 percent of multi-cloud maestros are using Docker containers and 63 percent are using containers as a service (CaaS). By 2021, 65 percent expect to use container orchestration. More than half (56 percent) of the multi-cloud maestros are already microservices and 59 percent plan to use serverless cloud services in their cloud environments, according to the survey results.

    Dolph said IT organizations in general would be well-advised to focus on more than just “lifting and shifting” applications into the cloud. They may need to leverage Docker containers to move an application into the cloud. But once there, an effort should be made to modernize that application by turning large swaths of it into a set of microservices that ultimately are easier to maintain and consume by developers building complementary applications, he said.

    The IBM survey makes it apparent that multiple clouds are the new IT normal. Almost half of respondents said they are establishing a multi-cloud architecture to develop new and enhanced products and services. A total of 62 percent of respondents require multiple clouds to create innovative business models, while 52 percent said they need them to produce new revenue streams. Sixty-six percent need them to enhance margin and 46 percent said they need a multi-cloud environment to support agile business processes.

    From a DevOps perspective, more than half (51 percent) of respondents use multiple clouds to cultivate a flexible, modular infrastructure that quickly absorbs and leverages technological advances.

    Obviously, one of the great paradoxes of multi-cloud computing is that in the pursuit of becoming more agile, many organizations are making their IT environments more complex by employing multiple platforms. The challenge they will face next is maturing their DevOps processes to the point where some order is finally brought to that multi-cloud computing chaos.

    — Mike Vizard

  • DevOps Chat: disrupt:Ops Brings Security Closer to DevOps

    DevOps Chat: disrupt:Ops Brings Security Closer to DevOps

    Shifting security left during the software development process can be tough for companies because of the sheer number of operational things that go along with implementing security that must occur somehow in the realm of DevOps. It can be easy to lose control over their security processes without some sort of guardrails in place to help.

    A brand new company is helping do just that. I’m excited to introduce disrupt:Ops, which bills itself as a guardrails platform to help companies include and enforce best practices across security and operations, bringing security into the DevOps process at the beginning. My friend Mike Rothman, disrupt:Ops co-founder and president, joins me in this DevOps Chat to discuss the vision of his company and how it hopes to help make security truly a part of DevOps.

    You can read more about disrupt:Ops and their guardrail approach to cloud security and devsecops from another co-founder of the company, my friend Rich Mogull at https://devopsy.kinsta.cloud/building-great-cloud-security-guardrails

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Audio

    Transcript

    Alan Shimel: Hey everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Breaking some big news today on DevOps Chat, the launch of a labor of love of some friends of mine. I’m happy to be joined by Mike Rothman, cofounder and president of disrupt:Ops. Mike, welcome.

    Mike Rothman: Alan, it’s great to be here, thanks. It’s always fun to catch up with old friends.

    Shimel: Yep, and old friends it is, Mike. We should let our audience know – maybe we shouldn’t let them know. But we are old friends. We go back a long time now, back to the beginning of blogging and security –

    Rothman: That’s right.

    Shimel: It’s got to be 2004, 2003 or something like that.

    Rothman: Yeah, that’s right, that’s right. When I had dark hair. Actually I didn’t have dark hair back then, but you know, it’s –

    Shimel: I had hair.

    Rothman: You had hair. That’s right.

    Shimel: I had hair. But you know what, Mike? I said it was a labor of love and everything else. Just in the way of background, why don’t you share with our audience your personal sort of journey in tech and, you know, work history.

    Rothman: Well sure. I started out, God, in what feels like 100 years ago, as a mainframe programmer. Ended up doing some local area networking stuff. Kind of found my way to network security. Went into research for quite a while. Did a couple of companies and then went back to research and for the last 10 or 11 years at this point I’ve been partnered up with Rich Mogull and Adrian Lane at Securosis, doing a bunch of kind of information, security-centric research. And actually about a year and a half ago we spun out some technology that we had prototyped within Securosis in cloud management and cloud automation into our new company disrupt:Ops, and have been just making a lot of progress over the last year with that. And that’s what we’re announcing this week.

    Shimel: Yep. So Mike, you’re being humble. When you say you did research, you were an analyst at Giga, were you not?

    Rothman: META Group, yes. META Group, exactly.

    Shimel: META Group. And then you were also – you had executive positions in several cyber – we didn’t call it cyber – infosec companies back then. 

    Rothman: It was. Yeah, you bet. So companies like TrueSecure and CypherTrust, you know, back early in the e-mail security days. I did a network monitoring company called EIQ. So yeah, I’ve been there, done that. Have the road rash and kind of the screw-ups under my belt to prove it. But you know, it’s interesting in terms of just how long and how small an industry security is for being actually a really big thing nowadays. 

    Shimel: Being so big, yeah. It really is. And Mike, again, for many people in our audience who come from DevOps or even from security – but they’re worldwide and they not be familiar with Securosis. You know, Rich Mogull comes over from Gartner. Adrian’s been CTO at major companies. You know, you guys are probably the leading, let’s call it independent security analyst research firm – you said 10 years-plus. Certainly in my time you guys laid the fundamental kind of – a lot of the fundamental backbone of what Cloud Security Alliance teaches and so forth. 

    Rothman: Yeah, that’s exactly right. So it was interesting. So I guess it was probably eight and a half years ago we partnered up with the Cloud Security Alliance and helped them build their curriculum for their CCSK certification. And that got us to be much deeper than your typical analyst in terms of having to play around with cloud, having to understand automation, having to build a number of the kind of scripts and – I’ll call it the inherent messiness of building out a stack in the cloud and really making it operational, because we were eating our own dog food. Our website, a lot of the stuff that we had prototyped was really demonstrative of the challenges that organizations face as they move a bunch of their stuff into the cloud, and really as they embrace a DevOps mindset of trying to be agile, trying to be innovative, trying to move towards more of a continuous type of deployment model. We really came across a number of the challenges these organizations have in terms of just really maintaining control, right? Once you have multiple DevOps teams doing things in multiple kind of accounts, you start to run into challenges relative to how you enforce best practices.

    And that’s really one of the challenges that security people have with this whole DevOps revolution, which is, “Hey, this sounds great, but we’ve got a playbook, a set of practices and policies that we really require all of our environments to adhere to in order to protect that data.” And obviously the faster you move, the more agile and continuous you try to operate, the harder it is to enforce those best practices.

    Shimel: More about that in a second. You touched on two things there that we’re going to dig into. Let me just finish laying my groundwork. Mike, besides you, Rich and Adrian, the Securosis gang, there are at least two other key people on the founding team of disrupt:Ops who also happen to be friends of ours for years –

    Rothman: Yeah, that’s right.

    Shimel: Give them their due. Why don’t you give them a shout out?

    Rothman: You bet. So back when we spun the technology out of Securosis, we brought on two guys, very experienced guys in the security space. Our CEO is Jody Brazil. So Jody was a cofounder and long-time CEO of a company called FireMon, which I know you’ve done work with for a long time –

    Shimel: Sure.

    Rothman: So Jody kind of is really the driving force behind a lot of what we’re doing at disrupt:Ops. And our CTO is a guy named Brandy Peterson, another longtime security professional. He was actually the CTO of FishNet Security for 15 or 16 years.

    Shimel: As was Jody before Brandy, right?

    Rothman: Yeah. I met both of them around the same time.

    Shimel: Yes, when I first met Jody, he was CTO and FishNet, and he said, “Who is this Jewish guy with the New York accent?” I went out to visit him in Kansas City to tell him why you shouldn’t use Qualys, and he should use StillSecure, and he made fun of my accent. [Laughs]. Good times. Anyway. But Mike, let’s turn back to a couple of things you were saying about kind of the genesis driving disrupt:Ops. You touched on one thing; you said, “multiple DevOps teams.” So this is something we see, and I call it almost second-generation DevOps. You know, when a company or an organization is first embarking on their DevOps journey and they do sort of a trail balloon, and they pick a project and they have a DevOps team – you know, and they may not call it DevOps. They  have a team that’s practicing DevOps sort of principles on this one particular project. And oftentimes, wow, great success, because you are – you know, it’s an isolated environment, right? In some ways you’re not in the quote/unquote real world.

    And then you say, “This is great. Let’s take it out. Let’s expand that beachhead. Let’s expand the bubble.” And all of a sudden – you know, larger organizations; small teams or smaller organizations not so much, but larger organizations – now you start having two, three, four, six different projects that are being run with a DevOps mindset and you have multiple instances of Jenkins and you have three different – this team’s using Puppet; that one’s using Chef. And serverless. And all of the complexities that we see in today’s modern software factory. And it’s chaos, right? And then at some point the CIO or VP or director of IT says, “Oh my God, this DevOps thing isn’t working so good. It’s all over the place.” Let alone the security guy’s screaming, you know, right out of the Phoenix Project, “No, no, no!”

    Rothman: That’s right.

    Shimel: And that’s a problem that we see with a lot of second-gen, if you will, DevOps installs, where it goes beyond the initial team to several teams.

    Rothman: That’s right.

    Shimel: That’s what you guys see.

    Rothman: We do. And it’s interesting, because what you have are really kind of two paths at that point, in terms of how these teams start to behave when starting to be put under scrutiny in terms of, kind of again, their ability to enforce operational and security best practices. You know, one group kind of buries their head in the sand and continues to move forward and really don’t do much, right? They’re like, “Oh, it’s okay,” and you know, they’ll maybe run a vulnerability scan to kind of prove that they’re sort of within policy, but they really don’t have an operational environment that is built to work in concert with the rest of their factory. I love that term, right? That really does encompass a lot of how software is being built today. And then the other part ends up being folks starting to build their own scripts, right? They take maybe an open source-looking thing, they start to build a bunch of their own scripts and code around it, and then what they find is they’ve got a suitcase full of these scripts that they end up having to maintain, upgrade. They don’t have any means of reporting or any scheduling for how they enforce and run a lot of these scripts.

    It’s really – you know, we call it automation, but it’s manual automation, which I get is a total oxymoron. But that’s really how it is. You know, you have a bunch of folks, they’re building these scripts out, they’re executing them manually or through very simple triggers, without any means of maintaining, upgrading or scheduling or reporting on what these really key aspects of their operational environment are doing. And it becomes, again, a big problem, where you have – it’s unwieldy because of just the sheer number of things that are happening in a modern DevOps environment with multiple teams. Then you compound it with a number of very tactical scripts that are built to surround the environment, if you’re doing anything. And then at some point somebody says, “Hold on a sec, folks. This is just not where we want to be. We really need to think about it more from an operational excellence perspective and make sure we’ve got an environment that we can enforce our best practices for security and operations.”

    And that’s kind of when they get the sense that, “Well, I’m either going to have to make a significant investment in this and become like” – Rich calls them the unicorns of cloud land, right? You know, the Netflix and the Capital One and, you know, the folks who have whole teams of environments that are building tools to do this. Or they have to start looking for some kind of commercial option, because there really is no option, but you’ve got to do something. You can build it yourself or you can look to buy it. And candidly, until the last handful of months, there really wasn’t a realistic commercial offering to start implementing some of these capabilities.

    Shimel: No, no. I think the bigger – you know, a lot of what you’re saying certainly obviously rings true. The other fundamental piece of this is, even if you are using one of these homegrown, cockamamie, you know Rube Goldberg kind of things –

    Rothman: Yeah.

    Shimel: – is it the security guys who are? You know what? Shame on me. It’s not the guys. Is it the security team who’s doing this, people? Is it the DevOps cross-functional team? Is it a little bit of both?

    Rothman: Yeah, yeah.

    Shimel: You know, how many of – look, here’s my basic, fundamental belief. You are not going to find a developer who raises his hand who says, “I want to write shitty code. I can’t wait to put something insecure out there.” No one says that. No one says that, right? They’ll tell you they’ve got time pressure, there’s pressure to release, pressure to release, they wish they could do more on security. And some of them, the enlightened ones, as you said, they’ll even try. They’ll run a vulnerability scan, because there are certainly open source kind of tools out there for them to mess with.

    The problem is, they go to a lot of the security guys, and as you alluded to before, security guys say, “Well we have our rules and procedures here. This is a no.” Right? And they don’t want to hear no. They’ve got to go.

    Rothman: You know, what’s interesting, Alan – I want to hit on a couple things that you alluded to there, because they’re important, right? The first is kind of where you start to see the agitation for better operational and security practices. And it’s not that it’s not the security folks. Obviously they’re always going to push for that. But it really is – you mentioned it, right? The DevOps teams that we’ve worked with, they’re very cognizant, they’re very serious stewards of kind of the corporate data, and they want to do the right thing. But in a lot of cases they don’t know what to do, right? So you have a whole bunch of folks who don’t know what they don’t know and they don’t want to go to the internal security team because of a lot of those things you were just alluding to. They’d say, “No, we can’t do that. We have all of these rules.” And in a lot of cases the security team has not updated their set of conditions, their set of policies – 

    Shimel: Exactly.

    Rothman: – to work in a DevOps type of environment, so these folks really struggle. They talk to – again, on the consulting side of our house at disrupt:Ops, we do a lot of work with those teams in terms of helping them understand what cloud native architecture looks like and how to integrate cloud security into their environment, and really just to help them build a program in order to do that. So then you do that with one DevOps team, then you end up talking to another DevOps team and maybe the third DevOps team, and finally IT, at some point, they’re like, “Um, what are you telling our folks? They seem to like what you’re doing and they seem to be doing things in a more secure way. So can you help us?” And then IT starts to get religion. The security team starts to get a little bit of religion about what they have to evolve to. And it really is kind of that interesting transition where it has started kind of out in the DevOps community with, again, those enlightened ones going, “I don’t know what I don’t know. Let me get some help,” and then it kind of back-ending into security. That’s not always the case, but probably three quarters of the time, that tends to be the transition that we see.

    And the other thing I want to hit is just about velocity, right? JA very forward-thinking DevOps leader that we’re working with, he said it the best. And he’s just like, “I’ve got 20 DevOps teams. I cannot slow these people down. They are doing stuff; they are making progress. The last thing I can do is put a bunch of onerous security requirements on what it is that they’re doing. They’ll ignore me, they’ll go around me and that’s counterproductive. I can’t do that. What I really need is to be able to build and enforce a set of guardrails around their environment.” Right? That term, “guardrails?” It just really kind of opened up and was a light bulb in our head when we were having this conversation. It’s not about stopping them from doing things. If anything, it’s about allowing them to go faster, yet making sure they don’t drive the car off the road, right?

    And that’s really the metaphor that resonates relative to what we’re doing at D:Ops, and really just kind of where the environment is going, in that we have to be able to move faster. Slowing things down is not an option, yet you want to make sure that you are not putting corporate data at risk. You are not opening up significant new attack services just – because people are moving so quickly. Sometimes they make mistakes. Sometimes automation isn’t done in the perfect way. Sometimes humans get involved and certainly are fallible from that standpoint. But with a lot of the inherent, underlying automation orchestration that’s possible within the cloud and DevOps and continuous, you have options to enforce these best practices in a way that we couldn’t do in our traditional data center with our traditional waterfall types of development and operational motions.

    Shimel: Absolutely. And you know, Mike – I mean, I’ve spoken about this a lot at various conferences and stuff. A lot of that has to do – you know, we use this term “shift left” in security and all that. A lot of it does have to do with getting to security earlier, faster. Yeah, we’re going to have to automate it. Yes, we can’t be the roadblock. But we can be the appendix after – when I say, “we,” I mean security. We can be the appendix after the delivery or after the deployment.

    Rothman: I’ll raise you. I see your point and I’ll raise you one, right? I’m of the opinion that security isn’t a thing. Right? Security should be within everything.

    Shimel: Yeah.

    Rothman: And that’s really the mentality that we’ve been trying to push on the industry for a long time.

    Shimel: Amen.

    Rothman: And it’s one of these things where we really can’t afford to wait and be in a position where we’re trying to retrofit poorly architected systems that are moving faster than we’ve ever seen before and then say, “Oh hey, we’ve got to change it because of security,” or because of any other operational gap that we see there. Right? What we have to do is be building in a way that allows us to use architecture as security control, right? Isolation, segregation, multi-factor authentical, identity management – all of these things that are fundamental building-blocks of what we do in the cloud and DevOps can be utilized to build in a much more secure fashion. And it’s really about those kinds of practices that are going to make the biggest difference relative to what it is that we see and what it is that we deploy over the next couple of years.

    Again, like I said, I don’t think security is a thing five to seven years from now. Because if it’s not embedded within everything that we deploy, we screwed up.

    Shimel: Yep. You know, I heard – because I never come up with original ideas. I just listen. But I heard something awhile back – “Security needs to become synonymous with quality.” And we’ve had QA as a concept in software technology forever or for a long time. Security needs to be just built in, like as part of the quality control. Right? The same way shit or stuff gets tested, whether it be hardware or software or whatever. Security needs to be part of that quality. 

    Rothman: You bet. To use a software term? Right? Why is a functionality defect different than a security defect?

    Shimel: Exactly.

    Rothman: Right? And they should all be within the same thing. And again, that’s a lot of when you start talking about immutable, when you start talking about kind of building security testing into your pipeline, which again, is stuff I know that you guys at devopsy.kinsta.cloud write about quite a bit. Again, it’s really not any different than you managing and making sure that you hit your functionality kind of specs from the standpoint of just building code, right? Part of that factory. So yeah, the code has to work, but it also can and should be secure, and there really aren’t different operational motions to make that happen. And that’s what’s so exciting. For a guy who spent 25 years trying to retrofit and get people to think about security as part of their core structure and to treat it as really a business function, it’s really enlightening to be in a situation where you actually can implement a lot of those functions within the core processes that you use to develop the technology.

    And again, are we there yet? Of course not. Is there a lot of education to be done? Without a doubt. But we at least have the option now, right? That’s the aspirational goal to what it is that we all do every day, which is to make sure that we’re pushing for the right types of decisions to be made as early in the process as we can, because we all know how much it costs to fix it later.

    Shimel: Absolutely. Mike, I’ve got to tell you, listening to this conversation, I’m thrown back to, “NAC: is it a feature or stand-alone?”

    Rothman: You’re dating both of us, man. Stop that out. Cut that out.

    Shimel: It’s such a pleasure to talk a little theory here with you. But let’s turn for a second to real stuff. Not that what we’re talking about isn’t real. But I wanted to give our audience a little bit of a taste – so disrupt:Ops – great people, great big problem to tackle out there. Give us some specifics. What are you guys doing?

    Rothman: Yeah, so what we’re doing is we’re basically building a cloud management platform to implement a lot of these ideas that we’ve been talking about for the last couple of minutes, Alan. Initially it’s a guardrails platform. So what we have is the ability to, in an AWS environment, enforce a set of best practices across security, operational and also cost management. So one of those things we don’t talk about too much is that developers don’t really like to shut their stuff down because, “No, it’s going to mess with something. I don’t have to. The CFO bitches but we don’t really care.” What we can do is also implement best practices in order for folks to clean up their room, right? We try to do that with our kids all of the time, too. You know, so we try to do that with our developers as well.

    So really implementing a platform for security, operational and cost management guardrails, to again, immediately not just find the issues – there are a lot of tools, as you said. A lot of vulnerability scanners can kind of look at your cloud environment and tell you you have problems. The issue with that is that most operations people are overwhelmed with the number of other things they have to fix every day. Adding more stuff to their list doesn’t really help them. So disrupt:Ops is really focused not just on finding the issues, but also fixing them and enforcing those best practices. Let me give you an example of that. So let’s just say some person decides they want to open up a server to the internet because it’s just easier for them to do that. Well obviously that would be a pretty significant security problem. So as opposed to just taking that server offline, what you can do is you can basically change the security group to make sure that it’s only accessible via a known, good IP address range within your corporation. That’s a good example of a guardrail. Or making sure that you’ve got MFA turned on for every account that gets spun up by your magical Terraform environment that may not build that out as a core security requirement.

    So again, there are 50 to 60 of these guardrails, in effect, that you can implement in your environment, to again, make sure that you are not just alerting when you get out of policy, but actually fixing the issues, in a lot of cases before you even know they’re issues, so that you’re not sitting there trying to pick up the pieces after half of your data is in Chechnya.

    Shimel: Exactly. Or Romania.

    Rothman: Or Romania. It’s somewhere.

    Shimel: Mike, how is the offering packaged? Is it a SaaS kind of offering? 

    Rothman: Yeah, it is. What we found is since – you know, remember the days where people were like, “Oh, I don’t know about this cloud thing or SaaS. I’m not sure I’m comfortable with that”? Now it’s like, “Wait, you want me to install software in my environment? What? What’s wrong with you?”

    Shimel: That would be kind of oxymoronic for a DevOps tool, but –

    Rothman: That’s right. That’s right. So we deliver as a SaaS service. It’s actually a very cool kind of one-click provisioning process where, when you’re logged into your AWS account, you run basically a cloud formation script that really kind of gives us cross-account privileges to your environment. And again, we’re security guys, right? So at the end of the day, we are basically not even just assuming but enforcing least privilege. So we only have read-only access. When we need to make changes, we grant the privilege as we need it, then we revoke it. Because again, what we’re trying to do is make sure we enforce operational best practices; not add more attack service. So again, we’ve built the environment to be secure by design. We use all of our own dog food, to make sure that we are adhering to kind of common best practices. And again, we’re all 25-plus years security folks, so we do take that pretty seriously.

    Shimel: Excellent. Mike, for people who maybe want to download – not download, excuse me – sign up for it or check it out more, where can they get more information?

    Rothman: Yeah, disruptops.com. So by the time you hear this little interview, the site will be live. There’s a place you can sign up. You know, first we start with a demo and show you what it’s about and then you can sign up for early access. And again, we’re kind of expanding towards the end of this month, in October. We’re going to add a bunch more folks to the system with the goal of kind of being ready to actually sell the product by the end of the year.

    Shimel: Fantastic, man. Well, I don’t have to tell you, Mike. I wish you, Rich, Adrian, Jody, Brandy and the rest of the team nothing but success. We’ll be checking in, I have a feeling, pretty often here, whether it be through Security Boulevard, devopsy.kinsta.cloud, Container Journal or what have you. And you know, fantastic. It’s good to have you back in the game.

    Rothman: Yeah, no, I’m excited. Again, really it’s not just a big promise and not just a huge opportunity. You know, and what Rich, Adrian and I, and now Jody and Brandy have always done is – you know, we’re trying to do the right thing to evolve the practice. Right? And that’s what it’s about. It’s a big world and there’s a lot of opportunity. There’ll be a lot of folks talking about guardrails over the next year. And that’s great, right? Because what we want to do is get to a position where we can move faster but move more securely, and I a lot of cases, in the world that we grew up in, those were counter indicators. Right? You couldn’t move fast and be secure. You couldn’t do operational excellence and make sure that you’ve got a lot of these things enforced. And now you can, and that’s what’s really exciting to us. Again, we’re just excited to partner with you, as always, and again, just go out and solve some bring problems for a lot of organizations.

    Shimel: Fantastic. All right. Mike Rothman, president and co-founder of disrupt:Ops, a hot new entrant in the DevSecOps space, with a great pedigree and a real reason for being our guest today on DevOps Chat. I hope you’ve enjoyed this. This is Alan Shimel for devopsy.kinsta.cloud. Until next time, everyone, have a great day. Bye-bye.

    — Alan Shimel

  • Talend Fall ’18 Enables Enterprises to Deliver Insight-Ready Data at Scale

    Major integration platform update adds advanced catalog and API development capabilities and dramatically accelerates machine learning at scale

    REDWOOD CITY, Calif. and LONDON – October 16, 2018 – Talend (NASDAQ: TLND), a global leader in cloud data integration solutions, today announced a major update to Talend Data Fabric, the company’s unified data platform for data integration across complex, multi-cloud and on-premises environments. Announced at Talend Connect UK, Talend’s Fall 2018 release delivers insight-ready data at scale with new features including Data Catalog, which creates a single source of trusted data, and Cloud Application Programming Interface (API) Services, which speeds access to standardized data for building data-as-a-service applications and increasing productivity. Talend’s new release also accelerates running machine learning algorithms at scale with extended serverless big data support.

    “Data is at the heart of digital transformation, yet the promise of data remains elusive for many companies,” said Ciaran Dynes, SVP of Products, Talend. “For businesses to survive and succeed in this ever-evolving, expanding and complex data landscape, there is a need to optimally automate every step of the data value chain and enable self-service for more data consumers. With this release, Talend helps enterprises dramatically improve how they can organize, process, and share data, which in turn enables organizations to collaborate and innovate on insight-ready data at scale.”

    “As part of our cloud-first strategy, Talend has enabled us to create a governed data lake with self-service access for business units and clients in an effort to monetize new services and generate additional revenues,” said Abderrahmane Belarfaoui, CDO, Euronext. “We compete in a highly regulated industry and face many requirements that push the processing, agility and cost of IT platforms. Using Talend, we have the confidence that our data is compliant, organized and highly sharable, with the efficiency and speed we require in order to stay ahead of the competition.”

    As part of the new Talend Data Fabric, Talend adds an API delivery platform enabling data-driven businesses to simplify data access. Talend Cloud API Services provides full API development lifecycle support from API design to test and deployment, enabling significant time savings in building and maintaining APIs. This API foundation serves as a standard and scalable way for companies to make data available to more users inside and outside an organization and to create new products and services that can drive new business models, revenue and profitability.

    “In this new era of digital transformation, nearly every organizational role is using data daily to inform decisions and actions,” said Stewart Bond, Research Director, Data Integration and Data Integrity Software, IDC. “Data cataloging software is experiencing a higher than market growth rate because it is filling a gap in the intelligence required to help organizations discover data and derive better insights to inform decisions and drive actions.”

    To organize data at scale, Talend adds new Data Catalog capabilities that enable a single source of trusted data. The Data Catalog automatically crawls all data, whether it is in a data lake, on-premises or in the cloud; profiles and classifies all the data; automatically finds and suggests relationships between data, and provides easy search-based access so end users can leverage trusted data when they need it to make immediate business decisions.

    The Fall 2018 Talend Data Fabric also accelerates running machine learning models at scale with native, extended serverless big data support for Microsoft Azure Databricks as well as Qubole in AWS cloud environments. Talend’s self-service Apache Spark processing makes it easy to spin up a cluster and control costs by just paying for what is used through serverless and spot instances. The new capabilities help to automate insight and enables businesses to process more data at lower costs.

    The comprehensive Talend Data Fabric delivers a unified platform for data integration across public, private,and hybrid cloud, as well as on-premises environments, and facilitates greater collaboration between IT and business teams. Combined with an open, native and extensible architecture for rapidly embracing market innovations, Talend enables customers to cost-effectively meet the demands of ever-increasing data volumes, users and use cases.

    The Fall 2018 update to the Talend Data Fabric will be available in November for free to customers with up-to-date paid subscriptions. For information on new features and capabilities, please visit https://www.talend.com/products/whats-new. Additionally, a webinar detailing the Fall 2018 update will take place on Nov. 8. Register here to join.

  • Are Legacy Apps the Antithesis to DevOps?

    Are Legacy Apps the Antithesis to DevOps?

    Do you have an application marked with cryptic warning signs and a wealth of cobwebs that is running on legacy hardware hidden away in the back corner of your data center? If you’re in enterprise IT, chances are high that you do. These old platforms are often considered a bane to IT. More importantly, legacy applications can present a real headache when attempting to uplift the people, processes and tooling needed to embrace a hybrid cloud model.

    Providing self-service, orchestrated workflows and delivery pipelines are just a few signature attributes to constructing a hybrid cloud. Imaging trying to apply these ideas to legacy technology, where the interface requires using an archaic console and software that looks like it was written for Windows 3.1. It’s not fun and often derails any efforts to deliver services living in an on-premises software-defined data center (SDDC) or public cloud environment. It also splits up a team focused on working within the realm of a DevOps cultural model, because concepts such as infrastructure as code, stateless configuration and collaborative code check-ins crumble. For most folks I speak with, legacy applications remain with a small skeleton crew that keeps the lights running and the old hardware humming.

    Hybridity is a reality for most infrastructures rather than being purely in the public cloud or purely on-premises. This is where the importance of DevOps and embracing an API-first mentality is multiplied. Having an API-first approach isn’t necessarily about having a single code repository or a single application that does “all the things.” It is about leveraging APIs across multiple repositories and multiple applications to weave together a single, programmatic software fabric in which everything can communicate and integrate regardless of whether legacy or cloud-native.

    The way to end chaos induced by the complexity with hybridity is a progressive, policy-driven state with proper implementation.

    Here are four tips on how to integrate your legacy crew with your DevOps team to create a high-functioning hybrid cloud model:

    • Find tools that play well with others: Legacy applications do not have to be the anchor holding you back from adopting a DevOps strategy. Find solutions that treat legacy applications, modern applications and cloud-native applications as if they were all first-class citizens. Legacy applications tend to run on old OSes, including some that are no longer supported. Eventually this configuration becomes increasingly fragile, requiring manual care and feeding. Adopting an automation strategy can reduce risk associated with build, testing, deployment, remediation and monitoring.
    • Strive for simplicity; don’t create silos: Make sure the workflows that deliver services are applicable to the vast majority of use cases. Historically there has been little to no incentive to building mechanisms that are shared across the enterprise. Systems thinking advises that there are dependencies, but if you only improve one application in the cluster, there will be no benefit. The entire cluster must be addressed for successful progress. The use of RESTful APIs is turning tables, allowing for a single set of tooling across many applications, platforms and services. Share data across the organization; leverage APIs to streamline how the team interacts with legacy workloads by extracting data from the legacy architecture.
    • Embrace a one-team mentality: Don’t form multiple teams or tiers of teams. If the decision is made to adopt DevOps, then it must be embraced by the entire organization. As the old adage goes: “There are no legacy systems, just legacy thinking.” DevOps isn’t about developers and operations folks collaborating; rather, it’s about two separate silos becoming one team. Focus on improving communication and set aside time for learning.
    • Avoid infrastructure-specific solutions: Abstract storage and compute; work instead on the application layer and how to deliver services for those applications. Legacy applications often are coupled tightly to the underlying hardware, making it challenging and risky to manage any application component individually. This means maintenance and upgrades are incredibly time-consuming, difficult and even expensive. The adoption of infrastructure-as-code, whether on-premises or in the cloud, gives teams the permissions and tooling provision infrastructure on-demand. As you can imagine, not every legacy application can be easily migrated to this type of infrastructure, but many can.

    Using automation increases the organization’s agility by reducing human interaction and orchestrating dependencies across the organization. Self-service consumption of infrastructure further eliminates silos. DevOps is not incompatible with legacy applications, but it requires an organization to evaluate what this implementation actually means and how to embark upon this transformation. Never fear: You can still depend on a legacy application and embrace DevOps. It requires additional design considerations, elbow grease and a little bit of creativity. The fundamental element is to apply these principles in a consistent and efficient manner in the context of all applications, both legacy and modern.

    — Rebecca Fitzhugh

  • Is NoOps the Future of Cloud Networking?

    Is NoOps the Future of Cloud Networking?

    As enterprises move more of their applications, data and operations to the cloud, competitive success becomes increasingly about speed. If the answer to a request to spin up and connect a new virtual cloud or deploy a new application across your hybrid cloud is “no,” or, “It will take us three months,” that’s a problem.

    Organizations operating in the cloud need to deploy in days, hours or even minutes, not months. They first sought greater infrastructure efficiencies and speed through ITOps, then DevOps, then DevSecOps and, most recently, CloudOps. Each of these nuances is a slightly better version of the old model, but all still revolve around humans. And the hard truth is that speed and humans don’t always mix.

    The painful implications of the impedance mismatch between humans and automation comes into sharp focus, but with a slight twist, when it comes to cloud networking. Now that networking infrastructure, meaning the actual routing of data, is moving out of the data center and into the cloud, it’s time to consider what a NoOps approach to cloud networking might look like.

    Bringing Networking Into the Cloud Era

    Way back in 2011, Forrester Vice President and Principal Analyst Mike Gualtieri famously wrote a blog post titled, “I Don’t Want DevOps. I Want NoOps.” In it, he said: “NoOps means that application developers will never have to speak with an operations professional again.” In this sense, NoOps is another term for automation.

    In the world of public cloud technology, the tension Gualtieri was pointing to—i.e., between dev and ops—lived almost exclusively in the realms of compute and storage. It didn’t apply to networking in the cloud, because networking is different. It’s more heterogeneous and distributed than compute or storage. It’s all about moving data, which has gravity and heft. Especially with big data, moving the data securely is an increasingly strategic priority for enterprises.

    Until recently, networking technologies used in public cloud environments have been either very limited-feature tools offered by the cloud provider, or virtualized versions of on-premises data center-era technologies. These virtualized technologies, such as vRouters, are complex (often by design) and difficult to master. They have kept CloudOps and DevOps teams dependent upon highly trained and certified networking experts to make and maintain cloud connections.

    This traditional networking paradigm slows things down and kills innovation. But it also puts developers and cloud operations people on the same side of the “I want NoOps” equation. That’s because unlike with cloud computing or cloud storage functions—where developers must wait for ITOps and DevOps teams to establish the computing or storage infrastructure their applications require—cloud networking has shifted the “operations” function to the hands of traditional IT networking experts.

    What NoOps Looks Like in Cloud Networking

    Recent technology advancements in cloud networking point the way toward a NoOps approach to cloud networking, meaning the automation of processes that now depend on direct control by human networking experts.

    One advancement is that the as-a-service revolution is finally reaching cloud networking infrastructure. What began with virtualized networking hardware has more recently progressed to software-defined (SD) cloud routing, enabling distributed, heterogeneous networking infrastructure to traverse public cloud, on-premises, hybrid cloud and multicloud environments.

    SD cloud routing centralizes and automates networking functions that previously required hands-on, time-consuming attention by highly certified human experts. As a result, SD cloud routing shifts networking infrastructure control directly to CloudOps and DevOps engineers, who are no longer dependent on networking professionals to establish or maintain their cloud networking infrastructure.

    Software-defined cloud routing alters the fundamental dynamics of cloud operations. It makes routing in the cloud point-and-click simple for CloudOps and DevOps experts.

    In a real sense, SD cloud routing achieves a NoOps approach to cloud networking—not by eliminating DevOps functions and roles, but by automating cloud networking and making it accessible to those who need to establish and maintain cloud connections. SD cloud routing empowers DevOps and CloudOps professionals to manage the networking connections required to move applications and data within, between, and to and from public cloud environments.

    — Steve Mullaney

  • Microsoft Puts Upgrade Plan in DevOps Motion

    Microsoft Puts Upgrade Plan in DevOps Motion

    Microsoft this week at its Ignite 2018 conference made it clear DevOps teams will have their hands full upgrading both databases and operating systems this Fall.

    Erin Chapple, corporate vice president for Windows Server, announced Windows Server 2019 will become generally available next month. As is the case with Windows Server 2016, there are three editions: Standard, Data Center and Essentials. The latter offering is for small businesses.

    However, the suite of management tools the company provides for its operating systems will not be available until early next year.

    Microsoft is putting the finishing touches on Windows Server Software Defined (WSSD) 2019, which provides another approach to combining compute, storage and networking on top of a hyperconverged infrastructure platform. That offering is expected to come bundled on a wide variety of hardware platforms early next year. Organizations looking to shift from a legacy rack-based approach of managing their data centers to a more modern HCI approach will have to wait for WSSD 2019 to become available.

    The company is simultaneously pushing forward with a public preview of Microsoft SQL Server 2019. In addition to adding support for faster queries and persistent storage in memory, Microsoft SQL Server 2019 packages the Hadoop Distributed File System and Apache Spark in-memory computing framework with a relational database. Microsoft is also making it possible to run SQL Server 2019 on Kubernetes clusters running on either Linux or Windows deployed on-premises or in the cloud. While the company has broadly embraced Kubernetes, its support for Kubernetes on Windows Server is not expected to be generally available until 2019.

    Microsoft also announced that a General Purpose service tier for the Azure SQL Database Managed Instance is slated to become available next week. That capability gives DevOps teams the option to deploy SQL Server databases either in a public cloud or on-premises.

    Microsoft touts the ability to run Windows Server on-premises and in the cloud as a core element of its hybrid cloud computing strategy. Right now, however, the most widely employed platform running on the Azure cloud is Linux. The company is also pushing another approach to hybrid cloud computing based on Microsoft Azure Stack, an on-premises edition of the cloud framework the company employs to drive its public cloud. An edition of Kubernetes is available in beta on Microsoft Azure Stack as well.

    Microsoft earlier this month revamped its AzureOps tools to make it easier to manage all those variations on its public cloud. But those tools have yet to be extended to on-premises environments.

    Of course, there’s no requirement that says any approach to a hybrid cloud using Microsoft technologies requires Azure. Windows Server and SQL Server both are already widely deployed across multiple public clouds. The challenge DevOps teams will face is managing all the different versions of Microsoft operating systems and databases that potentially will be strewn across an extended enterprise. It’s unlikely any organizations will be able to upgrade operating systems and databases in one wholesale effort.

    — Mike Vizard

  • Using Laravel and AWS: What You Need to Know

    Using Laravel and AWS: What You Need to Know

    Most likely, you have heard about and Laravel, the open source PHP framework, and you might be planning to go to the next level to modernize your PHP Laravel application. AWS offers a number of features to help organizations scale to support millions of users, not to mention the agility and cost benefits associated with a cloud infrastructure. These features and capabilities are the compelling promoters to transform your application into a high-performing and resilient PHP Laravel application.

    In this article, I will describe how to architect your software-as-a-service (SaaS) Laravel application with high-level scalability standards on AWS to help you understand what is required and discuss the considerations and action items to gain business value and resiliency. We chose the AWS cloud since is the most mature infrastructure-as-a-service (IaaS) provider and makes things easier to innovate; however, these practices can be applied to any cloud platform.

    I also will explain why AWS is a vital option to scale and grow your SaaS Laravel application and present the current/future enterprise architecture. In part two of this series, I will discuss the importance of the 12-factor app methodology and multiple lessons to include in your strategy when architecting on AWS DevOps.

    The Value of Migrating Your Laravel App to AWS

    In an article I wrote for my company blog on running the digital transformation on AWS for your enterprise or web application, I explained the business benefits that brings the AWS cloud adoption, including a reduced IT labor force, pre-configured services to achieve faster implementations and minimal downtime, due to the repeatable and flexible infrastructure. The cloud is an essential environment to scale SaaS Laravel applications—in there, you can decouple your architecture into multiple service pieces, including cloud storage (Amazon S3 and CDN) and automatic scaling EC2 instances (AWS Auto Scaling), and integrate various SaaS/platform-as-a-service (PaaS) AWS components for your modern, cloud-native application. Also, if you are planning to move to the AWS cloud, you could use this AWS Migration Checklist to prepare for your journey.

    “Web Applications and enterprises need to transform, they need to transform to a continuous transformation, and the cloud is the key.”

    – Stephen Orban

    I really enjoyed this phrase from AWS’ Stephen Orban, because you’re not only transforming your application; once on board the AWS cloud, you are in an infinite journey of continuous innovation: pay-as-you-go, microservices, serverless and so on.

    Laravel Architecture on Docker and Microservices

    Throughout the AWS cloud journey, there have been multiple disruptive evolutions, and one of them is the microservices, although, I won’t cover the microservices (Docker) approach. As a first step to transforming your application toward a cloud-native application, start with a monolithic approach. As your application matures and user base increases, I suggest jumping into a Laravel microservice approach, i.e., dockerize your Laravel PHP product using Docker, AWS ECS, Kubernetes or AWS Fargate. I’ve found many digital agencies that are adopting the AWS Cloud and DevOps practices with Docker/Serverless strategies and are getting overwhelmed with these sophisticated technologies, which has led to project failure because they were not ready to this stage.

    Here’s how to avoid that: Grow gradually.

    However, if your Dev team has enough experience with Docker and microservices, I wouldn’t hesitate to architect with microservices. This will be covered in the next article in this series, “Microservices with Laravel, Docker and AWS ECS.”

    Who is This Article For?

    Whether you are kicking off an MVP app on Godaddy/Hostgator, deploying Laravel on AWS with basic principles or ready to jump from hundreds of users into million users, this article is for you and will help you in your Laravel architecture on AWS.

    Laravel for Large-Scale SaaS Architecture on AWS

    So, let’s start by making clear what do you most need to scale a Laravel application on AWS.

    The diagram below illustrates how a next-generation Laravel application should be architected with the full range of AWS DevOps components. Each piece mentioned in this AWS stack is highly elastic, scalable and resilient.

    AWS Laravel Architecture Diagram

    AWS Auto Scaling and Load Balancing

    Initially, the dynamic traffic comes from AWS Route53, which will route all requests to the Application Load Balancing, which the AWS ALB will load balance across a fleet of EC2 instances ( 2…N Instances). This fleet should “scale” horizontally according to the traffic demands or the AWS CloudWatch metrics.

    Amazon S3 and CloudFront CDN

    All static content including images, videos and HTML will be hosted on Amazon S3, the cloud storage with infinite storage and elasticity. In front of Amazon S3, we will compound AWS CloudFront to cache the entire static content and reduce bandwidth costs. Integrating this pair of components is crucial.

    AWS RDS Aurora (Managed Database Services)

    The database data should be placed on AWS Aurora or RDS and AWS ElastiCache (Redis) for session management and cache user data. These AWS components are elastic and scalable, which helps to design the AWS Well-Architected framework with the highest fault-tolerance and resiliency.

    Amazon VPC and Networking

    In the AWS Networking level, we propose AWS VPC with a public/private enterprise network. Following PCI and HIPAA practices, there should be a VPN encapsulating the Laravel application within the private AWS enterprise network. Also included should be an AWS VPN connection and OpenVPN service to access the private network to your multiple Laravel environments.

    Route53

    From the DNS perspective, Route53 is needed to manage the application domain, DNS scalability, routing latency and more features to reduce failures.

    AWS Lambda

    This serverless component will be used for async jobs and backend scripts to minimize load in your AWS Auto Scaling cluster.

    We will explore these AWS components substantially in detail, later in the third article of this series.

    Web applications that can fit within this architecture:

    • API Laravel applications
    • Mobile backend applications
    • Laravel enterprise applications: CRMs, ERPs and back-office systems
    • SaaS Laravel applications
    • Front-end applications. i.e., Angular, React, Vue.js or any HTML site
    • ANY web platform: Python, Node.js, Ruby, PHP, Zend, CakePHP, Yii and Symfony

    A word of warning: Don’t use Cpanel for any web application. It is not recommended to scale as it can slow down your application performance.

    — Alfonso Valdés

  • Successfully Integrating DevOps while Deploying Cloud Applications

    Successfully Integrating DevOps while Deploying Cloud Applications

    Transforming to a high-performance DevOps environment can provide competitive advantages, but it requires change—and that doesn’t come easily to most organizations. DevOps requires cultural changes that are inherent to automating previously manual tasks: They are perhaps intimidating to those involved and they require flexibility and enterprisewide buy-in to get right. Furthering these people-side transformational complications are cloud application deployments, creating a need for enterprises to deliver innovative technologies and services with greater speed and quality—without hits to reliability and security.

    DevOps Blended with AWS Proficiency

    One approach to this problem is blending DevOps knowledge and Amazon Web Services (AWS) proficiency to guide development teams to better align the people, processes and technology needed to facilitate successful cloud initiatives.

    In transitioning to a more modern, DevOps-driven approach, companies often encounter a number of challenges, on both a technological and cultural level. The most successfully integrated DevOps occurs as a function of value analysis and process overview, addressing the individual needs of an enterprise rather than fitting them into a preset mold.

    Implemented correctly, DevOps blended with AWS proficiency allows organizations to maintain quality and configuration of their infrastructure environments, allowing application development teams to retain control over their processes and toolsets. AWS also allows the enterprise to maintain security: Rather than protocols that are standalone, they’re built into IT processes from the beginning, allowing customization with ease.

    AWS understands what is important to its partners when building tools, successfully addressing needs while staying ahead of the demand and, in turn, saving time. They accelerate the ability to deliver and allow an enterprise to focus on what’s most important during their journey to the cloud: value analysis.

    Fundamentally Designed to Refine Business Processes

    It is very important to approach projects first from a top-level view: Where does this piece of work fit into my broader organization?

    DevOps at its core is fundamentally about incorporating different disciplines of a business to create better practices than what are already in place. When an enterprise chooses to implement automation, it typically gains a better view of its business as a whole, incrementally increasing awareness at a birds-eye level. When complex business functions and operations are laid out in more mathematical terms and formulas, it suddenly becomes clear where applying automation can reduce waste.

    Getting these changes implemented quickly and with as little effort as possible naturally leads to successful automation. Enterprises remove such tasks that traditionally required an excess in resources by asking, “What value doe this process add? Is there a more efficient alternative?” With automation, these useless business practices are eliminated, freeing resources for more important initiatives.

    Increase Employee Awareness, Increase Productivity   

    One of the most important parts of the entire DevOps integration process is ensuring that everyone understands each other and their role. Properly embraced, a DevOps culture will be characterized by increased collaboration, shared responsibility and autonomous teams—leading to improved quality, valued feedback and the end goal of increased automation.

    People are naturally resistant to change. Members need to be aware of the business and how they—and will continue to—fit in.

    Ultimately, DevOps is not something that can simply be bought. Success during implementation requires the application of technologies, tools and products that are actually a fit between new processes and existing systems.

    — Ashley Stovall

  • Digital Transformation Strategy? Think Cloud

    Digital Transformation Strategy? Think Cloud

    A few years back companies used innovation and digital transformation mostly to differentiate themselves and to stay competitive. The drastic growth in digital and cloud computing over the last couple of years have changed this mindset.

    Today, organizations have to be innovative and leverage the latest technologies just to sustain and stay in business. Enterprises that implement online retail business, online banking and several other online services aren’t considering those channels simply as another route to increase their revenue—they have realized that online services are fast becoming their primary revenue channel. According to Forrester’s Report, “The Digital Business Imperative,” data analyzed just for three months showed that 84 percent of U.S. banking customers had used online banking for their transactions and 43 percent of the customers had used a mobile phone for these activities.

    Organizations are working at a very fast pace reviewing and analyzing their processes to seize opportunities for digital transformation. It is important to understand that to realize digital transformation, companies may need to completely re-engineer their current processes to make use of technologies such as the internet of things (IoT), big data analytics and artificial intelligence instead of doing patchwork on existing processes to adapt to digital technologies. It is also important for senior IT executives to consider digital initiatives in tandem with their cloud strategy instead of treating them in isolation. There is absolutely no doubt that the scale, speed and massive infrastructure requirements such as compute power, storage and other associated platforms to support these technologies are best provided by the cloud. The question to ask is, “Are we going to develop and host the infrastructure we need to support digital, or are we going to leverage the constantly improving robust infrastructure and services that cloud providers are offering?”

    Organizations that plan for massive growth and transformation are usually the ones that invest considerably in technology to support innovative ideas. Advancements in the digital space help fuel innovation, but without a solid cloud strategy accompanied by agile development processes, ideas are likely to remain simply as ideas on paper and will take forever to get materialized into products or solutions that provide value. In today’s context, digital and cloud are almost inseparable due to the robust infrastructure, services and tools available on the cloud to support digital initiatives. All leading cloud providers offer competing solutions and services to help organizations move forward with their digital initiatives at a rapid pace.

    In my previous article, “Transforming with the Cloud: If Not Now, When?” I touched upon some of the key areas that organizations should be considering with regards to transforming their businesses with the cloud. I have attempted to structure this article in a similar fashion, with the focus being on the role played by cloud to support some of the key digital technologies such as artificial intelligence, big data and analytics and IoT. The idea is to provide a very high-level overview of the digital landscape and how the leading cloud providers are helping enterprises with their digital initiatives.

    Artificial Intelligence

    The digital revolution, which centered around mass production of computers and communication devices, has changed the way businesses have operated over the last several decades, resulting in constant improvements ranging from generating new ideas for products and services to innovative product designs to improving the customer experience. At present, the world is going through another, possibly even stronger wave of revolution. It is the use of artificial intelligence to perform complex cognitive tasks to solve business problems in ways that previously were either highly complicated or extremely resource-intensive. Most organizations deal with business propositions that yield small to medium value and require a high volume of human work, such as reviewing large numbers of documents such as RFPs to understand requirements and estimate costs versus returns to determine whether to proceed with those proposals. AI is proving to be the best alternative to handle cases such as these that are currently being dealt by human beings, but the volume of work involved challenges businesses on the feasibility of continuing those without AI.

    AI systems try to mimic the human brain, which uses patterns to generate perceptions, and logic to drive the structured approach of analyzing a situation from a rational perspective. AI systems process large volumes of data that could come in from various sources such as sensors, online applications, textual data from social media and the like. AI processes the data using perception to analyze patterns and incorporates machine learning to utilize structured evaluation methods and rational decision-making, not only to extract pieces of meaningful information, but also to assemble such pieces of information to make valuable decisions.

    Cloud plays a significant role in enhancing the power of applications that incorporate AI. Almost all major players in the cloud business have developed AI services that use powerful cognitive engines to process structured data such as relational databases and unstructured data from NoSQL databases, sensors and other technologies that get uploaded to the cloud. The robust pattern-matching algorithms and the powerful logic component built into these cognitive engines are highly sophisticated. Data and compute power are the two most critical requirements to make these engines effective. The engines predict more accurately with larger datasets. AI applications such as image recognition, video analytics, natural language processing and speech recognition leverage machine learning using highly sophisticated neural networks that perform detection and prediction from large volumes of data. Parallel processing with the use of graphics processing units (GPUs) make these data processing and computations faster. Building and implementing such robust GPU-based parallel processing engines on-premises would be a very expensive and resource-intensive initiative. Cloud addresses this problem by providing APIs to access several machine learning services such as video analytics, speech recognition, process automation, vision detection and natural language processing. Behind these APIs are complex infrastructures that combine the power of clusters of GPU-based compute engines, neural networks and data lakes.

    Amazon’s AWS Marketplace offers machine learning and AI software solutions from several major software vendors to help AWS customers increase the efficiency and productivity of their businesses. AWS’ artificial intelligence solutions offer several pretrained services that can be utilized by developers to add intelligence to their applications via simple API calls. Advanced deep learning functionalities are provided by Amazon Lex, which chatbots could incorporate for automatic speech recognition and natural language processing abilities. Similarly, AWS provides Amazon Polly to build applications that can convert text into speech delivered in natural sounding male or female voices. Amazon Rekognition, meanwhile, uses technology to analyze billions of images, so applications can be developed to search images, detect and compare faces, objects and scenes.

    Google Cloud also offers competing products and services to support their customers’ AI initiatives. Google Cloud Natural Language provides powerful machine learning models that analyze text to provide meaningful insight such as customer sentiments about a product via conversations on social media. Google provides Cloud Speech-to-Text, a service that understands about 120 languages and converts audio to text. Similarly, Cloud Text-to-Speech can convert text to natural-sounding speech in 30 different voices and in several languages. Google also provides products such as Cloud AutoML, a suite of machine learning products, and Cloud Machine Learning (ML), a managed service that can be leveraged by developers and data scientists to build powerful machine learning models.

    Microsoft Azure provides the infrastructure, tools and services to build smart applications with data that can live in the cloud, on-premises or on edge devices. Several products and tools are available via marketplace, and Azure provides sophisticated AI services to integrate the tools to develop intelligent business applications. Powerful chatbots could be developed by integrating Azure Bot Service, language understanding and external systems such as Office 365 and Dynamics CRM to streamline and automate common activities. Another example is an automated failure detection system for assembly lines built based on Azure’s image classification system using a convolutional neural network to help improve the identification of faulty electronic components. Azure also provides Visual Studio Code Tools to build, test and deploy AI solutions. The powerful AI Toolkit for IoT Edge from Azure enables IoT devices with AI that runs locally through pre-built deep learning models. Last but not least, Azure’s Machine Learning Studio helps develop and manage predictive-analytics solutions.

    Big Data and Analytics

    With the growth of internet, cloud and social media, there has also been an exponential growth of data across the world. According to statistics on big data generation in the last five years, the average volume of data created in the world every day is about 2.3 trillion Gigabytes. When the nature of the data was more structured and organized, companies relied on data warehouses and BI applications to help make important data-driven business decisions. Traditional data warehouses were built based on relational databases that could be queried using SQL; the data could be extracted, transformed and loaded from one or more data sources via ETL jobs that ran up to several times a day. This overall approach has proved to be ineffective when it comes to handling and modifying continuous streams of real-time data flowing in from multiple dissimilar sources such as social media, IoT, the public web and relational databases. Big data analytics, which helps examine large structured and unstructured data sets, has become a major enabler for enterprises to make critical business decisions by providing insight and knowledge with data mining, predictive analytics and forecasting. As big data processing evolved, the concept of data lakes came into existence. A data lake is a centralized repository that allows the storing of structured and unstructured data as is and enables the use of a variety of tools and approaches to answer business questions.

    The ability of the cloud to scale vertically and horizontally makes it the ideal platform for big data hosting and analytics. With vertical scaling, it is possible to increase the capacity of a server by adding resources as needed by the applications. Horizontal scaling allows expanding hardware as the processing requirements increase. Hadoop, which leads the big data revolution, is designed as a distributed system so that it can take advantage of scaling. Parallel processing is an important part of its design so that the system can process several independent small tasks such as serving data stores and file systems, processing streaming data and handling queries in tandem. Cloud-based systems offer high bandwidth, enormous amounts of memory and scalable processing power to help big data applications with improved real-time processing and analysis of streaming data. Cloud undoubtedly is an obvious choice for applications running large workloads and storing enormous volumes of data. Cloud providers offer highly scalable database services coupled with tools and services to support information-management, business intelligence and analytics.

    All major cloud service providers offer competitive and comprehensive tools and approaches to process big data at the volume and velocity that your business demands, while at the same time being conscious to reduce the associated operational costs. To perform faster and efficient transfer of petabytes of data to the cloud, AWS provides solutions that employ different techniques such as network multiplexing, reduced storage footprint via data deduplication and compression and advanced data replication. AWS offers services including Amazon Elastic Compute Cloud (EC2) and Simple Storage Service (S3) to provide the elasticity needed for your data lake along with your choice of database products, ETL and reporting tools via AMS Marketplace solutions. Similarly, with Microsoft Azure HDInsight, solutions can be developed using open-source frameworks such as Hadoop, Spark and Hive to analyze streaming or historical data. To process large volumes of data, HDInsight offers a broad range of cost-effective solutions for ETL and data warehousing. Azure HDInsight offers specific cluster types such as Apache Hadoop, Apache Spark, R Server and Apache HBase, with cluster customization capabilities such as adding components, utilities and languages.

    Internet of Things

    IoT is the concept of connecting devices such as security sensors, surveillance cameras, smartphones, smart watches and other wearables and even household appliances such as washing machines, refrigerators and several others that have the ability to communicate and transfer data over the network without direct human interaction. This disruption in technology has not only empowered individuals and families with greater control of their household systems and appliances, but enabled organizations with data that provide greater insight into critical areas of business interest, opening the doors for innovative products, solutions and new business opportunities.

    The rate of adoption of IoT by businesses is growing exponentially as the number of connected devices continues to increase. As per Gartner’s recent study, businesses could already be using as many as 3.1 billion IoT devices today, and by 2020 this number is likely to rise to about 7.6 billion. According to an article in Forbes, IoT will have an impact in all industries, from manufacturing to logistics to health care. We are not too far from the days where nearly everything in the world will be connected. IoT is at present playing a dominant role in optimizing production, managing supply chains, tracking assets, making financial decisions and improving the customer experience. Digital transformation of the fitness industry has been remarkable. Wearable devices with built-in sensors can constantly collect data on physical activities such as distance travelled, calories burnt and sleep patterns to provide detailed analyses and insight for continuous health care. IoT applications are designed to use data from connected devices, and the sophisticated tools available in the cloud let you visualize, explore and build complex analytics. In complex IoT applications that use several devices, it is important to understand the state of the devices and to frequently communicate to the application components that leverage those devices. It is also essential to ensure secure identity and access between devices and applications.

    As with any revolution in technology, IoT also has its associated challenges. With more devices being connected to the internet, the volume of data generated is also immense, which in turn puts significant pressure on the internet. So there obviously is a need to support infrastructure that can transmit and store this data more efficiently. Due to the vast number of connected devices, and with that number constantly increasing, there is a push toward making these “edge” devices intelligent to perform some degree of processing and then send the results to the servers rather than sending massive amounts of data to central servers to perform all the processing. For example, surveillance cameras in the past would send all the videos to the central recording device, which in turn would record only when it detects motion. Consider the scenario where hundreds of high-definition surveillance cameras are sending video feeds constantly to the central server—the impact this could have on the network is unimaginable. It should also be noted that as the physical distance between the devices and the server increases, network transmission latency increases. With edge computing, the video cameras are smart enough to sense motion and they send videos to the cloud-based central recording system only whenever they detect motion. This drastically reduces the volume of data that is transmitted over the network and makes it a more efficient system. Cloud and IoT complement each other—connected devices generate huge amounts of data and cloud provides the infrastructure to store, process and analyze the data. It is also interesting to note that IoT is an important data source for big data analytics.

    Cloud IoT Core service, built by Google Cloud Platform (GCP) partners, offers connectivity to devices by providing the necessary hardware, software and solutions. These services include secure connection of devices (via identity and authentication) over industry standard protocols including MQTT and HTTP.  Cloud IoT Core also allows remote controlling of devices from the cloud by maintaining a logical configuration of the devices. Since this IoT service is serverless, there is no need for upfront software installation and the horizontal scaling of the GCP can instantly be leveraged when required. A host of tools such as Dataflow, BigQuery, Bigtable and Data Studio are available to help with data analytics and insights.

    Microsoft provides services for IoT via the Azure IoT Hub. Features offered by the Azure IoT Hub include ensuring bi-directional communication with billions of IoT devices, establishing device-specific access control, secure and scalable provisioning of devices and integration between cloud and edge to distribute intelligence across devices. Azure IoT Edge provides features for optimized performance of the edge and cloud such as ensuring the lowest latency between edge devices and the cloud. IoT Edge helps control IoT costs by reducing the volume of data transferred from the edge devices to the cloud. It leverages services such as Azure Stream Analytics to process data at the edge and transmit only the bare minimum data that is needed by the cloud for further analysis.

    Amazon provides a fleet of services such as AWS IoT Core, AWS IoT Device Management, AWS IoT Analytics and several others to support IoT initiatives. For easy deployment, programming and management, Amazon offers the FreeRTOS lightweight operating system for small edge devices. The AWS Greengrass software further enhances the power of edge devices by offering local compute, secure messaging and data caching. AWS IoT Analytics, a fully managed service, provides all the capabilities required to run complex analytics on IoT data to obtain insights and to build machine learning use cases.

    Taking Everything Into Account

    Technology is certainly a major driver in any business today and companies that struggle to integrate technology effectively will eventually find it difficult to sustain or do well in their marketplace. Connected devices, social media and massive volumes of several forms of structured and unstructured data feeds have paved the way to further leverage technology and transform businesses. This “digital transformation,” which includes use of artificial intelligence, big data analytics and IoT, is fast becoming a key requirement for organizations to be innovative and competitive. Digital has opened doors for better analytics and decision-making, leading enterprises to explore, analyze and obtain new insights and ideas on growing their businesses.

    With cloud reaching a high level of maturity with regards to services, tools and security, it has become the ideal platform for digital technologies. All leading cloud providers have comprehensive and competitive solutions, tools and associated services to address even the most complex digital transformation initiatives for their customers. Furthermore, the horizontal and vertical scaling of infrastructure offered by the cloud makes it highly suitable for the compute requirements demanded by digital technologies.

    Escaping from or ignoring the digital revolution will be disastrous to organizations as well as individuals. Humans and businesses are generating trillions of gigabytes of data every day. Whether we have realized it or not, digital is transforming our lives. It is important for organizations to get on board soon and devise a digital strategy that works together with their cloud initiatives and strategy.

    — Sridhar Asvathanarayanan

  • IBM Boosts On-Premises Clouds with Cloud-Native Development Capabilities

    IBM Cloud Managed Application Services synchs with IBM Cloud Private for Next-Gen Application Development

    ARMONK, NY – 21 June 2018: IBM (NYSE: IBM) has announced that for the first time, its industry-leading IBM Cloud Private application platform, will be made available to run on the company’s enterprise Cloud Managed Application Services (CMAS).

    Hundreds of world-leading banks, airlines and government organizations already rely on IBM Cloud Managed Application Services for managing enterprise applications on security-rich clouds. IBM is now enabling its CMAS customers to help extend their capabilities and maximise existing investments by synching with IBM Cloud Private – a new solution based on Kubernetes containers designed to rapidly build, modernize and deploy applications in client-managed environments.

    By bringing the two together, clients taking advantage of the security rich and production-ready cloud environment that CMAS offers, can now reap the benefits of the latest cloud-native DevOps capabilities of IBM Cloud Private, accelerating time to market for innovative, cloud-native applications.

    “We see that organizations are aggressively transforming to respond faster to changing market, customer and competitive demands,” said Mark Slaga, GM for IBM Cloud Managed Application Services. “To support them in their journeys, IBM is continuously innovating and adapting its cloud offerings. By combining the power of IBM Cloud Managed Application Services and IBM Cloud Private, customers will be able to benefit from a highly-secure and versatile environment for managing and modernizing apps and turning decades of data into competitive advantage.”

    Launched late last year, IBM Cloud Private is front and center of IBM’s hybrid cloud strategy. Thanks to the flexibility of container technology, it can be installed on a wide range of enterprise systems to create a private cloud with an architecture and capabilities consistent with the public IBM Cloud. IBM Cloud Private’s combination of innovation, flexibility and control is particularly important to companies in regulated industries such as finance with high standards for security and reliability.

    Coupled with CMAS’ built-in security, disaster recovery, and automated infrastructure and application management, IBM’s clients can now have a powerful combination that  helps provide the security, flexibility and speed they need to compete in the most dynamic markets.

    The solution uses the power of Kubernetes container technology to add self-service Platform-as-a-Service capabilities to IBM’s managed CMAS cloud infrastructure. In addition, IBM’s CMAS clients can benefit from faster access to IBM’s analytics, data, middleware and Watson portfolios available through the IBM Cloud Private platform.

    IBM Cloud Private running on CMAS will be fully operational by July 2018 and will be fully supported by IBM services.

    Contacts:

    Jonathan Jessop Batty
    IBM Europe
    jjbatty@ibm.com
    +44 7741 113871

    Scott Cook
    IBM US
    scotty@us.ibm.com
    +1-312-669-4743