Tag: time series database

Transcript

Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Tim Hall, Vice President, Products, at InfluxData, and our topic is their new offering, InfluxDB Cloud 2.0. Tim, welcome to DevOps Chat.

Tim Hall: Hey, Mitch. Thanks for having me.

Ashley: Excellent. Great to have you on the podcast. Would you start by telling us a little bit about yourself, what you do at InfluxData and maybe for somebody who doesn’t know InfluxData does, give us a little bit about Influx.

Hall: Sure. So, I’m the Head of Products here at Influx. I joined almost three years ago. I’ve got a background in big data open source technologies. I spent some time at Hortonworks running the Product Management Team there. They’re focused on the Hadoop space. Prior to that, I spent time at Product Management at Oracle addition HP, mostly in the monitoring and data integration spaces.

So, Influx is a great opportunity to join a small company at the time. We’ve grown to almost 160 people now over the past 3 years, and it was sort of at the convergence of all the things I’d done in my career, from monitoring technologies to integration to big data. And so, it was great to be invited by Evan Kaplan, our CEO, and Paul Dix, our Founder, to join the team and see what I could do to help build a great product.

Ashley: Fantastic. And tell us just briefly—InfluxData, what do you do?

Hall: So, InfluxData is focused on providing a platform for time series data, which is about anything with a time stamp. Time series can be broken into two categories—regular series, which are things that you sample. Think of it like sticking a coffee cup in the river and pouring it into a bucket, and we’re the bucket, and you do that on a frequent basis—every 30 seconds, every minute, every 5 minutes, but it’s regular. It’s a regular time interval. And irregular series are also supported by Influx, which are things like events or logs, things that happen at any point in time that could be somebody doing key card access to a building, putting their ATM card into an ATM machine—those sorts of things. Those are all events.

And so, we handle both of those two things and we have lots of folks using us, both in the open source community, on prem, or wherever they decide to play the software and now our cloud offering as well across a wide range of use cases, everything from DevOps monitoring use cases and building and assembling platforms for observability across to IoT use cases, both for consumer and industrial products.

Ashley: Well, in a world where we just can’t seem to generate enough data, it sounds like you all are well positioned in the market. [Laughter]

Hall: There’s more coming every day, right? So—

Ashley: Yeah, it’s not getting less.

Hall: Yeah, what do you do with it is really the question. And I think, again, we’re focused on providing a platform for developers. And our goal is really to drive developer happiness. And our motivation is trying to allow developers to solve problems quickly from the time of install to problem being solved. We call that the time to awesome. We want that to be as short as possible, and that’s not always the case with a lot of technologies out there. You’re fiddling with dependencies and other things and Influx comes out of the box with no dependencies and install it, get it running very, very quickly and get going and get your problems solved.

Ashley: I do really appreciate your focus on the developer community. Actually, we—I just did a webinar a few weeks back with Tom Crow, your Community Manager, and he did an excellent job, by the way, of talking about how he worked with the developer community.

So, let’s jump to InfluxDB Cloud 2.0. Is this your first SaaS online offering, is that correct?

Hall: It actually isn’t, Mitch, but I’m glad you asked. So, previously, we’ve had InfluxDB Cloud 1.0, which was really taking our enterprise edition and providing a managed service on top.

And the challenge we had with that is, we’ve seen good adoption and nice traction. We’ve had a large number of customers purchase and buy it, but the challenge is that it was a little on the expensive side, and it existed as a dedicated instance of our enterprise software that we ran and managed for them, but it was constrained on specific resources. And so, you would buy a plan, pay for it, and then if you continued to grow and use it, then you would have to upgrade, typically by contacting support, and we could do that in a matter of seconds, but it really required a lot more high level interactivity than maybe you might get from the vast majority of cloud services that you could think of, more consuming like a utility.

And so, our focus with InfluxDB Cloud 2.0 was to deliver the first completely serverless time series platforms, deliver that elasticity for our customers to just adopt and go.

Ashley: So, truly a cloud service, not just an application in the cloud, if you will.

Hall: That’s right, that’s right. And all the capabilities we’re exposing in a multi-tenanted fashion, which means we’re obviously testing and validating across a large group of folks all at the same time, and keeping that platform up and available for everybody is our goal.

Ashley: And I know you’re doing more than just storage in the cloud. You have visualization, UI—other capabilities? You wanna talk a little bit about that?

Hall: Yeah, that’s right. So, you know, there’s sort of three parts to this story, right? Part number one is, you need to be able to feed the data in, and you need to be able to do that with velocity, and we need to store and land that data and we need to do that quickly and make it available for query and access and all the way through to visualization.

And so, we have all those parts of the story together. The most recent addition also includes the ability to create monitoring checks that allow you to create queries against the data that you’ve stored and decide if they, let’s say, exceed a certain threshold or create a change of state in some way, shape, or form. And then from there, you can create a notification that can be sent to you through a variety of channels including HTTP POST, Slack—which is available in our free tier—and then, obviously, PagerDuty, which is a super popular mechanism for alerting your DevOps teams.

Ashley: Mm-hmm. And I think, in addition to kind of integration level type functionality, you’ve also updated the API to InfluxDB in the cloud, correct?

Hall: That’s right. Actually, one of the goals is, you know, through the 1.0 line of the TICK stack, T, I, C, and K, which many DevOps people might be familiar with, there were four parts—Telegraf, InfluxData (the database itself), Chronograf (which is our visualization tool) and Kapacitor (which was our anomaly detection).

And we’ve really taken the I, C, and K portions of that stack and integrated them all together in our InfluxDB 2.0 offering. This is both open source as well as cloud. And what we wanted to do was bring together the language, first and foremost, that you use to interact with data, both from a query and a task perspective, and we’ve done that through a new language you’ve introduced called Flux—not surprising, the name of the company is Influx. So, Flux is the new language we’ve introduced. And then we’ve obviously kept Telegraf, which is our data collection agent that can be distributed and used to send data in.

Now, one of the other things about T, I, C, and K is, each one of those elements had its own API. And so, if you worked with one, it didn’t necessarily translate to the other. So, we’ve unified and have created a common API as part of InfluxDB 2.0 that allows you to do everything from drive the creation of dashboard sales and dashboards themselves all the way through to submitting queries, checks, notification rules, et cetera, all behind a common API.

And that API is consistent across the open source and the cloud editions. That allows [Cross talk]—yeah, sorry, that allows developers to essentially write an application, either against the cloud or against open source. And if they need to swap it around for any reason—for example, if your journey starts with the open source and you’re like, “Okay, now I need the scale and elasticity offered by InfluxDB Cloud 2.0,” they don’t have to make any changes to their application code. They can just change the end point and point out our cloud.

Ashley: Very interesting. You know, we’re building applications that are API centric much more today versus let’s expose the application through an API. Did that change how you redesigned the API the way you’ve integrated all these components and does just kind of InfluxDB in the cloud more or less use its own APIs because it works together across these functions?

Hall: It does. We’re actually consumers of our own APIs. And frankly, that’s been our philosophy for a while. The challenge had been previously that it was fragmented across the four components of the TICK stack and we’ve really tried to unify that and think about, again, delighting developers and giving a consistent API experience across them all.

Ashley: One of the things you mentioned, too, about—and thanks for that information about TICK stack. I’ve got another question about that in a moment. But back to your 1.0 offering versus 2.0—have you changed kinda the pricing model, done anything to make that more accessible to either developers or enterprises

Hall: Yeah, completely. In InfluxDB Cloud 2.0, it’s a completely usage based pricing model. We’re also starting developers off with a free tier, and so that free tier is rate limited, but you have access to most of the features and capabilities. We’ve obviously held off some of those things behind the paywall, but generally speaking, it’s now all pay as you go and we’ve got that pricing model published and up on the website as well as in the documentation that describes exactly how much you pay for reads, rights, storages, and so on, and you only pay for what you use.

And that’s really the goal of providing that serverless offering. People wanna use it as a utility and, you know, compare and contrast with what we did with the managed service in the 1.0 line, you know, they were paying for things they weren’t using, right? It was a fixed price offering and so, you know, that’s why it looked a little more expensive. So, I think now, we’re, I think, price optimized for developers to grow as they’re successful.

Ashley: And one of the things that probably in any developers mind listening to this podcast is thinking about developers definitely don’t like crippleware. So, you’re focused around data retention limitations, rate, that kind of thing. I’m sure you’ve heard that from your developer community about how to properly construct a free tier offering that’s still very usable to them.

Hall: Yep, yep. I think we’ve got a pretty good mindset around that. On the data retention side, we’re currently allowing the storage of data for at least 3 days at 72 hours. That can be a lot of data in the world of time series. And the ingest rates are also, I would say, fairly reasonable exposed. Just to give you an insight, I actually am using the free tier myself. I haven’t arm wrestled by Product Manager to give me access to all of the features and capabilities, although I suppose that’s possible, but I wanted to see what I could do.

Ashley: Mm-hmm.

Hall: And so, one of the things I happened to do with my son recently is, we built a gaming PC for him. And, you know, one of the most expensive components, the two most expensive components in the machine are the graphics card and the CPU.

Ashley: Mm-hmm, yep. I would’ve said the same thing. [Laughter] Do the same project with my sons, but yes.

Hall: That’s right. And so, one of the challenges, of course, is that I wanted to make sure that we installed the CPU correctly and I was worried about it overheating. And so, we’ve got a nice, big, you know, heat sink on there. But I enabled the, what’s called the sensor’s plug in within Telegraf to basically give me access to the CPU temperature data. And I’m taking that and I’m actually sending it to Influx Cloud. And then what I’m doing is, I’ve created a check to check on the temperature and I’ve created warning and critical threshold levels, and then I send the notifications to myself in Slack.

And so, generally speaking, it’s worked quite well, Mitch, but there’s been an interesting side use case and benefit that I’ve found, which is, my son comes home and decides not to do his homework.

Ashley: [Laughter] Yep. I was just wondering.

Hall: [Laughter] So, now I have a very uniquely tuned child monitoring service that allows me to recognize when he is gaming when he should be doing his homework. And he still hasn’t figured out how I’m doing it, so I’m definitely not sending him this podcast.

Ashley: You’re too sneaky, dad. Parental controls and he doesn’t even know it. [Laughter] Too funny.

Well, I’d like to touch back on the TICK stack, because I know you’ve been very active with providing open source software through the Telegraf and InfluxDB, Chronograf, Kapacitor. What’s the plan, then? You said you’ve kinda done a lot to the open source or to the code and built that into this cloud offering. What’s the plan with the open source going forward?

Hall: Yeah, thanks for asking. I mean, one of the core values of our company is that we are strong believers in open source. And, as part of that, we’ve been working on the 2.0 offering for more than a year, and it’s gone through a series of alpha editions, and I think we’re currently up to alpha 18, which matches the features and capabilities that we have available as a GA instance for InfluxDB Cloud 2.0. And we may have some community members out there scratching their head, saying, “Um, so, what gives? How come this hasn’t gone GA?”

And I wanna be really, really clear, because we have such a large community that we value their feedback, but I also value not giving them something that’s broken and something that’s not ready for prime time. And so, we’ve been very careful about labeling the 2.0 code line in our GitHub repository as alpha code as we work to complete the features and capabilities that we want to have within the 2.0 line.

And so, the key things that are coming up for us next, there are sort of three things. Number one is, we’re moving towards sort of the feature completeness for Flux, our new language. We’re learning a lot of things about running Flux within the cloud environment and getting feedback from community members on that and there’s probably a handful of things that we’d like to finish there before we’re gonna call that sort of release one of Flux.

Second is, we really need to create migration tooling, and there’s two things that we’re doing in that regard. Obviously, folks that want to land and store their data for longer periods of time will want to move their data from InfluxDB 1 to InfluxDB 2. And we’ve created sort of a new model inside of InfluxDB 2, which includes the notion of a bucket, which is where you place your data, and how to secure that bucket. And so, there are some migration tools that are required to bulk move that data across.

And then last is, I mentioned, we’ve introduced a new primary language for working with data inside of InfluxDB 2, and it unifies the ability to both query that data for dashboard creation and report generation and these other use cases that people might build on top, but it also is the same language that’s used for tasks and sort of creating batch operations.

And that was certainly not the case between InfluxDB and Kapacitor, let’s say. Kapacitor used something called TICK Script and Influx was using InfluxQL. And InfluxQL is a SQL like language and it was an easy onramp for gaining access to the data within InfluxDB. And we’re certainly not abandoning that, but what we’re doing is we’re going to create the ability for you to use InfluxQL, but behind the scenes and on the fly, we will transpile that into something that’s executable by the Flux engine, and then return the results to the customer. And this should allow for existing dashboards and queries just to work natively and seamlessly with InfluxDB 2.0 and obviously InfluxDB Cloud 2.0 when we offer it there. And when we reach that point, then we will declare beta, which means we’ll have feature completeness which include those migration tools and then that’s the time for the community really to start piling in and trying it out, and then we’ll be focusing more on performance and sort of user experience changes between the beta declaration and when we declare 2.0 GA.

But in the meantime, we’ll continue to add features to the cloud edition, they will show up in the open source through our alpha cadence. But I would say that those three big things, you know, the migration tooling, InfluxQL support, and just completing Flux as a generally available language in version one is where we’re focused.

Ashley: Okay, very good. Well, certainly a well thought out strategy and you’ve done a lot of work that we’d look forward to seeing in the open source path as well.

You mentioned buckets, remind me to ask—what types of clouds are you offering the Cloud 2.0 service in?

Hall: Yeah, great question, Mitch. So, currently, we’re available in AWS and there’s two regions we’re available in—U.S. West 2, which is up in Oregon, and we just started the beta process in Europe, which is available in Frankfort, and it’s currently using the same code, but we just haven’t turned on the ability for folks in Europe to pay us yet. But we’ll run that beta probably for four to six weeks in Europe before announcing GA there.

In addition, we announced at Google Next earlier this year that we would be available on GCP. And so, our intent is to continue to drive forward. Most likely, that will appear in the U.S. East region, just to sort of fill out a geographic strategy, but we’re very excited about working with the team at Google. Some of my former colleagues from Oracle are now there, which is fun to reconnect with them.

Ashley: Mm-hmm.

Hall: And we’ll see that land hopefully by year end, and then we’ve been spending time with Azure on the Microsoft team, and so, I would look for that in Europe and Cuba.

Ashley: Okay. Very good. I’m really tying back to the very beginning when you told us about your background—I’m really interested in what do you see are the differences in the approach with InfluxData takes with the developer community being such a focus for the company versus a more traditional database company like you worked at at Oracle? Not that they don’t have developer products, they do, but clearly, this is the center of the universe for you. What are some of the things that you’ve learned or maybe differences in the approach you’re taking now?

Hall: I’d say it’s a mix, Mitch. Like, there are definitely some things in the Oracle landscape that were powerful and can apply to the open source community.

So, for example, feature flag items—so, don’t ship a new version of the database with all your latest and greatest and coolest features turned on by default. That really prevents people from moving rapidly from, you know, from older editions to newer editions. So, we try to keep those things turned off and provide highlights to folks and release notes about how to activate them and turn them on. And it is a double-edged sword, because people who are new that come in, you’d like them to use the latest and greatest and best technology and we do try to highlight that. But we also want people to move and eliminate that sort of long tail of support as quickly as possible.

So, a very smooth process. And this sort of came from Oracle—yeah, feature flag those new things, definitely highlight it so they can take advantage. But let the developers—they’re smart. They’ll take advantage of those new features if you just tell them where they live and how to turn them on.

Ashley: Mm-hmm.

Hall: On the flip side, on the community side, you know, listen to the community feedback. And I would say that’s one thing that’s been top of mind, if not totally obvious is, you know, the motivation behind introducing Flux as a new query language was not done lightly. But it really is in response to all of the outstanding query related questions and issues that were opened by our community members. They could not move certain workloads, they couldn’t do certain kinds of functions, they needed more flexibility with the language. And so, Flux is the genesis of listening to all that feedback.

You know, unfortunately, some of those issues have definitely stood out in the community for, you know, now going on two or three years. In some cases, I finally feel like we’re at the point where we can deliver on those requests from our community members.

Ashley: Very good. I feel like we’ve covered a lot of ground on our podcast today, certainly giving our listeners a great feel for what’s unique and new and also how to get started with Cloud 2.0 InfluxDB. I imagine on your website, there’s a pretty easy place to get started?

Hall: Yeah, absolutely. I think it’s the primary call to action on the website today. If you go to InfluxData.com, it’ll pop up and say, “Get started with Cloud 2.0 immediately.” Also, you can get the open source by going to our Downloads page, or you can find us on GitHub.

Ashley: Excellent. Well, Tom—Tim, [Laughter] I knew I would do that—Tim, thank you for being on the podcast today.

Hall: [Laughter] You’re welcome, Mitch.

Ashley: Okay. I owe you one. [Laughter] I’d like to thank my guest, Tim Hall, Vice President, Products, at InfluxData for joining us today—and, of course, thank you, our listeners, for joining us as well. This is Mitch Ashley with devopsy.kinsta.cloud. You’ve listened to another DevOps Chat podcast. Be careful out there.

— Mitchell Ashley