devopsy.kinsta.cloud

Tag: data scientist

The New AI Overseer Role in DevOps

Robot overlords are the stuff of science fiction fantasy. Today, with more teams implementing artificial intelligence (AI), DevOps teams need a hard, continuous look at reality. An AI overseer watching apps at work could make the difference between sadness and euphoria in your efforts.

We’re seeing AI misses more and more often. Facebook recently admitted its algorithms failed to stop uploads of a particularly heinous live-streamed video. Autonomous vehicles have struck cars and people. Jaywalkers in China have been misidentified and publicly shamed in error. It’s not for lack of effort in development. The adage “If it were easy, anyone could do it” applies.

A few weeks back, I was in a Twitter chat session on AI led by Kirk Borne, data scientist at Booz Allen Hamilton. The question came up of what organizations should consider when creating guidelines for use of data and bias-free AI. Here’s my response:

A4: DevOps teams should have someone whose role is AI oversight—actually watching what it’s doing and constantly testing/validating its findings #MITSMRChat

— Don Dingee (@StratisetDon) February 14, 2019

Common Problems

Most fails are due to fire-and-forget, wherein AI systems are trained, tested against training and turned loose. Even a well-designed AI system is subject to several problems:

Errors: This is where AI makes a factually incorrect decision. False positives and false negatives are in this category, along with miscategorization problems.

Gaps: There are situations in which AI understands conflicting options and is unable to choose. Then, there are situations in which the AI doesn’t recognize what it sees.

Bias: Maybe the system prefers green M&Ms or German shepherds because there was some subtle trend in the training data for “candy” or “dog.”

Ethics: This can be a higher level of bias, where preference is given based on age, gender or race. Sometimes it’s the classic no-win scenario in which both options are harmful.

An AI overseer guards against these types of problems in near real-time, every day, for as long as it takes to weed them out. Some problems are easier than others. An example of a more difficult problem came when autonomous vehicle researchers realized that not all traffic signs are brand new and well-lit. Shade, branches, wind, scratches, dings, bullet holes, stickers, graffiti and many more types of defects exist in the real world.

Increased Complexity

Inevitably, part of the solution the AI overseer leads is better scenario-planning and more training data. Retraining becomes a way of life and at some point, bigger data sets also lead to more complex implementations and another challenge: speed—after retraining to correct issues, the reloaded real-time classification engine runs like it’s stuck in molasses, producing correct results too late to be useful.

While the A system is running, a B version should be in development with new training and classifier boxes. For very large data sets, teams should consider designing a workload-optimized training server complex instead of relying on elastic cloud resources. This was behind NVIDIA’s recent acquisition of Mellanox—using InfiniBand for more bandwidth between GPU clusters in AI server complexes.

Where training systems often scale, real-time systems usually don’t. They are optimized for size and power consumption and can only accept so many additions. When a bottleneck emerges, it can kill the entire application. New generations of CPUs, GPUs and FPGAs keep pushing the envelope for designers tackling AI challenges.

Human-in-the-Loop

In many use cases, the job of the AI system is not to make every single decision autonomously. Instead, the system should make routine decisions and lead the AI overseer to look at exception cases. Directing humans quickly to areas needing attention is often more valuable than risking bad decisions. Human-in-the-loop is something DevOps teams should architect into systems more often.

This may all sound obvious, but it points out a big difference in developing AI versus developing code-based algorithms. If there’s a bug in code, you can debug it and find the offending lines of code to fix. But debugging AI is a lot more difficult. Digital twin implementations come into play using playback of real-work inputs to look at what happens inside the classifier black box. There are usually hints of problems developing before catastrophe strikes. Running data both good and bad can help spot issues.

Ideally, the AI overseer is a hybrid of a data scientist plus a systems architect. Look for someone who understands both data sets and hardware implementations, at least at a high level. They may have to call in other resources on the team for detailed investigations.

The last thing you want after all the hard work in developing a system is to have it run amok months or years later. It’s just Murphy’s Law. AI systems will do something entirely unexpected at the worst possible time. An AI overseer can help a DevOps team get ahead of AI problems.

March 27, 2019
NASA/JPL Tech-Incubation CEO Talks Artificial Intelligence, Part II

This is the second installment of a two-part Q&A series on artificial intelligence (AI) with AJ Abdallat. It covers AI’s potential significance to computing; how AI affects jobs; the employee specializations needed to fuel AI; big data; and cultural challenges in the enterprise. Be sure to check out Part 1 as well.

AJ Abdallat is a serial entrepreneur who, from 1998 to 2011, worked with NASA’s Jet Propulsion Labs (JPL) and Caltech on a series of tech incubation startups. The goal was to commercialize NASA technologies, many of which were artificial intelligence-related.

In 2014, Abdallat co-founded and became CEO of Beyond Limits, a startup that secured licenses to NASA AI technologies for use in commercial applications. Beyond Limits currently holds exclusive licenses to 42 blocks of sophisticated intellectual property developed through NASA R&D investment, a $150 million AI technology head start under the aegis of Caltech’s Office of Technology Transfer and Corporate Partnerships program. Abdallat’s company is a leader in cognitive technology, which goes beyond conventional AI, binding deep learning and machine learning tools with symbolic AIs to emulate human intuition.

We interviewed Abdallat while researching the story, “AI Delivering on the Business Analytics Promise.” With AI poised to potentially revolutionize business, Abdallat’s career, which blends AI science and commercialization of the technology, makes his insights fascinating.

devopsy.kinsta.cloud: How does the advent of AI as used by enterprises compare to earlier significant computing milestones like networking, the internet, or cloud computing?

Abdallat: Part of me wants to quash as much artificial intelligence speculation as possible. The hyperbole. But frankly, I think AI is potentially more transformative than the other things you mentioned, which are just infrastructure. AI is starting to help us pose the questions that we ultimately want to answer. What’s unique about AI is that it’s an introspective exercise. Every new success for AI is giving us some insight about how we work.

AJ Abdallat is co-founder and CEO, Beyond Limits, an artificial intelligence firm. He’s an engineer and entrepreneur.

As we integrate AI, the trend may not be that transformative initially. Improvements will likely be focused on doing it faster, bigger with lower downtime and more efficiency. But what comes next is thinking about all the humans involved in that process, and what they touch and how they work. I think once AI starts to integrate with that activity, and act at that level within the enterprise, this will truly change how we think about what a business is meant to do. I’m a little biased in this, but you might go so far as to say we are entering the era of the cognitive corporation, where man and machine at all levels of the enterprise will operate in a synergistic fashion to solve problems, each learning from the other.

If you think about most business processes and most problems that humans find interesting to solve, they are heterogeneous. They move through a pipeline of different specialists that look at separate pieces or provide specific oversight to balance specialized objectives. If I can couple my engineering and sales activities at the enterprise level, and use AI to mediate the trade-offs of both of those objectives, and I have these people processes synchronized with machine processes, it gives me the ability to learn how these two potentially juxtaposed business ends are coupled, where they’re divergent, and what kind of switches I can flick or knobs I can turn to make the net positive of these two forces all the greater.

devopsy.kinsta.cloud: Is artificial intelligence intended to replace humans or help them do their jobs?

Abdallat: In some places, as with any kind of automation, you will see some jobs go away. But the labor that was there before finds a new role working with AI, and I think those new roles, especially at the level of knowledge work, will be very interesting.

As good as artificial intelligence systems are, they’re not going to set our objectives. They can move toward them, and think outside the box, which is one of things that we work hard to do at Beyond Limits. But being able to deal with the nuances of problems is still something that humans excel at. So, finding the things that are worth teaching to AI, to learn and to figure out, that’ll still always be a job in the hands of humans. I imagine that’s also when it gets kind of cyclical. I’m going to learn something, the machine’s going to discover a new pattern, we’ll optimize an approach that I didn’t think could be optimized to such a degree. That’s going to give me new questions, which makes me give it new training data, and the process continues like that.

devopsy.kinsta.cloud: What types of software engineers are now or soon will be in demand to support AI, machine learning and this new industry?

Abdallat: One of the things we’re starting to see is a change in job definition. You probably hear about data scientists and machine learning scientists. What we’re going to see more of is taking that science and flipping it with engineer. There’s a ton of stuff in the laboratory that works and is great. And scientists can write papers and come up with nice coefficients for polynomials that solve problems. But at the end of the day, there’s an infrastructure that makes all this work. So, we will probably begin to see new job titles emerge, like data engineer. A data engineer is maybe the bridge between the work that the data scientist does and the infrastructure in which his data lives. He or she is much more of a software engineer but has some knowledge of the data science workflow and algorithms germane to data and AI work.

Data engineers are becoming more important, and we’re finding they fit somewhere between IT and the business. I think that’s important. It’s the same with machine learning people. For example, the actuary of the future will simply be a data scientist who works in insurance. Systems analysts will gradually become systems data scientists. Of course, to make all this stuff work securely, we’ll probably see a ballooning of the specialized security people to support the work that goes into the systems engineering. Unfortunately, the world is more adversarial than we would like.

devopsy.kinsta.cloud: Does AI present a cultural challenge to enterprises?

Abdallat: Earlier big data solutions and the fact that AI looks promising for big data have paved the way. AI asks for similar machineries, although it can be used in smaller footprints when available. Culturally, some IT organizations are more progressive than others. Some like to play with new toys and would be stoked to get this down, assuming they have the budget, and AI may be enough to motivate them to allocate budget.

In other cases, there may be entrenched opposition to change. But it’s such an exciting field that whenever you start talking about artificial intelligence, you’re going to get buy-in. Some healthy skepticism is a good thing too. You can try to channel it into constructive ways of alleviating the concerns that people bring up, and assess whether they’re just nay-saying the next new thing or they’re aware of a valid institutional or physical barrier to success.

— Scot Finnie

January 25, 2018
Understanding the Data Scientist’s Role in Cross-Functional Teams

In our previous article, “Data Science Industry Perspectives in the Cloud,” we discussed that evolution is key if you plan to grow your business. You can start with ready-made solutions, then, in time, you can switch to in-house solutions created with the help of a group of data scientists.

This time around, we’ll talk about the cases and solutions when machine learning as a service (MLasaS) doesn’t work. Your company’s first move is not to go out and hire a data scientist to solve your urgent business needs; rather, your company should invest in a custom-made solution. Only when you have a workable solution can you dive in deeper and create a proper team that can create an in-house data science solution.

Custom Made End-to-End Solution as a Start

Data Science in the classical sense involves data, goals and models to solve a pressing issue. The best way to jump start a data science process is by putting together various ready-made services into a single workable product to show your customers clear-cut results in short order. This can be done without any complex or global research, and you can comfortably formulate specifications taking into account all the feedback from your customers to create a more high-level data science offering later.

One of the biggest issues for any data science project is in formulating the specifications for it. The usual request is, “Create something for my company using data science. Analyze data for me.” This type of a job has lots of trial and error. Having a custom end-to-end solution from the start lets you add on extra services as needed, and enables greater insights and predictions into a specific business workflow.

I believe companies should build their data science system not from the data science research but from the point of view of an end-to-end solution. And then, bit by bit, they can adjust their services in the process.

Moving on to Data Science Research

Companies start data science research by employing a data scientist who matches their needs. You can use the standard data scientist qualifications: This employee should have a firm footing in the application environment, mathematics and programming. Also, their business skills are key to success—I believe an understanding of the topical area is key here because we are solving business issues. And a data scientist who solves problems that are more academic will be more focused on winning a Kaggle competition than on addressing the business needs.

It is important this person understands the product development cycle. This way, the models and analyses data will be built in such a way that they could be deployed in production. For example, if a person is using R (the language and framework for data scientists), we should note that it is more research-oriented and it is not production-ready. Correspondingly, the results of such research cannot be deployed in any end-to-end solution. Therefore, the data scientist should work with a programmer.

Although in the above diagram we see that the Data Scientist should have hacking skills, in reality, this is not the case. In their vast majority, Data Scientists are not able to write quality code. And if you need not just the research but a solution then process-wise you need to have a Data Scientist working together with a Data Engineer.

CAP Theory as an Analogy

It is impossible to have three database properties at once: consistency, availability and partition-tolerance. This basic rule is known to all developers. This is true for a data scientist as well: People envision a data scientist overlapping all three properties, but in real life, you cannot find such an ideal candidate. Usually, people tend to lean one or another way in their work and keeping a balance is not always a priority.

In principle, data scientist should have business insight, understand the math behind it and work in tandem with a skilled developer. Of course, a proper data scientist should be able to write any semblance of code, but a data scientist should work with a data engineer to write and realize code, being responsible for both the quality and sustainability of this solution.

The classic tragedy of a company that decides to initiate any data science research is when a data scientist says, “I’ve got 40,000 lines of Python code on my PC. Can you make it work in production?” And, of course, this is virtually impossible to do. When that happens, all the research is simply wasted.

Cross-Functional Teams

Any team that deals with data science has to be cross-functional. In other words, it has to cover a whole stack of the solutions it writes. In a normal infrastructure there should be a DevOps engineer, data scientist, data engineer and a product developer writing the web app and/or mobile app. This single team is responsible for the result. The team members should work together and solve tasks that are interconnected in their interactions.

All of this means that the whole team is responsible for the business result. This is also true for the transitionary research done by a data scientist that is impossible to use in production on its own.

Old-School vs. Vertical Teams

To dig in deeper, let’s take a classic old-school layered company organization structure when you have a department of data scientists, operations, UI developers, a big data department, QA engineers and so on. In this case, we have every project penetrating most of these teams. And the classic problem is that tickets and tasks are being thrown around by one team to another, and the real business goals are being watered down along the way and not solved in the end. So, instead of this horizontal division, we have divided the teams vertically. This allows us to create teams that see a clear-cut goal they need to achieve. And at the same time, they can improve their cross-skills and boost their responsibility levels.

As a result, such teams began to deliver, Scrum and Agile properly. It is not directly related to data science, but companies need to realize the difference between a data scientist, which works in academia, and a production data scientist. Companies should aim at employing the latter one within their teams, and not let a data scientist work alone remotely.

About the Author / Stepan Pushkarev

Stepan Pushkarev is the Head of DevOps Practice at Squadex.com and CTO of Hydrosphere.io. Co-founded and managed engineering teams for eCommerce, IoT and Ad-tech companies. He has been responsible for the full products stack: math models, infrastructure & operations, enterprise applications as well as hiring, establishing engineering culture and delivery process. Stepan combines strong technology, management skills and entrepreneurial spirit. Connect with him on LinkedIn.

August 31, 2017
Is DevOps Making Life Better for Data Scientists?
The path to success and happiness of the data science team working with big data project is not always clear from the beginning. It depends on the maturity of the underlying platform, their cross skills and DevOps processes around their day-to-day operations.

The past few years have been like a dream come true for those who work in analytics and big data. There is a new career path for platform engineers to learn Hadoop, Scala and Spark, while Java and Python programmers have a chance to move to the big data world. There they find higher salaries and new challenges and get to scale up to distributed systems.

But recently, I have started hearing some complaints and dashed hopes from engineers who have spent time working there.

Their expectations are not always met. Big data engineers spend most of their time in extract, transform and load (ETL), preparing data sets for analysts and coding models developed by data scientists into scripts. They have become cogs between data scientists and big data.

I see the same feedback from data scientists. Real big data projects differ from Kaggle competitions. Even minor tasks require Python and Scala skills as well as multiple interactions and design sessions with big data engineers and operation teams.

DevOps engineers get to go no further than just using the basic tools and running through tutorials. They grow weary of being given nothing more to do than deploying the next Hadoop cluster through the Cloudera wizard and semi-automated Ansible scripts.

In this environment, the big data engineer will never get to work on scalability and storage optimization. Data scientists will never invent a new algorithm. And the DevOps engineer will never put the entire platform on autopilot. The risk is the team’s ideas might not make it beyond proof of concept.

How do we change this situation? There are three key points:
1. Tools evolution—the Apache Spark/Hadoop ecosystem is great. But it is not stable and user-friendly enough to just run and forget. Engineers and data scientists should contribute to existing open-source projects and create new tools to fill the gaps in day-to-day operations.
2. Education and cross skills—when data scientists write code they need to think not just about abstractions but also consider the practical issues of what is possible and what is reasonable. For example, they need to think how long their query will run and whether the data they extract will fit into the storage mechanism they are using.
3. Improve the process—DevOps might be a solution. Here DevOps does not just mean writing Ansible scripts and installing Jenkins—we need DevOps working in optimal fashion to reduce hand-off and invent new tools to give everyone self-service to make them as productive as possible.
The Continuous Analytics Environment

With continuous analytics and self-service, data scientists own the data project from the original idea all the way to production. The more autonomous they are, the more time they can dedicate to producing actual insights.

Data scientists start from the original business idea and work through data exploration and data preparation. Then they move to model development. Next comes deployment and validating the environment. Finally, there is the push to production. Given the proper tools, they could run this complete iteration multiple times a day without relying on the big data engineer.

Big Data engineers, in turn, work on scalability and storage optimization, developing and contributing to tools such as Spark, enabling streaming architectures and so on. They provide an API and a DSL for the data scientist.

Product engineers should be given analytical models developed by data scientists as neatly bundled services. Then they can build smart applications for business users to allow them to use those in self-service.

Working like this, no one is stuck waiting on the next person. Each person is working with their own abstractions.

This might sound like a utopia. But I believe this could motivate big data teams to improve their existing big data platforms and processes, with the added benefit of making everyone happy.

About the Author / Stepan Pushkarev

Stepan Pushkarev is the Head of DevOps Practice at Squadex.com and CTO of Hydrosphere.io. Co-founded and managed engineering teams for eCommerce, IoT and Ad-tech companies. He has been responsible for the full products stack: math models, infrastructure & operations, enterprise applications as well as hiring, establishing engineering culture and delivery process. Stepan combines strong technology, management skills and entrepreneurial spirit. Connect with him on LinkedIn.
June 2, 2017
Why You Need to Embrace Automation

Just in case you hadn’t noticed: The robots are coming.

Well, you probably had noticed. It’s hard to miss. Newspaper articles and blogs everywhere herald their arrival. The stories usually follow the same general formula: A catalogue of amazing things that robots (really, AI systems) can do. Debate about forced unemployment vs. people freed up to do more interesting work. And then some comments about whether it’s for the good or bad. You can generally infer the general tone from the choice of image accompanying the story: a cute Japanese robot with a plastic smile or a more sinister face with the Terminator’s jawline.

Of course, workplace automation has been around for almost as long as we have had workplaces. Each new wave of technology disrupts the employment landscape in its own particular way. However, this time it’s serious: It’s not just manual labor that’s under threat, but increasingly, we’re told, white-collar workers are being supplanted by machines. A recent McKinsey report has generated a wave of headlines: “Even the CEO’s job is susceptible to automation,” wrote Patrick Nelson at Network World; and Ana Swanson at the Washington Post reported, “How robots will even affect the jobs of people we thought were immune.”

Not surprisingly, a closer read of the McKinsey material gives a more nuanced appraisal than a headline can offer. Few of us will actually be replaced by a robot, but many workers will lose some part of their current jobs to automation. The McKinsey report states that up to 45 percent of the activities that employees perform today could be automated, but very few occupations will be automated entirely in the near or medium term. The things that require creativity and emotion remain uniquely in the domain of humans, and they are likely to remain there for quite some time.

It’s against this background that we see a steady increase in what is often referred to as robot journalism: machines writing news stories. So far, these applications rely on scenarios where the general ‘shape’ of a story remains pretty consistent because of the regularity of the data underlying those stories. For example, a report on stock performance is always going to mention the current stock price, what the stock price was at some particular point in the past, the price highs and lows, and perhaps something on volume or analyst sentiment. These reports often are found in the back pages of a newspaper, and are the stories that few journalists actually want to write, since they allow almost no space for creativity and emotion.

The reality is that today’s news-writing software is still a long way from replacing human journalists. Which is not to say that what a robot journalist produces isn’t useful; it surely is. Last month I spoke about robot journalism at the news:rewired conference. It was fascinating to see how most of the major news media are exploring how to get robots on staff. So while we’re going to see more robot journalists, it doesn’t necessarily mean fewer human journalists. A recent study by researchers at Oxford University and Deloitte explored how susceptible various jobs were to automation—journalists ranked 285th out of 366 jobs, with an ‘automation risk’ of just 8 percent.

But if you really want to see where the action is regarding automated writing, just look at it from a different angle. Rather than Natural Language Generation displacing people from their jobs, it’s offering a solution to the problem of shortage: A shortage of workers or expertise. And that story isn’t about news; it’s the Big Data story. You can take issue with the astounding numbers that are thrown around, but even if they’re out by a factor of 10, they’re still significant: by 2020, about 1.7MB of new information will be created every second for each person on earth. But less than 0.5 percent of all data is ever analyzed.

In 2011, McKinsey noted, “The United States alone faced a shortage of 140,000 to 190,000 people with analytical expertise, and 1.5 million managers and analysts with the skills to understand and make decisions based on the analysis of big data.” And subsequent reporting points to massive year-on-year leaps in demand.

Regardless of how few or how many people are displaced by robots, they are not all going to become data scientists. The only way this volume of data will be analyzed and explained is if we have machines doing it for us. And that’s the real calling and promise of Natural Language Generation (NLG). Interactive visualizations and dashboards don’t cut it; the meaning in all that data will be lost unless we call in the robots. But you need more than robot journalists, tirelessly churning out one formulaic story after another. You need articulate robots that use sophisticated NLG to construct fluent narratives from idiosyncratic data sets. So, accept the data reality, embrace automation, and find an NLG robot to explain the insights buried in your data.

About the Author:

Dr. Robert Dale, Chief Technology Officer and Chief Strategy Scientist at Arria NLG, is recognized as one of the world’s foremost experts in Natural Language Generation (NLG) research and Development, having authored or edited seven books and 160 papers on computational linguistics. He was a Professor in the Department of Computing, Director of the Centre for Language Technology at Macquarie University. He co-authored the seminal textbook “Building Natural Language Generation Systems” with Arria NLG Chief Scientist and co-founder Professor Ehud Reiter.

January 6, 2016