How To Use Machine Learning in Big Data Analytics

Machine learning enables organizations to turn big data into profit-boosting insights. Learn how big data and machine learning are related and how to use them.

Years ago, business owners had to rely on their memory to customize how they served their clientele. When Ms. Jones walked in, a shop owner had to recall what she bought the last time, whether or not she ended up bringing it back, and whether she complained about it during her last visit.

Now, thanks to big data, tons of customer and business data sit at your fingertips. You know where Ms. Jones lives, what she bought over the past 10 years, how much she spent, how often she returns items, and dozens of other metrics. Using machine learning, you can turn this and other data into business-boosting insights. Here’s a breakdown of big data and machine learning and how you can leverage them to power your business.

What are big data and machine learning?

Big data and machine learning are different yet intimately connected.

What is big data?

Big data refers to huge or incredibly complex datasets that may be impossible to leverage without specialized tools. Some businesses never have to deal with big data. For example, if you have a restaurant with three locations producing sales and inventory data, that’s not “big data.”

On the other hand, if that same restaurant adds 10 more locations and a mobile app that enables customers to place orders online, take advantage of loyalty rewards, and chat with a customer service rep via text, you now have a big data situation. The app alone may produce data regarding:

• The meals customers order the most often

• The times of the day customers place orders

• Where customers order food from based on geo-location data

• Where customers live and the buying stats associated with each town

• Sales data from each location

• How customers use their reward points

• Purchasing data during peak and holiday times

These examples merely scratch the surface. This kind of app could generate dozens of datasets. Also, the information would be streaming in on a near-constant basis. That’s big data.

What is machine learning?

Machine learning (ML) refers to using computers to recognize patterns in data. Machine learning does this using algorithms, which are sets of instructions laid out step-by-step. A machine learning model uses the steps in an algorithm to learn patterns. This also includes recognizing when patterns are being broken and learning how to compare patterns to each other.

As a simple example, suppose you want to build a machine learning algorithm to analyze sales data. You have five years’ worth of sales figures. Your goal is to maximize summer profits by figuring out which products you should offer for sale between June and August.

Of course, you could take ML a step further and incorporate your cost of goods sold (COGS) for each product, including shipping, labor, storage, and other data. Then your ML model could recommend not just the products that have the highest summer sales volume, but it can also tell you which ones bring the most net profit.

What is machine learning in big data?

In the context of big data, anytime there may be patterns in data, you can use machine learning to discover them and provide useful insights. Also, you can use ML to make recommendations based on the patterns the algorithms analyze.

How machine learning works with big data

One of the most popular applications of machine learning is self-driving vehicles. The car uses machine learning to decide what to do in relation to data it gathers from its surroundings and other vehicles.

For example, when the cameras inside a self-driving vehicle “see” a stop sign, they can recognize it as such and automatically apply the brakes. The process behind this decision most likely began with a group of data scientists testing multiple machine learning algorithms. At a high level, this takes three steps:

1. Training

To analyze big data, data scientists first use a training set to teach one or more algorithms what they should be looking for.

For example, with a stop sign, the training set would be thousands of images of stop signs. Data engineers would present images of stop signs from different angles, in different lighting, and even with trees or other objects partially blocking them.

At the end of the training phase, the hope is that the algorithm has identified patterns in the shapes and colors of stop signs. In other words, it knows what a stop sign “looks like”—and in different lighting and from a variety of angles.

2. Validating

The validation set is used to figure out how accurate the ML model is using a completely different set of big data. The purpose of the validation phase is to discover ways to fine-tune the ML model.

For instance, suppose the ML model designed to identify stop signs was 95% accurate, and all of the images it got wrong were very dark. The developers could then use another formula that increases the contrast of each image, making important characteristics easier for the ML model to see.

3. Testing

The testing phase involves feeding the ML model more big data that’s completely different from what it saw during the training and validation phases.

For example, to test the stop sign model, the programmers could show the ML model 250,000 images of different kinds of signs, some of which are stop signs. They would then analyze the results to see how accurately the model was able to differentiate stop signs—as well as avoid misidentifying other kinds of signs.

Challenges with machine learning and big data

Two of the most daunting challenges facing data scientists using ML to study big data are inaccuracy and ethical dilemmas.

1. Inaccuracy

Naturally, even with advanced computational processes involved, you’ll still go through an element of trial and error anytime you use machine learning in big data analytics. This is because you never know which factors could skew your results as you train, validate, and test your model.

For instance, when identifying images—such as stop signs or human faces—multiple factors could contribute to poor performance in your ML model. For example, suppose you’re developing a machine learning model to improve your company’s security system. Specifically, you want a model that can identify the faces of executives and other high-ranking people so they can be granted access to sensitive areas of the building. During the validation phase, the system is only about 65% accurate. But this could be due to several variables, such as:

• Pixelated images of faces

• Images that are out of focus

• The person looking away during the facial scan

• The individual deciding to wear sunglasses, a face mask, scarf, or something else that could skew the identification results

2. Ethical dilemmas

There are also ethical challenges. For example, suppose an HR department uses machine learning to identify the most qualified candidates, pulling them out of a digital stack of 1,500 resumes.

If the ML model was trained using companies and hiring departments run only by men, the data may include bias. Some men may be more inclined to hire other males—for reasons other than their merits or qualifications. Therefore, the “successful” candidate the engineers trained the ML model to look for may, in most cases, be male. As a result, the model recommends men instead of women who may have been more qualified.

In a business context, how are big data and machine learning related?

In a business context, machine learning uses the big data your organization produces to improve or automate business-critical processes and enhance security and safety. The potential applications are literally endless—and as diverse as the different kinds of data you produce.

For instance, a factory or production facility could use machine learning to optimize temperature and humidity levels for its factory floor. For example, machine learning models can figure out:

• The temperature and humidity levels that maximize employee productivity while minimizing the number of unplanned breaks they have to take

• The ideal temp and humidity levels for sensitive equipment that could deteriorate faster given the wrong conditions

• The most cost-efficient temp and humidity conditions, given the expense of running HVAC systems and dehumidifiers

The system could then be used to automatically control your atmospheric system to achieve optimal results.

How are machine learning and big data analytics used in marketing?

Marketing offers some of the most promising applications of machine learning and big data analytics. Consider the following real-life example.

Harley Davidson’s Albert boosts leads by 2,930%

Harley Davidson built a robot named Albert that uses machine learning to make marketing decisions^[1]. This is how Albert helped Harley Davidson’s execs ride off into a brighter sunset.

Harley Davidson wanted to leverage their existing relationships with previous customers. They used Albert to analyze:

• How often people made purchases

• How much these customers spent

• How much time customers spent browsing Harley Davidson’s website

Albert then used this data to separate the customers into different segments. The marketing team then created test campaigns for each category of customers. After testing the success of the campaign, the team scaled it up to involve a wide swath of previous customers.

As a result, Harley Davidson increased its sales by 40%. They also generated 2,930% more leads. Half of those leads were directly identified by Albert him [it?] self. Albert studied the profiles of leads that were very likely to convert to paying customers and then studied the data profiles of other users and pinpointed “lookalikes,” or people who have a lot in common with the high-converting customers.

Whether you’re trying to figure out what Ms. Jones will buy next or optimizing the efficiency of a complex production facility, machine learning can turn seemingly random big data into transformational insights. With a little brainstorming and creative thinking, you can find ways to use ML and big data to outpace the competition and bring your organization to the next level.