The world is expected to produce 463 exabytes (463 billion gigabytes) of data every day by 2025. Just 5% of the world’s leading companies are using a fraction of that data to transform our lives.
We use big data and AI every day. It’s becoming significantly cheaper to run AI and store data through cloud services.
It’s no longer the domain of cutting-edge IT labs or industry giants. Numerous consumer products are built on it. Map apps give us directions, telling us which roads are congested and suggesting alternative routes, or where the best restaurants in our area are. Fitness apps tell us how far we’ve walked and how many calories we’ve burnt. Facial recognition apps unlock our smartphones and allow us to stroll through airport security as trusted passengers.
Everyone recognises the potential in big data and AI. But is it right for your business? And if so, how do you use it effectively?
This article, designed to complement the connections webinar, will help you decide if AI and big data analysis are right for your business and, if so, what you need to do to use it effectively.
What Big Data is, and where to source it
Big Data, in business terms, refers both to a large volume of data and to what you do with it.
Big Data is too large and complex, and often too unstructured, to be processed by ordinary methods. For this reason, it is usually processed by artificial intelligence that adapts and improves its analytical capabilities through machine learning, enabling it to spot trends and anomalies quickly enough to be useful in making better, more informed, and more timely business decisions.
Data comes in many different forms – not just text or binary columns; a passage of prose, a novel or textbook; photographs, voice recordings, videos. Big data covers everything.
In the broadest terms, data can be divided into two broad categories: internal data, from your own organisations' databases; and external data originating from outside your organisation.
Structured and unstructured data
We’ll discuss why cleaning data is important later, but for now, it’s worth noting that structured data is easier to prepare, semi-structured data somewhat harder, and unstructured data requires the most time and effort.
Internal data
Every organisation generates and collects a wide range of internal data; much of it is unused outside of very specific purposes. The advantage of internal data is that it is readily available and there are few constraints in using it. However, because it’s generated in a variety of ways and processed by different departments, it will almost certainly need cleaning and restructuring for AI analysis.
There are three primary types of internal data an enterprise is likely to use.
User-generated data – data created by your staff and customers. These can provide insights into customer behavior, likes, and dislikes. You should log your customers’ recorded preferences and interactions with your chatbots and human agents, which can provide valuable data points.
Machine data comes from sensors installed in industrial equipment, weblogs or cookies tracking consumer behavior online, smart meters and the like. As the Internet of Things grows, the amount of machine data available is likely to grow significantly.
Transactional data comes from invoices, payment orders, storage records, delivery receipts, and the like. These likely come from an organisation’s own databases and should not be overlooked as a valuable source of data to analyse.
External data
External data comes in several broad groups, each with its own challenges.
Open-source data may come from governments, NGOs or companies with open data platforms. It is usually free to access but is often unstructured or semi-structured.
Professional data providers may supply paid data at a flat cost or through a subscription. This will usually be structured in a way that makes it easier to prepare for analysis.
Shared data comes from partners. It may be their own internal data, shared with your organisation through a bilateral partnership or an intermediary. Depending on your agreement there may be fees to access it. It may be structured or unstructured.
Social media data is generated by users on social media platforms. It may be harvested directly through official access points or web-crawling social media sites. It is usually freely available, but it may be subject to copyright and care should be taken to avoid harvesting personally identifiable data. Social media data is usually unstructured.
Why Big Data needs Artificial Intelligence
(and vice versa)
AI and Big Data complement each other – you could even say they need each other. Big Data is the input; AI is the process by which it can be analysed and made useful.
AI requires large amounts of data to learn. The more data it gets, the better it gets – as Dr Awadallah puts is, our brains need food to grow, and data is the food for an AI’s brain.
To start the process, the AI needs representative, clean and structured data and human supervision to prime it. As its capabilities grow and it becomes more capable of analysing data on its own, with human supervision to validate its answers; at this stage, incorrect answers are as valuable for the training process as correct ones. As the algorithms develop they get better at analysing data and spotting trends and patterns until the AI is ready to do its analysis with little or no human intervention.
In this way, training an AI is a little like training a human, from child to expert adult. But at the end of the process, the AI can perform its analysis faster and more accurately than any human. It doesn’t need breaks, and it doesn’t get tired. This allows it to process far more information, with a consistently high quality of analysis.
The results of this rapid analysis allow better human decisions more quickly. As Dr. Awadallah says, it turns average humans into superhumans.
How machine learning works
What we usually refer to as AI is a form of machine learning. There are several types of machine learning, all of which automatically become more accurate by repetitively performing their function and being validated for accuracy. Machine learning explores data and identifies patterns. The more data you have for analysis and testing, the better the algorithms learn.
Machine learning is generally broken down into supervised machine learning and unsupervised machine learning.
In supervised learning the algorithms develop a predictive model based on both input data and the output is tested against expected results. The algorithms become more accurate as training progresses.
Supervised learning algorithms are used for chatbots, image recognition, speech recognition, product recommendations, translation, and autonomous driving.
Unsupervised learning algorithms analyse data structures to generate models when input isn’t labeled, and the results aren’t known.
Examples of unsupervised learning are cluster recognition (used in market research, social network analysis, and scientific analysis) and anomaly detection (used in fraud prevention and other applications).
Using AI and Big Data in your organisation
Data analysis and AI are not miracles or cure-alls. They’re best suited to solving specific, recurring problems. You need to determine your use case.
The first step in using Big Data and AI, therefore, is to decide exactly what problem you want it to solve, and whether AI and Big Data analysis is a suitable solution. Problems should be specific. You must then examine what data you may bring to bear.
In the webinar, Dr Awadallah gives the example of a neonatal ward that found certain expert nurses were able to determine why a premature baby was crying – such as whether it was hungry, needed changing or needed medical intervention - while others could not. The ward wanted to use AI to let all the nurses know what the expert nurses could tell, and was able to use data from monitors, combined with the expert nurses’ decisions, to build an algorithm to do so with a high degree of accuracy.
It’s a good example of applying supervised machine learning to a specific problem with a set of structured and unstructured data and a known outcome from the expert nurses to validate the AI’s output.
You should also consider what challenges you may face in applying big data analysis.
Some firms overestimate the maturity of their data infrastructure, and not everyone in the organisation is ready to implement it – such people may concentrate on actions at the expense of logging the records necessary to make the data analysis effective.
The solution is to educate your staff before implementing Big Data and AI. For some, this will involve getting senior managers to buy into it, for others training lower-level employees. Perhaps you must act as a Big Data ambassador. Ensuring that everyone understands what the introduction of big data analysis is intended to achieve, and their part in it, will make the path to implementation far smoother.
Another strategic challenge is the speed and effectiveness of your decision-making – are your procedures fast enough to take advantage of real-time analysis? Are you prepared to trust the results of the analysis and act on them?
Technological challenges include the volume of data you intend to analyse and the type of data. Analysing a limited set of data in numerical format might be relatively straightforward; analysing a combination of numerical, textual, voice and video data from multiple sources is more challenging, but still possible. Which you do may influence the type of hardware and software that you require and its cost. Cloud options – which are the easiest way forward – each have their specialties.
It’s a good idea here to develop your own use case in conjunction with experts, such as those with Unifonic, who can guide you through the options to find the best solution.
Finally, there are operational challenges. Do you have sufficient data analysts or the right data analysts? That’s a challenge that cloud solutions can certainly help with. What about data security – not only in protecting the data that you’ve gathered to analyse but in the security of any external data you’re analysing?
As with any business process, proper planning and preparation minimises the challenges and paves the way for successful implementation.
Use cases
Because of the wide range of specific uses, these examples focus on customer-facing use cases in general terms. There are many other examples of AI use to improve efficiency in different sectors, some of which are discussed in the webinar.
Effectively used AI and big data analysis can reduce costs and improve efficiency.
Marketing
AI-powered marketing analytics can help you identify customer groups more accurately, identifying loyal customers, and retargeting customers who have expressed an interest but failed to complete. It can track media coverage, providing insights into what’s driving engagement and revenue.
AI can help identify companies understand customers’ habits and interests better, allowing personalised marketing.
AI can help identify the contexts in which your marketing materials appear, allowing you to develop context-sensitive campaigns.
Analysis can spot trending keywords in queries by potential customers, enabling you to ensure your marketing addresses the issues potential customers care about.
Sales
AI analysis can forecast likely sales with greater accuracy, allowing you to prioritise sales reps’ activity based on lead scores, leading to a higher closure rate.
AI, chatbots can field initial contact by potential customers, passing over to a human rep when they can’t answer queries.
AI can give sales reps response suggestions based on detailed analysis of customer tone and queries.
Customer Engagement
Natural language chatbots are one of the most prominent uses of AI in customer engagement. As well as responding to customers, chatbots collect information from customers to help themselves improve and to help companies better understand their customers.
As with sales uses, AI can recommend particular responses to human agents.
Analytics can reveal insights that help you improve customer engagement and satisfaction by spotting patterns.
Business development
AI can help you spot market trends and visualise large data sets more easily, helping you spot opportunities for expansion.
unifonic provides AI-compatible cloud-based communications solutions. find out more about integrating AI with unifonic’s omnichannel platform.
