MACHINE LEARNING WITH PYTHON

The Definitive Tool to Improve Your Python Programming and Deep Learning to Take You to The Next Level of Coding and Algorithms Optimization.

© COPYRIGHT 2021 – ALL RIGHTS RESERVED

The content contained within this book may not be reproduced, duplicated or transmitted without direct written permission from the author or the publisher.

Under no circumstances will any blame or legal responsibility be held against the publisher, or author, for any damages, reparation, or monetary loss due to information contained within this book. Either directly or indirectly.

Legal notice

This book is copyright protected. This book is only for personal use. Y ou cannot amend, distribute, sell, use, quote or paraphrase any part, or the content within this book, without the consent of the author or publisher.

Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and nobody knows why.

( Albert Einstein )

Table of Contents

Introductio n               5

Chapter 1. What Is Machine Learning ?               11

Chapter 2. Giving the Computers the Ability to Learn From Dat a               17

Chapter 3. Basic Terminology and Notation s               23

Chapter 4. Evaluating Models and Predicting Unseen Data Instance s               30

Chapter 5. Building Good Training Dataset s               36

Chapter 6. Combining Different Models for Ensemble Learnin g               48

Chapter 7. Applying Machine Learning to Sentiments Analysi s               51

Chapter 8. Conditional or Decisional Statement s               57

Chapter 9. Function s               63

Chapter 10. Actual Machine Learning Algorithm s               70

Chapter 11. Applications of the Machine Learning Technolog y               75

Chapter 12. Data Mining and Application s               82

Conclusion

Introduction

Machine learning is a computer program that will learn without being explicitly programmed.

Example:

You train a computer to recognise cats and dogs in images. You give examples of images of cats and dogs. You tell the computer that the cats are on the left side of the images and the dogs are on the right side.

Once you've done that, the computer creates some rules that distinguish cats and dogs.

Tests, where the computer can tell the difference between cats and dogs and can't tell the difference in other images, will be run. These tests will show that the computer is learning and its new rules are better than the ones it had at the start.

What Happens When you Try to Teach a Machine to Do Mathematics?

After a few years of training the machine, the only things the computer can do reliably are some elementary things like creating the rules of the game of chess. He best methods that we can follow and focus our time on when it comes to Machine Learning include:

Logistic Regression

Uses a neural network to produce a complex logistic regression system.

Decision Trees

Uses a set of rules to identify a specific tree that classifies the data into a pre-defined set of classes, e.g. a tree that classifies X people into 2 groups, e.g. either black or white.

Naive Bayes

Artificial Neural Networks

Run on a computer, they have a set of nodes and a list of weights for each node which is stored and fed into the system along with the data. The nodes are connected to each other in some way, which is held in the model, and it uses that to see if it falls into a certain category or perspective. This approach often produces more non-linear results than logistic regression systems.

Co-Regression Models

A set of predictive models that can be used to build generalizations of data sets.

Clustering Algorithms

Data can be categorized into groups, e.g. computers can be grouped into departments, each department having a group of students. An algorithm, such as k-means clustering, uses the data to create the groups that result in the best performance.

Why Python and Data Science?

Python is a very powerful and easy-to-use programming language that can be used for many things. It is easy to move on to other things if you get stuck.

Is it Possible to Apply Machine Learning to Aspects of Mathematics?

Yes. Mathematics involves the manipulation of symbols (like numbers and letters). Data sets can be manipulated. Computers can manipulate data.

Can MATLAB in a Computer Do Machine Learning?

No. MATLAB is a very limited programming language, and it is only suitable for very specific things. People use MATLAB because they want to do those things it is good at.

Is it Possible to Use MATLAB in a Computer to Do Machine Learning?

Yes. It is possible to install MATLAB on a computer, install some libraries and modules to use MATLAB to do Machine Learning tasks, or install a C++ compiler so that the MATLAB program can be modified to do Machine Learning tasks.

How Do You Teach a Computer to Do Machine Learning?

There are many ways to teach a computer to do Machine Learning, and the best way will largely depend on the problem that needs to be solved. As a rule, a computer will need lots of examples of something for it to learn from. In Machine Learning, a computer will need examples of the data it is trying to classify and to predict. The best data set is from a data set that is at least as large as the data set that the computer will be predicting from.

Can You Automate Machine Learning?

Yes. Once a computer is taught how to do a Machine Learning task, it is possible to automate that task. Many websites run competitions that run Python scripts that perform Machine Learning tasks. Python on websites is usually written like an entry to a competition. Competition can be run any time there is a problem where a machine is needed to learn a problem and produce an output.

Do Computers Ever Make Mistakes?

Machines are logical and follow the rules they are given. They do not use judgement or language to solve problems. The answers produced are subject to the rules that it has been given.

Who Is to Blame When a Machine Makes a Mistake?

No one can be to blame when a machine makes a mistake. A machine cannot be to blame. A machine can only follow the rules that are given to it.

Relation Between Big Data and Machine Learning (ML)

Big Data is the data too complex and scattered in such a way, that it needs special algorithms and methodologies to process them, so that they can be properly analyzed. The traditional processing system cannot analyze extremely large volumes of data. Machine learning is a refined type of class of algorithms that will process big data. The algorithms like predictive analytics, Text algorithms, Social network mining, etc., play an important role in the Machine Learning process. Machine learning algorithms are based on the algorithms which are capable of analyzing large volumes of data, structured or unstructured. The process to use ML is to prepare the data by cleaning the data to prepare it for the machine learning algorithms, and train the machine learning algorithms to perform the algorithm by making effective combinations of the machine learning algorithms, pre-processing the data to form the input for the machine learning algorithms, passing the output obtained to the Machine Learning algorithms, and training them to perform the function. The ultimate objective is to obtain the best result in the best time, and these are achieved by creating effective algorithms.

Uses of Machine Learning

The applications of Machine Learning include real-time decisions, clinical medicine, fraud detection, search engine results, and oil analysis to name just a few.

Implementing Machine Learning is a very powerful way of making predictions using the algorithms that follow. The algorithms developed have constantly provided better predictions over time so that the performance of the machine learning algorithms is constantly improving.

The predictive power of those algorithms also enables them to be useful in a wide range of areas. The machine learning algorithms will work in all data types including text, images, audio, social media, and financial market data.

Put another way, machine learning techniques are applied to end up with a solution that can reveal meaningful distinctions in data that the naked eye can’t see.

Many different kinds of dependencies (or interrelationships) occur within the data that machine learning algorithms need for their learning. In many examples, this data is made up of large volumes of unstructured textual data. Machine Learning is improved by having all the data that can be used for the learning of the Machine Learning algorithms.

Going forward, Machine learning has been going from strength to strength and it is expected that it will execute tasks that were once only the domain of specialists.

Shared Sensing is an emerging service offered by a mix of established cloud providers and emerging smart city service providers. It refers to the ability for multiple smart city stakeholders to share live information from their networks, resources, and devices for the benefit of the wider community. This will bring numerous benefits, including operational efficiency.

Modern companies recognize the importance of big data for their success, not only to compete with others but also to strengthen their business relation, attract customers. It has already changed the way how the business works, and consumers use the service or buy the products.

For example, Amazon collects data from its customers to provide the most relevant result. It uses the data about what people search for, what they buy, where they live, etc. Some customers may not want their personal information disclosed as they may look like a potential threat to others. However, most customers accept the terms and conditions and agree to the usage of their data by Amazon.

On the other hand, ISPs use the data from customers to produce revenue. Different ISPs have different ways to produce revenue from customers. For example, your web browser is required to use Google ads. ISPs also use the data from customers to provide different services. For example, some ISPs sell your data to market research companies or insurance companies. These data can be used to verify the email address of customers. This increases the reliability of the email address.

The customers also use data in different ways. But, some people use it in an unethical way by sharing the data with others without the knowledge of others. As the number of people using data grows every day, the methods of sharing the data are improved to make it easier to share the data instantly.

Actual Machine Learning Algorithms

We began the process of Machine Learning by creating an algorithm that had the capability of learning. This created the pattern of breaking the problem into a series of smaller problems to be solved on the data which had been obtained.

There are many techniques that can be applied to Machine Learning, and experts use different techniques for different data sets. The process begins with an algorithm that has the capability of learning to a degree.

This enables the system to learn what works and what does not work. With each tiny step taken, the algorithm can learn if it was possible to align these two groups in a way that the machine learning systems could learn from.

Chapter 1. What Is Machine Learning?

The first thing that we need to take a look at here is the basics of Machine Learning. Machine Learning will be one of the applications of artificial intelligence that can provide a system with the ability to learn, all on its own, without the help of a programmer telling the system what to do. The system can even take this a bit further and improve based on its own experience, and none of this is done with the plan being explicitly programmed in the process. The idea of Machine Learning will be done with a focus on the development of programs on the computer that can access any data you have, then use that presented data to learn something new, and how you would like it to behave.

There will be a few different applications that we can look at when using Machine Learning. As we start to explore more about what Machine learning can do, you may notice that throughout the years, it has been able to change and develop into something that programmers are going to enjoy working with more than ever. When you want to make your machine or system do a lot of the work independently, without you having to step in and program every step, then Machine Learning is the right option for you.

When it comes to the world of technology, we will find that Machine Learning is pretty unique and can add to a level of fun to the coding that we do. There are already many companies in various industries (which we will talk about in a bit) that will use Machine learning and are already receiving a ton of benefits.

There are many different applications for using Machine Learning, and it is amazing what all we can do with this kind of artificial intelligence. Some of the best methods that we can follow and focus our time on when it comes to Machine Learning include:

1. Research on Statistics

Machine Learning is already making some headway when it comes to the world of IT. You will find that Machine Learning can help you to go through a ton of complex data, looking for the large and essential patterns in the data. Some of the different applications of Machine Learning under this category will include things like spam filtering, credit cards, and search engines.

2. An Analysis of Big Data

Many companies have spent time collecting what is known as Big Data, and now they have to find a way to sort through and learn from that data in a short amount of time. These companies can use this data to learn more about how customers spend money and even help them make important decisions about the future. If we had someone go through and manually do the work, it would take much too long. But with Machine Learning, we can get it all done. Options like the medical field, election campaigns, and even retail stores have started to turn to Machine Learning to gain some of these benefits.

3. The Financial World

Many financial companies have been able to rely on Machine Learning. Stock trading online, for example, will depend on this kind of work, and we will find that Machine Learning can help with fraud detection, loan approvals, and more.

To help us get going with this one and understand how we can receive the value that we want out of Machine Learning, we have to make sure that we pair the best algorithms with the right processes and tools. If you are using the wrong kind of algorithm to sort through this data, you will get a lot of inaccurate information, and the results will not give you the help you need. Working with the right algorithm, the whole time will make a big difference.

As we are working on some of the models that we want to produce, we will also notice many tools and other processes available for us to work with. We need to make sure that we pick the right one to ensure that the algorithm and the model you are working with will perform the way you would like.

The different tools that are available with Machine learning will include:

1. Comprehensive management and data quality.

2. Automated ensemble evaluation of the model to help see where the best performers will show up.

3. GUIs for helping to build up the models you want and the process flows being built up.

4. Easy deployment of this so that you can quickly get reliable and repeatable results.

5. Interactive exploration of the data and even some visualizations help us view the information easier.

6. A platform that is integrated and end to end to help with the automation of some of the data to decision process that you would like to follow.

7. A tool to compare the different models of Machine learning to help us identify the best one to use quickly and efficiently.

The Benefits of Machine Learning

We also need to take some time to look at a few of the benefits of machine learning. There are many causes why we would want to choose to go with Machine Learning to help our Data Science Project. It is impossible to create some useful algorithms or models that can accurately make predictions out of the data you send through it. There are a lot of other benefits that can come with this as well. Some of the best services that we can see when we decide to work with Machine learning include:

1. Marketing Products Are More Comfortable

When you can reach your customers right where they are looking for you, online and social media, it can increase sales. You can use Machine Learning to figure out what your target audience will respond to, and you can make sure that the products you are releasing work for what the customer wants.

2. Machine Learning Can Help with Accurate Medical Predictions

The medical field is always busy, and it is believed that a lot of the current job openings are going to be left unfilled. Even a regular doctor with no specialties will need to deal with lots of patients throughout the day. Keeping up with all of this can be a hassle. But with the help of Machine learning, we can create a model that can look at images and recognize when something is wrong or not. This can save doctors a lot of time, hassle, and can make them more efficient at their jobs.

This is just one area where Machine learning will be able to help out with the medical field. It can assist with surgeries, take notes for a doctor, look for things in x-rays and other imaging, and even help with front desk operations.

3. Can Make Data Entry Easier

There are times when we need to make sure that all the information is entered into a database efficiently and quickly. If there is a ton of data to sort through and short on time, this can seem like an impossible task. But with Machine Learning and the tools that come with it, we can get it all done in no time.

4. Helps with Spam Detection

Thanks to some of the learning processes that come with Machine Learning, we find that this can prevent spam. Most of the primary email servers right now will use some form of Machine learning to handle spam and keep it away from your regular inbox.

5. Can Improve the Financial World

Machine Learning can come in and work with many different financial world tasks. It helps with detecting fraud, offering new products to customers, approving loans, and so much more.

6. Can Make Manufacturing More Efficient

Those in the manufacturing world can use Machine Learning to help them be more competent and better at their job. It can figure out when things will be slowing the process down and need to be fixed, and it will look at when a piece of a machine is likely to die out, and so much more.

7. It Requires us with a Better Understanding of the Customer

All companies want to know as much about their customers as possible, ensuring that they can learn how to market to these individuals, what products to offer, and which methods they can take to make the customer as happy as possible.

Supervised Machine Learning

The first type of Machine learning algorithm that we will take a look at is supervised Machine learning. This Machine learning type is the kind where someone is going to train the system, and the way they do this is by making sure to provide input, with the corresponding output, to the system to know the right answers. You also have to take the time to furnish the feedback into the system, based on whether the system or the machine was accurate in the predictions that it made.

Unsupervised Machine Learning

Now we can move on to the idea of Unsupervised Machine learning and see how this one is going to work compared to supervised learning. With unsupervised Machine learning, we will find that there will be a big difference when compared to the other methods but can train the system how to behave without all of the examples and labeled data along the way.

With unsupervised learning, the model will not be provided with the output for it to be taught how to behave. This is because the goal of this kind of knowledge is that we want the machine to learn what is there, based on the unknown input. The device can know how to do this all on its own, rather than having the programmer come in and do all of the work on it.

Reinforcement Machine Learning

The third type of Machine learning method that we need to take a look at here will be reinforcement Machine learning. This algorithm type is newer than the other two, and it is going to be the one that we work with any time that the presented algorithm has examples, but these examples are not going to have any labels on them at all.

Chapter 2. Giving the Computers the Ability to Learn From Data

We need a programming language to provide instruction to the machine to execute the code to use machine learning. We will learn the basics of the Python language, how to install and launch python. We are also going to learn some Python syntax and some useful tools to run Python. We also cover some necessary Python libraries that useful for machine learning. First of all, why would we use Python and not another programming language?

Why Use Python for Machine Learning?

Python is a programming language extensively used for many reasons. It is a free and open-source language, which means it is accessible to everybody. Although it is free, it is a community-based language, meaning that it is developed and supported by a community that gathers its effort through the internet to improve the language features.

Other reasons people would use Python are:

1. Quality as a readable language with a simple syntax

2. Program portability to any operating system (e.g., Windows, Unix) without or with little modifications

3. Speed of execution: Python does not need compilation and run faster than similar programming languages

4. Component integration which means that Python can be integrated with other programs, can be called from C and C++ libraries, or call another programming language.

Python comes with basic and powerful standard operations and advanced pre-coded libraries like NumPy for numeric programming. Another advantage of Python is automatic memory management and does not require variable and size declaration. Moreover, Python allows developing different applications such as developing Graphical User Interface (GUI), doing numeric programming, do game programming, database programming, internet scripting, and much more.

How to Get Started with Python?

Python is a scripting language, and like any other programming language, needs an interpreter. The latter is a program that executes other language programs. As its name indicates, it works as an interpreter for computer hardware to execute Python programming instructions. Python comes as a software package and can be downloaded from Python’s website. When installing Python, the interpreter is usually an executable program. If you use UNIX and LUNIX, Python might be already installed, and it probably is in the /usr directory. Now that you have Python installed let’s explore how we can run some necessary code.

To run Python, you can open your operating system’s prompt (on Windows, open a DOS console Window) and type python. If it does not work, you don’t have python in Shell’s path Environment variable. In this case, you should type the full path of the Python executable. On Windows, it should be something similar to C:\Python3.7\python, and in UNIX or LUNIX is installed in the bin folder: /usr/local/bin/python (or /usr/bin/python).

When you launch Python, it provides two lines of information, with the first line is the Python version used as in the example below:

Python 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]: Anaconda, Inc. on win32

Type "help", "copyright", "credits" or "license" for more information.

>>>

Once a session is launched, Python prompts >>>, which means it is ready. It is prepared to run the line codes you write in. The following is an example of a printing statement:

>>> print ('Hello World!')

Hello World!

>>>

When running Python in an interactive session as we did, it displays the results after >>>, as shown in the example. The code is executed interactively. To exit the interactive Python session, type Ctrl-Z on Windows or Ctrl-D on Unix/Linux machine.

Now we learned how to launch Python and run codes in an interactive session. This is a good way to experiment and test codes. However, the code is never saved, and it needs to be typed again to rerun the statement. To store the code, we need to type it in a file called a module. Files that contain Python statements are called modules. These files have an extension ‘.py.’ The module can be executed simply by typing the module name. A text editor like Notepad++ can be consumed to create the module files. For instance, let's create a module named text.py that prints ‘Hello World’ and calculates 3^2. The file should contain the following statements:

print ('Hello World! ')

print ('3^2 equal to ' 3**2)

To run this module, in the operating system’s prompt, type the following command line:

python test.py

If this command line does not work, you should type the full path of Python’s executable and the full path of the test.py file. You can also change the working directory by typing the full cd path of the test.py file, then type python test.py. Changing the working directory to the directory where you saved the modules is a good way to avoid typing the modules' full path every time running the module. The output is:

C:\Users>python C:\Users\test.py

Hello World!

3^2 equal to 9

When we run the module test.py, the results are displayed in the operating system’s prompt, and they go away as the prompt is closed. To store the results in a file, we can use a shell syntax by typing:

python test.py > save.txt

The output of test.py is redirected and saved in the save.txt file.

We are going to learn Python syntax. For now, we are going to use the command line to explore Python syntax. We will learn how to set and use some powerful Python programming platforms.

Python Syntax

Before we learn some Python syntax, we will explore the main types of data used in Python and how a program is structured. A plan is a set of modules, which are a series of statements that contain expressions. These expressions create and process objects, which are variables that represent data.

Python Variables

In Python, we can use built-in objects, namely numbers, strings, lists, dictionaries, tuples, and files. Python supports the usual numeric types, the integer, and float, as well as complex numbers. Strings are character chains, whereas lists and dictionaries are ensembles of other objects like a number or a string or other lists or dictionaries. Lists and dictionaries are indexed, and they can be iterated through.

The main difference between lists and dictionaries is how items are stored, and how they can be fetched. Items in a list are ordered and can be fetched by position, whereas they are stored and fetched in dictionaries by key. Tuples like lists are positionally ordered set of objects. Finally, Python also allows creating and reading files as objects. Python provides all the tools and mathematical functions to process these objects.

Python does not require variable declaration, size, or type declaration. Variables are created once they are assigned a value. For example:

>>> x=5

>>> print (x) 5

>>> x= 'Hello World! '

Hello World!

In the example above, x was assigned a number then it was assigned a string. In fact, Python allows changing the type of variables after they are declared. We can verify the type of any Python object using the type () function.

>>> x, y, z=10,'Banana,2.4

>>> print (type(x))

<class 'int '>

>>> print(type(y))

<class 'str '>

>>> print (type(z))

<class 'float '>

To declare a string variable, both single and double quotes can be used.

Only alpha-numeric characters and underscores can be used (e.g., A_9). Note that the variable names are case-sensitive and should not start with a number. For instance, price, Price, and PRICE are three different variables. Multiple variables can be affirmed in one line, as seen in the example above.

Chapter 3. Basic Terminology and Notations

Mathematical Notation for Machine Learning

You will realize that mathematical nomenclature and notations go hand in hand throughout your project in your machine learning process. There are various signs, symbols, values, and variables used in mathematics to describe whatever algorithms you may be trying to accomplish.

You will find yourself using some of the mathematical notations within this field of model development. You will find that values that deal with data and the process of learning or memory formation will always take precedence. Therefore, the following six examples are the most commonly used notations. Each of these notations has a description for which its algorithm explains:

Algebra

- To indicate a change or difference: Delta.

- To give the total summation of all values: Summation.

- To describe a nested function: Composite function.

- To indicate Euler's number and Epsilon where necessary.

- To describe the product of all values: Capital pi.

Calculus

- To describe a particular gradient: Nabla.

- To describe the first derivative: Derivative.

- To describe the second derivative: Second derivative.

- To describe a function value as x approaches zero: Limit.

Linear Algebra

- To describe capitalized variables are matrices: Matrix.

- To describe matrix transpose: Transpose.

- To describe a matrix or vector: Brackets.

- To describe a dot product: Dot.

- To describe a Hadamard product: Hadamard.

- To describe a vector: Vector.

- To describe a vector of magnitude 1: Unit vector.

Probability

- The probability of an event: Probability.

Set Theory

- To describe a list of distinct elements: Set.

Statistics

- To describe the median value of variable x: Median.

- To describe the correlation between variables X and Y: Correlation.

- To describe the standard deviation of a sample set: Sample standard deviation.

- To describe the population standard deviation: Standard deviation.

- To describe the variance of a subset of a population: Sample variance.

- To describe the variance of a population value: Population variance.

- To describe the mean of a subset of a population: Sample mean.

- To describe the mean of population values: Population means.

Terminologies Used for Machine Learning

The following terminologies are what you will encounter most often during machine learning. You may be getting into machine learning for professional purposes or even as an artificial intelligence (AI) enthusiast. Anyway, whatever your reasons, the following are categories and subcategories of terminologies that you will need to know and probably understand getting along with your colleagues. Here are machine-learning terms that you need to know:

1. Natural Language Processing (NLP)

Natural language is what you, as a human, use, i.e., human language. By definition, NLP is a way of machine learning where the machine learns your human form of communication. NLP is the standard base for all, if not most, machine languages that allow your device to use human (natural) language. This NLP enables your machine to hear your natural (human) input, understand it, execute it, and then give a data output. The device can realize humans and interact appropriately or as close to appropriate as possible.

There are five primary stages in NLP: machine translation, information retrieval, sentiment analysis, information extraction, and finally, question answering. It begins with the human query, which straight-up leads to machine translation, then through all the four other processes, and finally ending up in question explaining itself. You can now break down these five stages into subcategories as suggested earlier:

Text classification and ranking - This step is a filtering mechanism that determines the class of importance based on relevance algorithms that filter out unwanted stuff such as spam or junk mail. It filters out what needs precedence and the order of execution up to the final task.

Sentiment analysis - This analysis predicts the emotional reaction towards the feedback provided by the machine. Customer relations and satisfaction are factors that may benefit from sentiment analysis.

Document summarization - As the phrase suggests, this is a means of developing short and precise definitions of complex and complicated descriptions. The overall purpose is to make it easy to understand.

Named-Entity Recognition (NER) - This activity involves getting structured and identifiable data from an unstructured set of words. The machine learning process learns to identify the most appropriate keywords, apply those words to the speech context, and try to develop the most appropriate response. Keywords are things like company name, employee name, calendar date, and time.

Speech recognition - An example of this mechanism can easily be appliances such as Alexa. The machine learns to associate the spoken text to the speech originator. The device can identify audio signals from human speech and vocal sources.

It understands Natural language and generation - As opposed to Named-Entity Recognition; these two concepts deal with human to computer and vice versa conversions. Natural language understanding allows the machine to convert and interpret the human form of spoken text into a coherent set of understandable computer format. On the other hand, natural language generation does the reverse function, i.e., transforming the incorrect computer format to the human-understandable audio format.

Machine translation - This action is an automated system of converting one written human language into another human language. Conversion enables people from different ethnic backgrounds and different styles to understand each other. An artificial intelligence entity that has gone through the process of machine learning carries out this job.

2. Dataset

A dataset is a range of variables you can use to test your machine learning's viability and progress. Data is an essential component of your machine learning progress. It gives results that indicate your development, areas that need adjustments, and tweaking for fine-tuning specific factors. There are three types of datasets:

Training data - As the name suggests, training data is used to predict patterns by letting the model learn via deduction. Due to the enormity of factors to be trained on, yes, there will be more critical factors than others. These features get a training priority. Your machine-learning model will use the more prominent features to predict the most appropriate patterns required. Over time, your model will learn through training.

Validation data - This set is the data used to micro tune the small tiny aspects of the different models at the completion phase. Validation testing is not a training phase; it is a final comparative phase. The data obtained from your validation is used to choose your final model. You get to validate the models' various aspects under comparison and then make a final decision based on this validation data.

Test data - Once you have decided on your final model, test data is a stage that will give you vital information on how the model will handle in real life. The test data will be carried out using an utterly different set of parameters from the ones used during both training and validation. Having the model go through this kind of test data will indicate how your model will handle other types of inputs. You will get answers to questions such as how will the fail-safe mechanism react. Will the fail-safe even come online in the first place?

3. Computer Vision

Computer vision is responsible for the tools, providing a high-level analysis of image and video data. Challenges that you should look out for in computer vision are:

Image classification - This training allows the model to identify and learn what various images and pictorial representations are. The model needs to retain a familiar-looking image to maintain the mind and identify the correct image even with minor alterations such as color changes.

Object detection - Unlike image classification, which detects whether there is an image in your model field of view, object detection allows it to identify objects. Object identification enables the model to take a large set of data and then frames them to detect pattern recognition. It is akin to facial recognition since it looks for patterns within a given field of view.

Image segmentation - The model will associate a specific image or video pixel with a previously encountered pixel. This association depends on the concept of a most likely scenario based on the frequency of association between a particular pixel and a corresponding specific predetermined set.

Saliency detection - In this case, it will involve that you train and get your model accustomed to increase its visibility. For instance, advertisements are best at locations with higher human traffic. Hence, your model will learn to place itself in positions of maximum social visibility. This computer vision feature will naturally attract human attention and curiosity.

4.  Supervised Learning

You achieve supervised learning by teaching the models themselves by using targeted examples. If you wanted to show the models how to recognize a given task, then you would label the dataset for that particular supervised task. You will then present the model with the set of labeled examples and monitor its learning through supervision.

The models get to learn themselves through constant exposure to the correct patterns. You want to promote brand awareness; you could apply supervised learning where the model leans by using the product example and mastering its advertisement art.

5. Unsupervised Learning

This learning style is the opposite of supervised learning. In this case, your models learn through observations. There is no supervision involved, and the datasets are not labeled; hence, there is no correct base value as learned from the supervised method.

Here, through constant observations, your models will get to determine their right truths. Unsupervised models most often learn through associations between different structures and fundamental characteristics common to the datasets. Since unsupervised learning deals with similar groups of related datasets, they are useful in clustering.

6. Reinforcement Learning

Reinforcement learning teaches your model to strive for the best result always. In addition to only performing its assigned tasks correctly, the model gets rewarded with a treat. This learning technique is a form of encouragement to your model to always deliver the correct action and perform it well or to the best of its ability. After some time, your model will learn to expect a present or favor, and therefore, the model will always strive for the best outcome.

This example is a form of positive reinforcement. It rewards good behavior. However, there is another type of support called negative reinforcement. Negative reinforcement aims to punish or discourage bad behavior. The model gets reprimanded in cases where the supervisor did not meet the expected standards. The model learns that lousy behavior attracts penalties, and it will always strive to do good continually.

7. Neural Networks

The neural network is a concept of interconnected models connected through artificial intelligence. These models though synthetic, have the same level of interactions that is observable between humans. Due to the long period for training and learning models, their interconnectivity level will depend on an automated base system.

Chapter 4. Evaluating Models and Predicting Unseen Data Instances

How Is Python Chosen Over other Tools for Data Science?

Python has been the programming favored language for the regular exercises that information researchers address in a few circumstances and is one of the fundamental information examination systems used around the business. Python is ideal for information researchers who need to actualize numerical programming into their yield frameworks or join information into electronic applications. It's likewise reasonable for applying frameworks, something that PC researchers additionally need to do.

Suppose the application is written in a natural and average manner. In that case, it is called 'Pythonic.' Past that, Python is consistently prominent for certain limits that have gotten data science planner minds.

Direct Learning

Python's most appealing characteristic is that any individual who has to know it, even amateurs, can do so rapidly and effectively, so that is one reason why disciples favor data science to python. That, in like manner, fits well for dynamic people who contribute brief period mulling over. - most notably, R empowers a truncated learning measure to understand the phonetic structure instead of various lingos.

Data Science Vast Libraries

The actual favored situation of using python for data science would be that python offers associates with a sweeping scope of resources for data mining and PC taking care of. Most data scientists who use Python feel that this universal programming language settles a broad arrangement of troubles by giving inventive approaches to managing late saw as unsolvable issues.

Expandable

Like all various vernaculars, including R, when it relates to flexibility, Python outstands. This is way less unpredictable than the Stata and MATLAB tongues. This backings size gives flexibility and different streets to data scientists to deal with various issues — one explanation YouTube moved to the language. Python can be found across various ventures, enlivening the quick application progression for use examples, taking everything into account.

Colossal Community for Python

One explanation this is so incredibly known to Python is an immediate consequence of her lifestyle. While the data science pack starts to get a handle on it, more people volunteer by building new data science storage facilities. It further stimulates the improvement of the most advanced programming and enlisting strategies open today, which explains many individuals use Python for data science.

The lifestyle is a nearby one, and it has sometimes been more clear to find a reaction to a problematic issue. An actual yield of the web is all you require, so you can quickly find the response to specific issues or talk with someone who may maintain it. On Stack Overflow and Code Mentor, engineers can even associate with their companions.

Why Python and Data Science Mix Well?

Information science incorporates extrapolating helpful information from enormous records, information bases, and information stores. Such outcomes are typically unsorted and difficult to gauge with any sensible precision. ML may connect assorted datasets; however, it needs critical fallacy and force in the calculation. Python addresses that requirement by being a programming language of general use. It enables you to construct CSV performance in a spreadsheet for fast reading of the results. Additionally, more complex outputs of files that can be processed for processing by ML clusters.

For example, climate predictions are based on historical measurements from weather reports over a century old. ML can also render forecasting forecasts more reliable, based on historical weather patterns. Python could do that because code execution is efficient and lightweight, although it is multi-functional. Python can also enable structured, functional programming, and object-oriented patterns, meaning an implementation can be found anywhere.

The Python Package Index now contains over 100,000 libraries, and that number is continuing to grow. As described earlier, Python provides several data science-focused libraries. A simple examination on Google reveals lots of Top 10 Python libraries for lists of data science. The most common library for analyzing data is probably an open-source library named pandas. It is a highly tuned set of applications that makes Python's data analysis a much-simplified task.

Python has the tool-set to perform a wide range of powerful functions, no matter what experts are looking to be doing with Python, whether prescriptive analytics or predictive causal analytics. It is no wonder that data scientists adopted Python.

Data Science Statistical Learning

Statistical learning is a method for statistical-based interpretation of results, categorized as unsupervised or supervised.

One straightforward approach to explain statistical learning is to evaluate the relationship among predictor variables (features, independent variables) & responses (dependent variable) and to create an objective model that could predict the variable response (Y) based on predictor variables (X).

Inference and Prediction

In cases in which a set of inputs, X is readily accessible. Still, output Y is not understood. We sometimes view f as a black box (not connected to the exact shape of f), as much as it produces precise predictions for Y. It is a foretaste.

There are cases when we want to consider the way Y is influenced when X improves. We want to estimate f in this situation, but our aim is not really to generate forecasts for Y. We're more interested in explaining the connection between X and Y here. But f cannot be viewed as a black box, as we need to know the precise structure. This is inferential. Throughout actual life, you can find various issues going into the environment of assumptions, the environment of inferences, or a mixture of both.

Parametric and Non-Parametric Functions

If we take the statistical model off and attempt to approximate f by measuring the parameters' collection, such methods are considered parametric techniques.

Non-parametric techniques don’t make clear statements regarding the shape of f but rather aim to approximate f that comes as near as possible to the datasets.

Model Interpretability and Prediction Accuracy

Among the various approaches we use to study statistics, others are less versatile or more rigid. If the inference is the target, comfortable and reasonably inflexible mathematical analysis methods have advantages. When we are just involved in modeling, we use accessible modular models.

Model Accuracy Assessing

Estimates do not have a free meal, ensuring that no approach beats all the others for all available data sets. The most widely used factor in the regression framework is the MSE (Mean Squared Error). The most commonly used metric in the classification framework is the uncertainty matrix. The essential property of mathematical learning is that training error may decrease as model variability grows, but the test error does not.

Variance and Bias

Bias is the simplifying premises a designer creates for a smoother understanding of the goal task. Parametric models have a high bias, making them easy to know and more straightforward to understand but less versatile in general. Low-bias ML algorithms are Decision Trees, k-Nearest Neighbors, and Auxiliary Vector Machines. Linear Regression, Conditional Logistic Regression, and Discriminant Analysis are all methods in high-bias ML.

Variance is the rate that the goal role prediction might alter if specific training data were used. There is a large variance of non-parametric equations that provide a lot of variabilities. Logistic and Linear Regression, Linear Discriminant Analysis are techniques for learning machines with small variances. Decision Trees, k-Nearest Neighbors, and Help Vector Machines are ML algorithms with large variance.

Variance and Bias Relationship

The relation in statistical learning among variance and bias is such that:

- The variance may decline with rising bias.

- Rising variance can reduce bias.

There is an exchange-off between these two considerations and the templates we use, and with our question, the approach we want to customize them seeks various compromises in this trade-off.

Choosing the appropriate degree of versatility in both the classification and regression settings is essential to every predictive learning process's performance. The exchange-off of variance-bias, and the resultant U-shape in the testing mistake, will render this a challenging challenge.

Relation Between Big Data and Machine Learning (ML)

With the amount of data produced by individuals and companies at a skyrocketing speed, several concepts such as big data, deep learning, etc., have arisen. It's very natural to inquire if each other profits from these kinds of stuff. We will explore how big data helps ML to assist in making decisions.

Modern companies recognize the importance of big data, but they also realize that it can be much more efficient when combined with automated processes, and this is precisely where ML's strength falls into the frame. ML systems support businesses in various ways, including maintaining, assessing, and using the data captured far more effectively than ever.

In the general definition, ML is a series of technologies that allow linked computers and machines to know, create, and enhance through various approaches, based on their own experience. All the large companies, major software organizations, and computer scientists are forecasting these days that big data can create a considerable change in the world of machine-learning.

Fundamentally, ML is indeed an advanced type of artificial intelligence designed to learn new information from datasets of its own. It is focused on the premise that machines can learn from results, identify user trends, and make decisions without any human involvement.

Even as ML has been out for decades, models that can analyze more complicated, larger datasets, and generate more reliable data quickly and on a large scale – have become feasible nowadays. By developing such kinds of templates, a company becomes more likely to locate lucrative prospects out.

ML means no previous hypotheses. When ML algorithms are presented with the correct data, they will process the data and recognize trends. You will also use specific findings on other datasets. Such an approach is typically applied to high-dimension datasets. This ensures the more details you will provide, the more reliable the predictions would be. So here's precisely where big data 's influence falls in.

Chapter 5. Building Good Training Datasets

We introduce managing data as a Pandas dataframe and typical exploratory data analysis (EDA) techniques for querying your data.

As a crucial part of data inspection, EDA summarizes your dataset's critical characteristics in preparation for further processing. This includes understanding the data's shape and distribution, scanning for missing values, learning which features are most relevant based on correlation, and familiarizing yourself with the dataset's contents. Gathering this intel helps inform algorithm selection and highlight aspects of the dataset that require cleaning to prepare for further processing.

Using Pandas, we can use a range of simple techniques to summarize the data and additional options to visualize the data using Seaborn and Matplotlib.

Let’s begin by importing Pandas, Seaborn, and Matplotlib inline using the following code in Jupyter Notebook.

import pandas as pd

import seaborn as sns

%matplotlib inline

Note that using the inline feature of Matplotlib, we can display plots directly below the applicable code cell within Jupyter Notebook or other frontends.

Import Dataset

Datasets can be imported from various sources, including internal and external files, and random self-generated datasets called blobs.

The following sample dataset is an external dataset downloaded from Kaggle, called the Berlin Airbnb dataset. This data was scraped from Airbnb and contained detailed accommodation listings in Berlin, including location, price, and reviews.

Overview of the Berlin Airbnb dataset

After registering a free account and logging into Kaggle, download the dataset as a zip file. Then, unzip the downloaded file called listings.csv and import it into Jupyter Notebook as a Pandas dataframe using pd.read_csv.

df = pd.read_csv('~/Downloads/listings.csv')

Note that you’ll need to assign a variable name to store the dataset for ongoing reference. Common variable names for dataframes are “df” or “dataframe,” but you can also choose another variable name on the condition that it fits with Python’s naming conventions

Remember that your dataset's file path will vary depending on its saved location and your computer’s operating system. If saved to Desktop on Windows, you would import the .csv file using a structure similar to this example:

df = pd.read_csv('C:\\Users\\John\\Desktop\\listings.csv')

Preview the Dataframe

We can now use the Pandas’ head() command to preview the dataframe in Jupyter Notebook. The head() command must come after the variable name of the dataframe, which is df.

df.head()

To preview the dataframe, run the code by using right-click, and selecting “Run” or navigating from the Jupyter Notebook menu: Cell > Run All

Run All from the navigation menu

After the code has run, Pandas will populate the imported dataset as a dataframe, as shown in the screenshot.

Previewing a dataframe in Jupyter Notebook using head()

Notice that the first row (ID 2015, located in Mitte) is indexed at position 0 of the dataframe. The fifth row, meanwhile, is indexed at position 4. The indexing of Python elements starts at 0, which means you will need to subtract one from the actual number of rows when calling a specific row from the dataframe.

The dataframe’s columns, while not labeled numerically, abide by this same logic. The first column (ID) is indexed at 0, and the fifth column (neighbourhood_group) is indexed at 4. This is a fixed feature of working in Python and something to keep in mind when calling specific rows or columns.

By default, head() displays the first five rows of the dataframe, but you can expand the number of rows by specifying n number of rows inside parentheses, as demonstrated in Figure 9.

Previewing the first ten rows of a dataframe

The argument head(10) is used to preview the first ten rows of the dataframe. You can also view columns concealed to the right by scrolling to the right inside Jupyter Notebook. Regarding rows, you can only preview what’s specified in the code.

Lastly, you will sometimes see n= inserted inside the head(), an alternative method to specify n number of previewed rows.

Example Code:

df.head(n=10)

Dataframe Tail

The inverse operation of previewing the top n rows of the dataframe is the tail() method, displaying the bottom n rows of the dataframe. Below, we can see an example of previewing the dataframe using tail(), which by default also displays five rows. Again, you will need to run the code to view the output.

Previewing the last five rows of a dataframe using tail()

Find Row Item

While the head and tail commands are useful for gaining a general idea of the dataframe’s basic structure, these methods aren’t practical for finding an individual or multiple rows in the middle of a large dataset.

To retrieve a specific row or a sequence of rows from the dataframe, we can use the iloc[] command as demonstrated.

df.iloc[99]

Finding a row using .loc[ ]

Here, df.iloc[99] is used to retrieve the row indexed at position 99 in the dataframe, ID 151249 (a listing located in the neighborhood group Friedrichshain-Kreuzberg).

Shape

A quick method to inspect the size of the dataframe is the shape command, which yields rows and columns in the dataframe. This is useful because the dataset's size is likely to change as you remove missing values, recreate features, or delete features.

To doubt the number of rows and columns in the dataframe, you can use the shape command preceded by the dataset's name (parenthesis are not used with this command).

df.shape

Inspecting the shape (number of rows and columns) of the dataframe

In the case of this dataframe, there are 22,552 rows and 16 columns.

Columns

Another useful command is columns, which prints the dataframe’s column titles. This is useful for copying and pasting columns back into the code or clarifying the name of specific variables.

df.columns.

Print columns

Describe

The describe() method is convenient for generating a summary of the dataframe’s mean, standard deviation, and IQR (interquartile range) values. This method performs optimally with continuous values (integers or floating-point numbers that can be aggregated).

df.describe()

Using the describe method to summarize the dataframe

By default, describe() excludes columns containing non-numeric values and instead provides a statistical summary of those columns containing numeric values. However, it’s also possible to run this command on non-numerical values by adding the argument include='all’ within parentheses to obtain the summary statistics of both numeric and non-numeric columns (where applicable).

df.describe(include='all').

All variables added to the description

Having consolidated methods to inspect and query the size of the dataframe using Pandas, we’ll now move on to generating visual summaries of the data using Seaborn and Matplotlib.

Pairplots

One of the most popular exploratory techniques for understanding patterns between two variables is the pairplot. A pairplot takes the form of a 2-D or 3-D grid of plots that plot variables against other variables taken from the dataframe, as shown in Figure 16.

sns.pairplot(df,vars=['price','number_of_reviews','availability_365'])

Example of a pairplot grid based on three chosen variables

Using a pairplot from Seaborn, we’ve plotted three chosen variables against each other, which helps us to understand relationships and variance between those variables. When plotted against other variables (multivariant), the visualization takes the form of a scatterplot, and when plotted against the same variable (univariant), a simple histogram is generated.

Heatmaps

Heatmaps are also useful for inspecting and understanding relationships between variables. The variables are structured as columns and rows on a matrix, with individual values represented as colors on a heat map.

We can build a heatmap in Python using the corr (correlation) function from Pandas and visualize the results using a Seaborn heatmap.

df_corr = df.corr()

sns.heatmap(df_corr,annot=True,cmap='coolwarm')

Example of a heatmap with annotated correlation values

Chapter 6. Combining Different Models for Ensemble Learning

When making important decisions, we generally prefer to collate multiple opinions instead of listening to a single voice or the first person to add their opinion. Similarly, it’s essential to consider and trial more than one algorithm to find your data's best prediction. In advanced machine learning, it can be advantageous to combine models using ensemble modeling, which amalgamates outputs to build a unified prediction model. By combining the output of different models (instead of relying on a single estimate), ensemble modeling helps build a consensus on the data's meaning. Aggregated estimates are also generally more accurate than any one technique. It’s vital, though, for the ensemble models to display variation to avoid mishandling the same errors.

In classification, multiple models are consolidated into a single prediction using a voting system based on frequency or numeric averaging in regression problems. Ensemble models can also be divided into sequential or parallel and homogenous or heterogeneous.

Let’s start by looking at sequential and parallel models. In the former's case, the model’s prediction error is reduced by adding weights to classifiers that previously misclassified data. Gradient boosting and AdaBoost (designed for classification problems) are both examples of sequential models. Conversely, parallel ensemble models work concurrently and reduce error by averaging. Random forests are an example of this technique.

Ensemble models can be generated using a single technique with numerous variations, known as a homogeneous ensemble, or through different techniques, known as a heterogeneous ensemble. An example of a homogeneous ensemble model would be multiple decision trees to form a single prediction (i.e., bagging). Meanwhile, an example of a heterogeneous ensemble would be the usage of k-means clustering or a neural network in collaboration with a decision tree algorithm.

Naturally, it’s crucial to select techniques that complement each other. For instance, neural networks require complete data for analysis, whereas decision trees are competent at handling missing values. Together, these two techniques provide added benefit over a homogeneous model. The neural network accurately predicts the majority of instances where a value is provided. The decision tree ensures no “null” results that would otherwise materialize from missing values using a neural network.

While an ensemble model's performance outperforms a single algorithm in most cases, the degree of model complexity and sophistication can pose a potential drawback. An ensemble model triggers the same trade-off in benefits as a single decision tree and a collection of trees. The transparency and ease of interpretation of, say, decision trees are sacrificed for the accuracy of a more complex algorithm such as random forests, bagging, or boosting. The model's performance will win out in most cases, but interpretability is a crucial factor to consider when choosing the right algorithm(s) for your data.

In terms of selecting a suitable ensemble modeling technique, there are four main methods: bagging, boosting, a bucket of models, and stacking.

A bucket of models trains multiple different algorithmic models using the same training data as a heterogeneous ensemble technique. It then picks the one that performed most accurately on the test data.

Bagging, as we know, is an example of a parallel model averaging using a homogenous ensemble, which draws upon randomly drawn data and combines predictions to design a unified model.

Boosting is a popular alternative technique that is still a homogenous ensemble but addresses error and data misclassified by the previous iteration to produce a sequential model. Gradient boosting and AdaBoost are both examples of boosting algorithms.

Stacking runs multiple models simultaneously on the data and combines those results to produce a final model. Unlike boosting and bagging, stacking usually combines outputs from different algorithms (heterogenous) rather than altering the same algorithm's hyperparameters (homogenous). Also, rather than assigning equal trust to each model using averaging or voting, stacking attempts to identify, and emphasize well-performing models. This is achieved by smoothing out the error rate of models at the base level (known as level-0) using a weighting system before pushing those outputs to the level-1 model. They are combined and consolidated into a final prediction.

Stacking algorithm

While this technique is sometimes used in industry, the gains of using a stacking technique are marginal in line with the complexity level, and organizations usually opt for the ease and efficiency of boosting or bagging. However, stacking is a go-to technique for machine learning competitions like the Kaggle Challenges and the Netflix Prize. The Netflix competition, held between 2006 and 2009, offered a prize for a machine learning model that could significantly improve Netflix’s content recommender system. From the team BellKor’s Pragmatic Chaos, one of the winning techniques adopted linear stacking that blended predictions from hundreds of different models using different algorithms.

Chapter 7. Applying Machine Learning to Sentiments Analysis

Natural Language Processing (NLP) is a widely used field within Artificial Intelligence, which mainly involves the interactions between the human language and the computer. You can find its applications in a large variety of areas such as Sentiment analysis, Spam detecting, POS (Part-Of-Speech) Tagging, Text summarization, Language translation, Chatbots, and so on.

1. How Would you Explain NLP to a Layman? Why Is it Difficult to Implement?

NLP stands for Natural Language Processing, the ability of a computer program to understand the human language. It is an extremely challenging field for obvious reasons. NLP requires a computer to understand what humans speak. But human speech is very often not precise. Humans use slang, pronounce the words differently, and have the context in their sentences, which is very hard for a computer to process correctly.

2. What Is the Use of NLP in Machine Learning?

At present, NLP is based on Deep Learning. Deep Learning algorithms are a subset of Machine Learning, which needs a large amount of data to learn high-level features from data independently. NLP also works on the same approach, uses deep learning techniques to learn human language, and improve upon itself.

3. What Are the Different Steps in Performing Text Classification?

Text classification is an NLP task used to classify text documents into one or more categories. Classifying whether an email is spam or not, analyzing a person’s sentiments from his post, etc., are text classification problems.

A Text classification pipeline involves the following steps in order:

A.                           Text cleaning

B.                           Text annotation to create the features

C.                           Converting those features into actual predictors

D.                           Using the predictors to train the model

E.                           Fine-tune the model to improve its accuracy.

4. What Do you Understand by Keyword Normalization? Why Is it Needed?

Keyword normalization, also known as text normalization, is a crucial step in NLP. It is used to transform the keyword into its canonical form, making it easier to process. It removes stop words such as punctuation marks, words like "a," "an," "the" because these words generally do not carry any weight. After that, it converts the keywords into their standard forms, which improves text matching.

For instance, reducing all words to lower cases, converting all tenses to simple present tense. So, if you have "decoration" in one document and "Decorated" in the other, then both of them would be indexed as "decorate." Now, you can easily apply a text-matching algorithm on these documents, and a query containing the keyword "decorates" would match with both of the documents. Keyword normalization is an excellent means of reducing dimensionality.

5. Tell me about Part-Of-Speech (POS) Tagging.

Part-of-speech tagging is a process of marking the words in a given text as a part of speech, such as nouns, prepositions, adjectives, verbs, etc. It is an extremely challenging task because of its complexity and because the same word could represent a different part of speech in different sentences.

There are generally two techniques used to develop POS tagging algorithms. The first technique is stochastic, which assumes that each word is known and can have a finite set of tags that are learned during training. The second technique is rule-based tagging, which uses contextual information to tag each word.

6. Have you Heard of the Dependency Parsing Algorithm?

Dependency Parsing algorithm is a grammar-based text parsing technique used to detect noun phrases, noun phrases, subjects, and objects in the text. "Dependency" implies the relations between the words in a sentence. There are various methods to parse a sentence and analyze its grammatical structure. Some of the standard methods include Shift-Reduce and Maximum Spanning Tree.

7. Explain the Vector Space Model and its Use.

Vector Space Model is an algebraic model used to represent an object as a vector of identifiers. Each object (such as a text document) is written as a vector of terms (words) present in it with their weights.

For instance, you have a document "d" with the text "This is an amazing journal for the interview preparation."

The corresponding vector for this document is:

There exist many ways to calculate these weights. They can be as simple as just the frequency (count) of the words in a document. Similarly, any query is also written in the same fashion. The vector operations are used to compare the query with the documents to find the most relevant documents that satisfy the query.

Vector Space Model is used extensively in the fields of Information Retrieval and Indexing. It provides a structure to the unstructured datasets, thereby making it easier to interpret and analyze them.

8. What Do you Mean by Term Frequency and Inverse Document Frequency?

Term Frequency (tf) is the number of times a term occurs in a document divided by the total number of terms in that document.

Inverse Document Frequency (idf) is a measure of how relevant is the term across all the documents. Mathematically, it is logarithmic (total number of documents divided by the number of documents containing the term).

9. Explain Cosine Similarity in a Simple Way.

Cosine similarity captures the similarity between two vectors. As explained in the vector space model, each document and the query is written as a vector of terms.

The cosine is calculated for the query vector with each document, which is the average cosine between two vectors. The resulting cosine value represents the similarity of the document with the given query. If the cosine value is 0, then there is no similarity at all, and if it is 1, then the document is the same as the query.

10. Explain the N-Gram Method.

Simply put, an N-gram is a contiguous sequence of n items in the given text. N-gram method is a probabilistic model used to predict an item in a sequence based on the previous n-1 items. You can choose the items to be either the words, phrases, etc. If n is 1, then it is called 1-gram; for n = 2, it is 2-gram or bigram.

N-grams can be used for approximate matching. Since they convert the sequence of items into a set of n-grams, you can compare one sequence with another by measuring the percentage of common n-grams in both of them.

11. How Many 3-Grams Can Be Generated from this Sentence "I Love New York Style Pizza"?

Breaking the given sentence into 3-grams, you get:

A.                           I love New

B.                           love New York

C.                           New York style

D.                           York-style pizza

# We will use the CountVectorizer package to demonstrate how to use N-Gram with Scikit-Learn.

# CountVectorizer converts a collection of text documents to a matrix of token counts.

# In our case, there is only one document.

from sklearn.feature_extraction.text import CountVectorizer

# N-gram_range specifies the lower and upper boundary on the range of N-gram tokens

# to be extracted. For our example, the range is from 3 to 3.

# We have to specify the token pattern because, by default, CountVectorizer treats single character words as stop words.

vectorizer = CountVectorizer(ngram_range=(3, 3),

token pattern = r"(?u)\b\w+\b",

lowercase=False)

# Now, let's fit the model with our input text

vectorizer.fit(["I love New York style pizza"])

# This will populate vectorizer's vocabulary_ dictionary with the tokens.

# Let's see the results of this vocabulary

print(vectorizer.vocabulary_.keys())

12. Have you Heard of the Bag-Of-Words Model?

The Bag-of-Words model is a widespread technique used in Information Retrieval and Natural Language Processing. It is also known as the Vector Space Model, described in detail in question 6 above. It uses the frequency of occurrence of the words in a document as the feature value.

One of the limitations of this method is that it does not take into account the order of the words in a document, due to which you cannot infer the context of the words. For instance, if you take these two sentences, "Apple has become a trillion-dollar company" and "You should eat an apple every day," the Bag-of-Words model won’t be able to differentiate between Apple as a company and Apple as a fruit. To address this limitation, you can use the N-gram model, which stores the words' spatial information. Bag-of-Words is a particular case of the N-gram method with n=1.

Chapter 8. Conditional or Decisional Statements

These will be an essential part of the code that we work with because they will ensure that your system can respond to the input that the user provides to you. It is hard to predict how a user is going to work with the system. However, you can set up some of the conditions you would like to look at and work from there to develop the way your program will work.

As we can imagine here, it is pretty much impossible for a programmer to create something and guess ahead of time what answers or input the user will provide to the program. And the programmer can’t be there watching each use work with the program either, which means that they need to work with the conditional statements. When these are set up correctly, it will ensure that the program will run properly and respond to any information that the user is providing to you.

There are many different types of programs that will respond well to the conditional statements we will discuss in this guide. These are pretty simple to work with, and we will take a look at some of the examples of how you can code with these conditional statements.

We will look at the three main types of conditional statements: the if statement, the if-else statement, and the if statement. Let’s take a look at how each of these statements works and use these conditional statements.

The If Statement

As we mentioned, there are three types of conditional statements that we can take a look at. The first one that we need to explore a bit is the if statement. Out of the three that we will spend some time on, the if statement is the most basic. These are not going to be used as much as other options because they often leave a bit desired. However, they are a good springboard for learning what these conditional statements are about and working with them.

With the if statement, the program is set up only to proceed if the user provides us with an input that works with the conditions we set ahead of time. If the input that we get from the user doesn’t match our conditions, then the program will just stop, and nothing is going to happen.

As we can see already, there will be some issues with this because we don’t want the program to stop with the answer. It should still provide us with some of the basis that we need.

There are a few things that will show up with this code. If you have a user go to the program and state that their age is under 18 years, the program will display the listed message. The user can read this message and end the program right there.

But, things can go wrong if the user puts in that their age is above 18 years. This is true for the user, but because it doesn’t meet the conditions you coded in. Thus, the program will see it as false. Like the code is written right now, nothing will happen because it isn’t set up to handle this. The user will just see a blank screen any time they enter an over 18 years of age.

The If-Else Statement

Now that we have had some time to look at the simple if statement, it is time for us to move on to the if-else statement. The if the statement is an excellent way to get a bit of practice in coding, there will not be all that many times when we are programming and need to work with this kind of statement.

When your user works with the program, you want to make sure that something shows up on the screen no matter what input they use.

If you use the if statement, like in the example above, and the user puts in an answer (above 18 years), the screen will come back blank using the code that we had from before. This is not something that we want to see, so we need to move on to the if-else statement to see what we can do regardless of what information the user puts into the program.

The if-else statement will provide us with output and ensure that we provide these outputs to the user, regardless of the age or other information we provide to the program. With the example above, if the user comes in and says that they are 40 years old, then the code will still respond to it.

There are a few options that you can use with this one, but with the idea of the voting option that we talked about with the if statement.

With this option, you add in the else statement, which will cover every age that doesn’t fall under 18. This way, if the user does list that as their age, something will still appear on the screen for them. This can provide you with more freedom when working on your code, and you can even add in a few more layers to this. If you want to divide it so that you get four or five age groups and each one gets a different response, you simply need to add more if statements to make that happen. The else statement is at the end to catch everything else.

For example, you can take the code above and ask the user what their favorite color is. You could then have if statements to cover some of the primary colors, such as red, blue, green, yellow, orange, purple, and black.

If the user puts in one of those colors, then the corresponding statement will show up on the screen. The else statement will be added to help catch any other colors that the person may try to use, such as pink or white.

The Elif Statements

The third type of conditional statement that we can work within this process is known as the elif statement. These are going to help us add another level to what we did with the other step. However, they are still going to make sure that the codes we write are as easy as possible.

We can create as many of these elif statements as possible in the code, as long as we add in the else statement at the end. The else statement ensures that we can handle any of the other answers that the user puts in, even those we may not have thought of ahead of time.

When working with the elif statement, it will be similar to giving the user a menu to pick from. You can choose how many of these elif statements you would like to have present in the menu, similar to what is found in many games. Then, the user can pick and choose which one they would like to work with. You can then have a particular action happen, or a confident statement shows up on the program to meet your needs.

Another thing to notice with the elif statement is that you can add many different options as your code needs. It is possible to make a small menu that just has two or three items in it, or it is possible to expand this out to as many of these as you need to make the code work properly.

The fewer options you work within this one, the easier your code writing will be, so keep that in mind when determining how many options are needed.

Now that we know a little bit about elif statements and how they work, let’s dive in and take a look at a good example of one that you can write out. Open up your compiler and type in the following code:

Print(“Let’s enjoy a Pizza! Ok, let’s go inside Pizza hut!”)

print(“Waiter, Please select Pizza of your choice from the menu”)

pizzachoice = int(input(“Please enter your choice of Pizza:”))

if pizzachoice == 1:

print(‘I want to enjoy a pizza Napoletana’)

elif pizzachoice == 2:

print(‘I want to enjoy a pizza rustica’)

elif pizzachoice == 3:

print(‘I want to enjoy a pizza capricciosa’)

else:

print(“Sorry, I do not want any of the listed pizza’s, please bring a Coca Cola for me.”)

With this option, the user can choose the type of pizza they want to enjoy, but you can use the same syntax for anything you need in your code. If the user enters the number 2 in the code, they will get a pizza rustica. If they don’t like any of the options, they tell the program that they want to have something to drink, in this case, a Coca Cola.

Control Flow

The control flow in a program highlights the order of steps of the program execution. In a Python program, control flow is carried out by function calls, conditional statements, and loops. Here, we will deal with if statements, while loops, and for-loops.

Chapter 9. Functions

When you are working with a language like Python, there will be times when you will need to work with something that is known as a function.

These functions will be blocks of reusable code that you will use to get your specific tasks done.

But when you define one of these functions in Python, we need to have a good idea of the two main types of functions that can be used and how they work.

The two types of functions that are available here are known as built-in and user-defined.

The built-in functions are the ones that will come automatically with some of the packages and libraries that are available in Python.

Still, we will spend our time working with the user-defined functions because these are the ones that the developer will create and use for special codes they write.

In Python, though, one thing to remember no matter what kind of function you are working with is that all of them will be treated like objects.

This is good news because it can make it a lot easier to work with these functions than what we may see with some other coding languages.

The user-defined functions will be essential and can expand out some of the work we are doing. But we also need to look at some of the work that we can do with our built-in functions. The list above includes many of the ones that are found inside of the Python language. Take some time to study them and see what they can do to help us get things done.

Why Are User-Defined Functions so Important?

- To keep it simple, a developer will have the option of either writing out some of their functions, known as a user-defined function or going through and borrowing a function from another library, which may not be directly associated with Python. These functions are sometimes going to provide us with a few advantages depending on how and when we would like to use them in the code. When working on these user-defined functions and to gain a better understanding of how they work, some things to remember will be the functions that will be made with reusable code blocks. It is necessary to write them out once, and then you can use them as many times as you need in the code. You can even take that user-defined function and use it in some of your other applications as well.

- These functions can also be handy. You can use them to help with anything you want, from writing out specific business logic to working on standard utilities. You can also modify them based on your requirements to make the program work properly.

- The code is often going to be friendly for developers, easy to maintain, and well-organized all at once. This means that you can support the approach for modular design.

You can write out these functions independently, and your project's tasks can be distributed for rapid application development if needed. A user-defined function that is thoughtfully and well-defined can help ease the process for the development of an application. Now that we know a little bit more about the basics of a user-defined function, it is time to look at some of the different arguments that can come with these functions before moving on to some of the codes you can use a function.

Options for Function Arguments

Any time you are ready to work with these kinds of functions in your code, you will find that they can work with four types of arguments. These arguments and the meanings behind them are something that will be pre-defined, and the developer is not always going to be able to change them up. Instead, the developer will have the option to use them but follow the rules there. You do get the option to add a bit to the rules to make the functions work the way you want. As we said before, there are four argument types you can work with, and these include:

- Default arguments: In Python, we will find a bit different way to represent the default values and the syntax for your functions' arguments. These default values will be the part that indicates that the function's argument is going to take that value if you don’t have a value for the argument that can pass through the call of the function. The best way to figure out where the default value is will be to look for the equal sign.

- Required argument: The following type of argument is going to be the required arguments. Some kinds of arguments will be mandatory for the function that you are working on. These values need to go through and be passed in the right order and number when the function is called out, or the code won’t be able to run the right way.

- Keyword arguments: These are going to be the argument that will be able to help with the function call inside of Python. These keywords will be the ones that we mention through the function call and some of the values that will go all through this one. These keywords will be mapped with the function argument to identify all of the values, even if you don’t keep the same order when the code is called.

- Variable arguments: The last argument that we will take a look at here is the variable number of arguments. This is good for working when you are not sure how many arguments will be necessary for the code you are writing to pass the function. Or you can use this to design your code where any number of arguments can be passed, as long as they have been able to pass any of the requirements in the code that you set.

Writing a Function

Now that we have a little better idea of what these functions are like and some of the argument types available in Python, it is time for us to learn the steps you need to accomplish all of this.

There are going to be four basic steps that we can use to make all of this happen, and it is really up to the programmer how difficult or simple you would like this to be. We will start with some of the basics, and then you can go through and make some adjustments as needed. Some of the steps that we need to take to write out our user-defined functions include:

- Declare your function. You will need to use the “def” keyword and then have the function's name come right after it.

- Write out the arguments. These need to be inside the two parentheses of the function. End this declaration with a colon to keep up with the proper writing protocol in this language.

- Add in the statements that the program is supposed to execute at this time.

- End the function. You can choose whether you would like to do it with a return statement or not.

An example of the syntax that you would use when you want to make one of your user-defined functions includes:

def userDefFunction (arg1, arg2, arg3, …):

program statement1

program statement2

program statement3

Return;

Working with functions can be a great way to ensure that your code will behave the way you would like. Making sure that you get it set up correctly and working through these functions, getting them set up in the manner you would like, can be important. There are many times when the functions will come out and serve some purpose, so taking the time to learn how to use them can be very important to your code's success.

Python Modules

Modules consist of definitions as well as program statements. An illustration is a file name config.py that is considered as a module. The module name would be config. Modules are sued to help break large programs into smaller manageable, organized files, and promote code reusability.

Example

Creating the First module

Def add(x, y):

“This is a program to add two numbers and return the outcome."

outcome=x+y

return outcome

Module Import

Keyword import is used to import.

Example

Import first

The dot operator can help us access a function as long as we know the module's name.

Example

Start IDLE.

Navigate to the File menu and click New Window.

Type the following:

import mine

import mine

import mine

mine.reload(mine)

Dir() built-in Python function

For discovering names contained in a module, we use the dir() inbuilt function.

Syntax

dir(module_name)

Python Package

Files in python hold modules, and directories are stored in packages. A single package in Python holds similar modules. Therefore, different modules should be placed in different Python packages.

Chapter 10. Actual Machine Learning Algorithms

Decision trees are built similarly to support vector machines, meaning they are a category of supervised machine learning algorithms capable of solving both regression and classification problems. They are powerful and used when working with a great deal of data.

You need to learn beyond the barebones basics so that you can process large and complex datasets. Furthermore, decision trees are used in creating random forests, which is arguably the most powerful learning algorithm.

An Overview on Decision Trees

Decision trees are essentially a tool that supports a decision that will influence all the other decisions that will be made. This means that everything from the predicted outcomes to consequences and resource usage will be influenced somehow. Take note that decision trees are usually represented in a graph, which can be described as some kind of chart where the training tests appear as a node. For instance, the node can be the toss of a coin, which can have two different results. Furthermore, branches sprout to represent the results individually, and they also have leaves, which are the class labels. Now you see why this algorithm is called a decision tree. The structure resembles an actual tree. As you probably guessed, random forests are exactly what they sound like. They are collections of decision trees, but enough about them.

Decision trees are one of the most powerful supervised learning methods you can use, especially as a beginner. Unlike other more complex algorithms, they are fairly easy to implement, and they have a lot to offer. A decision tree can perform any common data science task, and the results you obtain at the end of the training process are highly accurate. With that in mind, let’s analyze a few other advantages, as well as disadvantages, to gain a better understanding of their use and implementation.

Let’s begin with the positives:

1. Decision trees are simple in design and, therefore, easy to implement even if you are a beginner without a formal education in data science or machine learning. The concept behind this algorithm can be summarized with a sort of a formula that follows a common type of programming statement: If this, then that, else that. Furthermore, the results you will obtain are very easy to interpret, especially due to the graphic representation.

2. The second advantage is that a decision tree is one of the most efficient methods in exploring, determining the most important variables, and discovering the connection between them. Also, you can build new features easily to gain better measurements and predictions. Don’t forget that data exploration is one of the most important stages in working with data, especially when there are many variables involved. You need to detect the most valuable ones to avoid a time-consuming process, and decision trees excel at this.

3. Another benefit of implementing decision trees is that they are excellent at clearing up some of your data's outliers. Don’t forget that outliers are noise that reduces the accuracy of your predictions. Besides, decision trees aren’t that strongly affected by noise. In many cases, outliers have such a small impact on this algorithm that you can even choose to ignore them if you don’t need to maximize the accuracy scores.

Finally, there’s the fact that decision trees can work with both numerical as well as categorical variables. Remember that some of the algorithms we already discussed can only be used with one data type or the other. On the other hand, decision trees are proven to be versatile and handle a much more varied set of tasks.

As you can see, decision trees are powerful, versatile, and easy to implement, so why should we ever bother using anything else? As usual, nothing is perfect, so let’s discuss the negative side of working with this type of algorithm:

1. One of the biggest issues encountered during a decision tree implementation is overfitting. Please note that this algorithm sometimes creates very complicated decision trees with issues generalizing data due to their complexity. This is known as overfitting, and it is encountered when implementing other learning algorithms as well, however, not to the same degree. Fortunately, this doesn’t mean you should stay away from using decision trees. All you need to do is invest some time to implement certain parameter limitations to reduce overfitting.

2. Decision trees can have issues with continuous variables. When continuous numerical variables are involved, the decision trees lose a certain amount of information. This problem occurs when the variables are categorized. If you aren’t familiar with these variables, a continuous variable can be a value set within a range of numbers. For example, suppose people between ages 18 and 26 are considered of student age. In that case, this numerical range becomes a continuous variable because it can hold any value between the declared minimum and maximum.

3. While some disadvantages can add to additional work in decision trees, the advantages still outweigh them by far.

Classification and Regression Trees

We discussed earlier that decision trees are used for both regression tasks as well as classification tasks. However, this doesn’t mean you implement the same decision trees in both cases. Decision trees need to be divided into classification and regression trees. They handle different problems; however, they are similar since they are both decision trees.

Take note that classification decision trees are implemented when there’s a categorical dependent variable. On the other side, a regression tree is only implemented in a continuous dependent variable. Furthermore, in the case of a classification tree, the training data result is the mode of the total relevant observations. This means that any observations that we cannot define will be predicted based on this value, representing the observation we identify most frequently.

Regression trees, on the other hand, work slightly differently. The value that results from the training stage is not the mode value but the total observations' mean. This way, the unidentified observations are declared with the mean value, which results from the known observations.

Both types of decision trees undergo a binary split, however, going from the top to bottom. This means that the observations in one area will spawn two branches divided inside the predictor space. This is also known as a greedy approach because the learning algorithm seeks the most relevant variable in the split while ignoring the future splits that could lead to an even more powerful and accurate decision tree.

As you can see, there are some differences as well as similarities between the two. However, what you should note from all of this is that the splitting affects the accuracy scores of the decision tree implementation. Decision tree nodes are divided into subnodes, no matter the type of tree. This tree split is performed to lead to a more uniform set of nodes.

Now that you understand the fundamentals behind decision trees, let’s dig a bit deeper into overfitting.

The Overfitting Problem

You learned earlier that overfitting is one of the main problems when working with decision trees, and sometimes it can have a severe impact on the results. Decision trees can lead to a 100% accuracy score for the training set if we do not impose any limits. However, the major downside here is that overfitting creeps when the algorithm seeks to eliminate the training errors, increasing the testing errors. This imbalance, despite the score, leads to terrible prediction accuracy in the result. Why does this happen? In this case, the decision trees grow many branches, and that’s the cause of overfitting. To solve this use, you need to impose limitations on how much the decision tree can develop and how many branches it can spawn. Furthermore, you can also prune the tree to keep it under control, much like how you would do with a real tree to make sure it produces plenty of fruit.

To limit the decision tree's size, you need to determine new parameters during the tree's definition. Let’s analyze these parameters:

1. min_samples_split: The first thing you can do is change this parameter to specify how many observations a node will require to perform the splitting. You can declare anything with a range of one sample to maximum samples. Just keep in mind that to limit the training model from determining the connections that are very common to a particular decision tree, you need to increase the value. In other words, you can limit the decision tree with higher values.

2. min_samples_leaf: This is the parameter you need to tweak to determine how many observations are required by a node, or in other words, a leaf. The overfitting control mechanism works the same way as for the sample split parameter.

3. max_features: Adjust this parameter to control the features that are selected randomly. These features are the ones that are used to perform the best split. To determine the most efficient value, you should calculate the square root of the total features. Just keep in mind that the higher value tends to lead to the overfitting problem we are trying to fix in this case. Therefore, you should experiment with the value you set. Furthermore, not all cases are the same. Sometimes a higher value will work without resulting in overfitting.

4. max_depth: Finally, we have the depth parameter, which consists of the decision tree's depth value. To limit the overfitting problem, however, we are only interested in the maximum depth value. Take note that a high value translates to many splits, therefore a high amount of information. By tweaking this value, you will control how the training model learns the sample's connections.

Modifying these parameters is only one aspect of gaining control of our decision trees to reduce overfitting, boost performance, and accuracy. The following step after applying these limits is to prune the trees.

Chapter 11. Applications of the Machine Learning Technology

Virtual Personal Assistants

The most popular examples of virtual personal assistance are Siri and Alexa. These systems are capable of providing relevant information using simple voice commands. Machine learning is at the heart of these devices and systems. They collect and define the information generated with every user interaction and use it as training data to learn user preferences and provide an enhanced experience.

Predictions While Driving

Most of the vehicles today utilize GPS navigation services, which collects information such as our current location and driving speed on a centralized server that can generate a map of the current traffic. This helps in managing traffic and reducing congestion. With machine learning, the system can estimate the regions where and the time of the day when traffic jams occur frequently. Machine learning algorithms allow ride-sharing services such as Lyft and Uber to minimize detours on their routes and provide users an upfront estimate of how much the ride will cost.

Video Surveillance

Machines have taken over the monotonous job of monitoring multiple video cameras to ensure the security of premises. Machines can track unusual behavior like standing motionless for an extended period, sleeping on benches, and stumbling. It can then send an alert to the security personnel, who can decide to act on the tip and avoid mishaps. With every iteration of reporting, the surveillance services are improved as the machine learning algorithms learn and improve upon themselves.

Social Media

Social media platforms such as “Facebook,” “Twitter” and “Instagram” are using machine learning algorithms to train the system from user activity and behavior to be able to provide an engaging and enhanced user experience. Some of the examples of the functionalities that are being driven by machine learning algorithms are the “People you may know” feature on Facebook (that collects and learns from user activities such as the profiles they visit often, they’re own profile and their friends to suggest other Facebook users that they can become friends with) and “Similar Pins” feature on Pinterest (that is driven by computer vision Technology working in tandem with machine learning to identify objects in the images of user’s saved “pins” and recommend similar “pins” accordingly).

Email Spam and Malware Filtering

All email clients such as Gmail, Yahoo Mail, and Hotmail use machine learning algorithms to ascertain that the spam filter functionality is continuously updated and cannot be penetrated by spammers and malware. Some of the spam filtering techniques powered by machine learning are Multi-Layered Perceptron and C 4.5 decision tree induction.

Online Customer Service

Nowadays, most e-commerce sites allow users to chat with a customer service representative, usually supported by a Chatbot instead of a live executive. These bots use machine learning technology to understand user inquiries and extract information from the website to resolve customer issues. With every interaction, Chatbots become smarter and more humanlike.

Refinement of Search Engine Results

Search engines such as “Google,” “Yahoo,” and “Bing” use machine learning algorithms to provide improved search results pertinent with the user-provided keywords. For every search result, the algorithm observes and learns from user activity such as opening suggested links, the order in which the opened link was displayed, and time spent on the opened link. This helps the search engine understand which search results are more optimal and any further modifications to improve the search results.

Product Recommendations

The product recommendation feature has now become the heart and soul of the online shopping experience—machine learning algorithms, combined with artificial intelligence, fuel the product recommendation functionality. The system observes and learns from consumer activity and behavior such as past purchases, wish lists, recently viewed items, and liked or added to cart items.

Online Fraud Detection

Financial institutions rely heavily on machine learning algorithms and artificial intelligence to secure cyberspace by tracking potentially fraudulent monetary transactions online. For example, PayPal is using Machine learning algorithms to prevent money laundering through its platform. They are using artificial intelligence tools in combination with Machine learning algorithms to analyze millions of transactions and discriminate between legitimate and illegitimate transactions between the buyer and the seller. With every transaction, the system learns which transactions are legitimate and which transactions could be potentially fraudulent.

Predictive Analytics

As per SAS, the prescient examination is the "utilization of information, accurate calculations, and AI methods to extricate the probability of future results dependent on verifiable information. The objective is to go past, realizing what has ended up giving the best appraisal of what will occur after on." Today, organizations are burrowing through their past with an eye on the future. This is where human-made consciousness for promoting becomes an integral factor, using proactive examination innovation. The prescient examination's accomplishment is straightforwardly relative to the nature of large information gathered by the organization.

Here is a portion of the broadly utilized prescient examination applications for showcasing:

Prescient Analysis for Customer Behavior

For the modern goliaths like "Amazon," "Apple," and "Netflix," examining client exercises and conduct is essential to their everyday activities. More modest organizations are progressively accepting their function to actualize prescient investigation in their plan of action. The advancement of an altered set-up of prescient models for an organization isn't just capital-concentrated yet requires general labor and time. Showcasing organizations like "AgilOne" offer generally straightforward prescient model sorts with wide materialness across modern areas. They have distinguished three fundamental sorts of prescient models to dissect client conduct, which are:

"Inclination models" – These models are utilized to produce "valid or exact" expectations for client conduct. Probably the most well-known penchant models include: "prescient lifetime esteem," "inclination to purchase," "affinity to turn," "affinity to change over," "probability of commitment," and "inclination to withdraw."

"Bunch models" – These models are utilized to separate and gather clients dependent on shared characteristics, such as sex, age, buy history, and socioeconomics. The absolute most basic group models incorporate "item-based or class base bunching," "conduct customs grouping," and "brand based bunching."

"Communitarian separating" – These models are utilized to create items, administrations, and proposals just as to suggested notices dependent on earlier client exercises and practices. Probably the most widely recognized community sifting models incorporate "upsell," "strategically pitch," and "after sell" proposals.

Organizations' main apparatus to execute prescient examination on client conduct is "relapse investigation," which permits the organization to build up relationships between's offer of a specific item and the particular ascribes showed by the buying client. This is accomplished by utilizing "relapse coefficients," which are numeric qualities portraying how much the client's conduct is influenced by various factors and building up a "probability score" for the item's future offer.

Capability and Prioritization of Leads

There are three introductory classes utilized in business-to-business or B2B prescient examination promoting to qualify and organize planned clients or "leads."

These classifications are:

- "Predictive scoring" is utilized to organize forthcoming clients based on their probability to make a real buy

- "Identification models" are utilized to distinguish and get new imminent clients dependent on properties imparted to the organization's current clients.

- "Automated division" is utilized to isolate and characterize planned clients dependent on shared characteristics to be focused on the same customized advertising techniques and missions.

The prescient examination innovation needs a huge volume of deal information that fills in as a structure square and preparing material to increment the prescient models' exactness and proficiency. Little physical organizations can't bear to grow their figuring assets; consequently, they can't proficiently gather client conduct information from their in-store deals. This converts into a serious edge for the bigger organizations with a further developed registering framework, which fuels bigger organizations' pointless development in contrast with independent ventures.

Distinguishing Proof of Current Market Trends

Organizations can utilize "information representation" devices that permit business heads and administrators to assemble experiences on the organization's present status, basically by picturing their current client conduct information on a "report or dashboard." These dashboard reports will, in general, rouse and create client conduct driven activities. For instance, an organization can distinguish the basic client requests pattern in explicit areas with information representation devices and likewise plan to stock their stock for only stores. Similar data can uncover the best items and administrations for the organization to be dispatched depending on the current market drifts that can trick the client requests. The market pattern bits of knowledge can likewise be applied to expand its effectiveness gracefully chain the executives model.

Client Segmentation and Targeting

One of the least difficult and profoundly successful methods of streamlining an item offered to accomplish a fast turnaround on the organization's quantifiable profit is the capacity to target "right clients" with the suitable item offers at the "perfect time." This additionally turns out to be the most well-known and broadly utilized utilization of prescient investigation in the realm of advertising. As indicated by an exploration study directed by the "Aberdeen Group," organizations utilizing prescient investigation in their showcasing methodologies are multiple times bound to distinguish "high worth clients effectively." This is the place where the nature of the organization's current informational index comes first. The energetically prescribed practice utilizes chronicled buyer conduct information of every current client, investigates it to portion, and targets clients with comparable buying credits with a customized proposal and promoting efforts.

The absolute most basic prescient examination models utilized in this application are "liking investigation," "agitate examination," and "reaction demonstrating." Using these applications, organizations can assemble understanding, for example, "if consolidating advanced and print memberships of their item contributions or list is a smart thought" or "whether their item or administration will be more effective whenever offered as a month to month membership model or one-time buy charge." One of the major deals and showcasing stage organizations is "Salesforce," which offers a cloud-based stage that organizations can utilize to create client profiles due to the information gathered from free sources, including client relationships, the executives (CRM) applications, and other organization applications. By specifically and carefully adding inputted information to this stage, organizations can consistently follow their client conduct to continuously build up a client social model to take care of the organization's emotional cycle continuously and over the long haul.

Advancement of Marketing Strategies

Another use of prescient investigation and showcasing is giving admittance to an assortment of client-related information, such as information gathered from online media stages and organizations' inward organized information. The client conduct model would then be produced by examining all accessible information and applying "social scoring." All the organizations across various mechanical areas must adjust to changing or developing client conduct through multiplying promoting mediums or channels. For instance, organizations can utilize any of the proactive investigation models portrayed above to foresee if their arranged advertising effort would accomplish the online media stages or their versatile applications.

Organizations can utilize the prescient examination model to comprehend how their clients are interfacing with their items or administrations, in light of their sentiments or feelings shared on the online media stages concerning a specific theme.

Chapter 12. Data Mining and Applications

What’s the point of ads? They’re on our monitors, TV screens, smartphone displays, inside our favorite radio broadcasts, and mailboxes. No matter where we turn, we’ll find ads constantly hawking something we’re not interested in. Those ads represent the traditional shotgun approach where companies simply propel as many as they can in our general direction and hope at least one hits. As we can imagine, this kind of marketing costs a lot. Still, companies don’t know any better and keep pumping money into wacky, bizarre, and embarrassing ads, hoping anything works. Thanks to Machine Learning, we might be nearing a future where computers produce dirt-cheap ads that are scarily tailored to our behavior and applied at the exact moment when they’ll have the strongest effect. We might already be living in one such future.

One thing about consumer behavior is that most purchases are made automatically, but major life events can break these habits and put us on the cusp of trying new things. This means Fig Newtons ads aren’t necessarily aimed at people who’d never try Fig Newtons but at those who like sweets and might try something different because they’re undergoing a major life event, such as divorce, car purchase, or pregnancy. How does the advertising company know which person is which? Enter data mining, harvesting as much data about people to have computers try to predict their behavior, desires, and motivations to target them with just the right kind of ad at just the right moment. Of course, ads would never work for us, but machines can learn to be persuasive.

One thing to note here is that data mining processes are going to be used to help us build up Machine Learning models. These models that rely on Machine Learning can power up applications, including the recommendation programs found on many websites and the technology that can keep search engines running.

How Does Data Mining Work?

So, why is data mining such an important process to focus on? You will see the staggering numbers when it comes to the volume of data that is produced is doubling every two years. Just by looking at unstructured data on its own, but just because we have more of this information doesn’t mean that we have more knowledge all of the time. With the help of data mining, you can do some of the following tasks:

- Sift through all of the noise, whether repetitive or chaotic, is found in your data.

- You can better understand what is relevant in all of that information and then make good use of the information to assess what outcomes are the most likely for your needs.

- It can help you accelerate the pace of making decisions informed and driven by data and more likely to help your business thrive and grow.

Now that we have that figured out, it is time to look at how all of this data mining will work. We will not grab the data, these trends will show up with us having to do no more work on them, and this is where we will be able to work with data mining. Data mining is a great way for us to explore and analyze a large amount of information to find all of the insights, trends, and patterns that we can use out of that information.

For example, we can work with data mining to learn more about the opinions and the users' sentiment, help us learn how to properly detect fraud, help out with risk management, filter the spam out of email, and even with marketing. All of these are going to be important to many different kinds of companies, and when you use them properly, you will find that they are going to ensure that you can better serve your customers over time.

There are five basic steps that we will see when it comes to working with data mining. In the first step, the company will spend some time collecting the data they want to use, and then they will make sure that all of this will be loaded up properly to their data warehouse. When this is all done, the company can then store and manage the data. Sometimes, this is done on the company's in-house servers, and other times it is going to be sent to the cloud.

When we go through with this, the management teams, IT professionals, and even business analysts will gain access to this data. Then they can determine the way that they would like to organize all of this information. We can then work with application software to sort out the data based on the results that the user is going to put in. In the last step, our end-user will present their findings and all of that information in a certain format that makes the most sense, that will be easy for those in charge of making decisions to read through and understand.

While we are on this topic, we need to work on data warehousing and mining software. The different programs that you decide to use with data mining will be responsible for analyzing the patterns and the relationships that we can find in the data. All of this is going to be done based on the requests that the user sends out. A company may use this software to help them create some new classes on that information.

We can go through and illustrate this point a bit more, as well. Imagine that we are a restaurant that would like to work with all of the data mining steps to determine the right times to offer some specials. The restaurant would be able to do this by looking at all of the information they have been able to collect on the specials, and then see how the specials do at different times of the day and on different days of the week. They can then create classes based on when the customers visit and what the customer is most likely to order when they come to the restaurant to eat.

We can take this to the following level as well. In some cases, a data miner will find clusters of information based on a logical relationship, or they may take some time to see if there are sequential patterns and associations that they can draw some conclusions to learn more about their customers in the long run.

Warehousing is going to be another important part that we see in the data mining process. This is a pretty simple process to work with, and it is going to include when a company can centralize their data into one database or one program, rather than spreading out the information in more than one place. With the warehouse for data, an organization can spinoff some of the data segments for the right users to analyze regularly and for the specific users to gain access to when they need it.

However, there are also times when we will see that the analyst will take the work on a different course during this process. For example, the analyst may choose to start with some of the most useful data, and then they will be able to create their warehouse for the data based on the specifications there. No matter how a business wants to organize their data, they will use it to help support some of the decision processes that the company's management is going to make.

With this in mind, we also need to take some time to explore data mining examples along the way. A good example of this is grocery stores. Many of the supermarkets that we visit regularly give away free loyalty cards to customers. These are beneficial to the customers because it provides them with access to reduced prices and other special deals that non-members at that store will not be able to get.

This is a great way for both parties to win. The customer will enjoy that they can potentially save money so that they will sign up for it. The store will enjoy that they get a chance to learn more about the customers, set prices to bring in more people, and make them the most money possible.

Here, we need to keep in mind that there are a few concerns that data mining can bring up for the customer and a company. Some customers are concerned about this data mining process because they worry about the company not being ethical with their use. It could even be an issue with a legitimate and honest company because the sampling they use could be wrong, and then they will use the wrong kind of information to make their decisions.

Most companies need to take some caution when they decide to work with the data mining process to ensure that they will reach their customers better and do well in their industry through some useful insights and more that they can learn along the way. You need to focus on when learning what patterns and insights are found in all of that data.

All of this data is going to be important when it comes to working in the process of data science. But we have to make sure that we understand how this data is supposed to work and what is found inside of all that data. When we can learn all of this, we will find that it is easier than we may think to handle the data and work for our needs.

Unbalanced Data Set

Although imbalanced data is a common problem with datasets, there is no universal technique for dealing with this issue. Generally, when classifiers are fed imbalanced data, the classification output will be biased, resulting in always predicting the majority class and incorrectly classifying the minority class. Therefore, we need to detect when the output is biased and deal with this issue, to improve its accuracy. We will over-sample the minority class by employing the Synthetic Minority Over-Sampling Technique (SMOTE) and the stratified K-Fold cross-validation method for dealing with the class imbalance.

Conclusion

Machine learning is an exciting and rapidly evolving field. While mastery of the subject can involve many years of study, it is possible to get started quickly by gaining some basic familiarity with machine learning methods and goals.

Despite the mysterious aura surrounding the field, many of the machine learning methods are relatively simple mathematical tools that have been around for centuries. It is just now that they are being applied to the massive amounts of data, the so-called big data, that is being collected by companies and other large organizations.

Python is an excellent tool to use for learning about–machine learning. Python is a very simple programming language that most people can pick up rather quickly. Libraries have been developed for python that is specifically designed for machine learning. So it is easy for a developer to play around with the tools and solve simple machine learning problems.

The way to go forward is actually to practice and study more. Begin by going through any exercises that you can find that entail covering all of the major algorithms used in machine learning. Using both supervised and unsupervised learning is important, as anyone who wants to understand machine learning needs to become intimately familiar with both. You should also practice by using many of the standard algorithms like linear regression and k-nearest neighbors.

Something that I would suggest is to avoid getting trapped into only using generated test data. To enhance your learning and development, get a hold of real-world data sets that you can run your algorithms on to gain an even greater familiarity with the practice of data science.

Many people who are new to the concept of machine learning ask what specific educational credentials they need to get into the field. While there are some general guidelines, the truth is there are no specific rules. We can begin by saying that in all likelihood, anyone who is involved in a scientific or technical field of study would be in a position to get involved in machine learning. That certainly applies to electrical or computer engineers.

However, some people who might be better placed to get into machine learning are mathematicians experts in statistics and probability. Some crossover knowledge can be helpful, but in some ways, machine learning is a statistical field when it comes down to a data scientist's day-to-day practice. Certainly, a high level of knowledge of statistics and probability is helpful.

Since it is considered a crossover discipline background in computer science can be helpful. The ideal candidate would have a substantial computer science background that has also demonstrated a high-level education in statistics. The more advanced your education, the more deeply you can go into the field, including doing AI research and designing more advanced systems. If you are playing around with some models, you will not be designing machine learning systems for use in some new robotic systems. That will require advanced education in computer science.

However, there are varying roles and levels of machine learning. Those who study computer systems in business school provided that they have a good understanding of statistics will be well-suited for doing machine learning tasks as a data scientist at many companies. Simply analyzing customer data or internal company data for trends and patterns is not something that requires a deep understanding of artificial intelligence. Your role is to use machine learning tools available to extract the kind of information useful for the enterprise.

So, machine learning and data science are fields that have a wide range of complexity and application. There is virtually some level of expertise suitable for many different levels and types of education, background, and taste. It is a growing field for the future.

We have learned what machine learning is and how it is applied today by businesses to many different tasks. We learned that there is supervised, unsupervised learning and how they are different. We also learned the issues that might crop up with various tradeoffs in machine learning.

We also learned many of the major algorithms used in machine learning, including regression methods, k-nearest neighbor methods, and decision trees. A large part of building a solid and reliable machine learning system is selecting the most appropriate training data sets and the best algorithm for a given situation. This, in part, will be determined by your experience, and the more experience that you get practicing machine learning, the better you are going to be when it comes to selecting the right algorithms for a given problem.

We also saw how python could be used to implement some of the most common machine learning tasks. We used python for regressing, k-nearest neighbors, and other classification methods. We looked at the TensorFlow library, the Sciikit learn library, and the Numpy library. We also learned about Keras and saw how to build a neural network. The power of Numpy lies behind many of the tools used to build machine learning models with python.

So where to go from here? The first step is to keep learning. You should keep practicing by building more models and using different tools to build your models. However, there is more to machine learning than simply playing around with tools. You should read as many journals as you can and watch videos from reputable sources so that you can learn the theory and fundamentals that lie behind the concept of machine learning.

If you go further than this, it will largely depend on your current situation and your future goals. If you are already a working professional, you might not need to go to school and get a computer science degree. You might be learning the tools for the sake of practical application at your current job. If that is the case, then practice along with self-study is the best path forward for you, although of course, if you are willing and able to return to school to get an in-depth education on the subject, that is always an option.

For those who are just getting exposed to the field and looking to it to pick a career path, getting a college degree in computer science or a related subject is probably the best way forward for you, especially if you hope to attain employment. Data science and machine learning are not likely to be fields where too many people can get employment without some college degree in a related field. If possible, find a school that will let you concentrate on artificial intelligence and machine learning.

I would also advise taking many math classes that are focused on statistics and probability. Some “business acumen” is often advised, so it can’t hurt to take some management classes as well. This is recommended even though many technical types are not that enthusiastic about business school. You are not going there to become an MBA. Still, you should get some idea of business operations at a large corporation and learn about many business concepts like business intelligence, predictive analytics, and data mining, since these are useful concepts for corporations. They prefer people who understand this to join their team, ready to hit the ground running.

Computer engineering is a related field that can also be pursued, and you can even consider mechanical engineering. That might not come to mind right away, but remember that there is a lot of research in robotics in mechanical engineering. But remember that college is nothing more than an entry ticket. Machine learning is a very practical field, and many of the tools described are going to be used in the real world.

I hope that has stimulated your interest in machine learning and that it will help propel you to continue your education and development in this exciting area.

Table of Contents

Introduction........................................................................................................................5

Chapter 1. What Is Machine Learning?...................................................................11

Chapter 2. Giving the Computers the Ability to Learn From Data..................17

Chapter 3. Basic Terminology and Notations........................................................23

Chapter 4. Evaluating Models and Predicting Unseen Data Instances..........30

Chapter 5. Building Good Training Datasets..........................................................36

Chapter 6. Combining Different Models for Ensemble Learning.....................48

Chapter 7. Applying Machine Learning to Sentiments Analysis......................51

Chapter 8. Conditional or Decisional Statements.................................................57

Chapter 9. Functions.....................................................................................................63

Chapter 10. Actual Machine Learning Algorithms...............................................70

Chapter 11. Applications of the Machine Learning Technology......................75

Chapter 12. Data Mining and Applications.............................................................82

Conclusion