Disaster Predictors use NLP to classify statements as disasters or not. This is an introductory NLP project, and it introduces core concepts such as tokenization, embeddings, and different models.
Text to numbers:
Tokenization - A straight mapping from a word, character, or sub-word to a numerical value. There are three main levels of tokenization:
1.1) Using word-level tokenization with the sentence "ChatGPT uses NLP" might result in "ChatGPT" being 0, "uses" being 1, and "NLP" is 2. Every word in a sequence is considered a single token.
1.2) Character-level tokenization, such as converting the letters A-Z to values 1-26. In this case, every character in a sequence is considered a single token.
1.3) Sub-word tokenization is in between word-level and character-level tokenization. It involves breaking individual words into smaller parts and then converting those smaller parts into numbers. For example, "my favorite food is pineapple pizza" might become "my, fav, avour, rite, fo, oo, od, is, pin, ine, app, le, piz, za". After doing this, these sub-words would then be mapped to a numerical value. In this case, every word could be considered multiple tokens.
Embeddings - An embedding is a representation of natural language which can be learned. Representation comes in the form of a feature vector. For example, the word "India" could be represented by the 5-dimensional vector [-0.8547, 0.4559, -0.3332, 0.9877, 0.1112]. It's important to note here the size of the feature vector is tuneable. There are two ways to use embeddings:
2.1) Create your embedding - Once your text has been turned into numbers (required for an embedding), you can put them through an embedding layer (such as tf.keras.layers.Embedding), and an embedding representation will be learned during model training.
2.2) Reuse a pre-learned embedding - Many pre-trained embeddings exist online. These pre-trained embeddings have often been learned on large corpora of text (such as all of Wikipedia) and thus have a good underlying representation of natural language. You can use pre-trained embedding to initialize your model and fine-tune it to your specific task.
Naive Bayes is a probabilistic machine learning model based on Bayes' theorem, which provides a way of calculating the probability of an event occurring given the probabilities of certain related events. In the context of classification problems, a Naive Bayes model makes a prediction based on the maximum a posteriori (MAP) estimate of the class label, given the features in the input data.
There are three main variants of Naive Bayes: Gaussian, Multinomial, and Bernoulli. We use the pre-existing Multinomial Naive Bayes from the sklearn library as our baseline. Multinomial Naive Bayes is used for discrete count data.
The "naive" part of the name comes from the assumption that the data's features are independent, which is often not the case in real-world data. Despite this limitation, Naive Bayes models can still perform well in practice and are widely used in text classification, spam filtering, and sentiment analysis.
A feedforward neural network (FFNN) is an artificial neural network in which information flows in only one direction, from the input layer to the output layer, without looping back. In natural language processing (NLP), FFNNs are used for tasks such as text classification, named entity recognition, and part-of-speech tagging.
In an NLP-specific FFNN, the input layer represents the features extracted from the text data, such as word embeddings, bag-of-words representations, or sentence-level features. The hidden layers in the network process the input information and learn meaningful data representations. The output layer produces the final predictions, such as the class label for a text classification task.
FFNNs can be trained using various optimization algorithms and loss functions, such as stochastic gradient descent (SGD) and categorical cross-entropy. They are widely used in NLP due to their simplicity and versatility and have been found to perform well on many NLP tasks, especially when combined with techniques such as pre-training and fine-tuning.
For our first model, we create a basic feed-forward neural network with the following structure.
InputLayer -> TextVectorization -> Embedding -> GlobalAveragePooling1D -> Dense (Activation)
LSTM stands for Long Short-Term Memory, a Recurrent Neural Network (RNN) architecture used to process sequential data. LSTMs are designed to handle the problem of vanishing gradients in traditional RNNs, allowing them to model long-term dependencies in data more effectively. An LSTM cell consists of gates controlling the flow of information into and out of the cell, allowing it to selectively remember or forget information over time. This makes LSTMs well-suited for natural language processing, speech recognition, and time series forecasting tasks.
