The SkimLit project attempts to replicate the deep learning model implemented by the *PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts* in 2017. The goal is to convert a medical paper into an abstract. We are working on an NLP classification problem. The notebook with the deep learning models can be found in the GitHub repository.
We have a get_lines function that reads a text file and returns the lines of text as a list and a preprocess_text_with_line_numbers function that reads the contents of a file and sorts through each line, extracting things like the target label, the text of the sentence, how many sentences are in the current abstract and what sentence number the target line is. We also numerically label and encode the words. In NLP, categorical labels can represent the sentiment of a text (e.g., positive, negative, neutral), the type of named entity (e.g., person, organization, location), or the genre of a text (e.g., news, fiction, scientific). These categories are assigned numerical values in numerical labelings, such as 0, 1, 2, etc. This allows the model to use mathematical operations on the labels, making it easier to train and optimize the model.
Naive Bayes is a probabilistic machine learning model based on Bayes' theorem, which provides a way of calculating the probability of an event occurring given the probabilities of certain related events. In the context of classification problems, a Naive Bayes model makes a prediction based on the maximum a posteriori (MAP) estimate of the class label, given the features in the input data.
There are three main variants of Naive Bayes: Gaussian, Multinomial, and Bernoulli. We use the pre-existing Multinomial Naive Bayes from the sklearn library as our baseline. Multinomial Naive Bayes is used for discrete count data.
The "naive" part of the name comes from the assumption that the data's features are independent, which is often not the case in real-world data. Despite this limitation, Naive Bayes models can still perform well in practice and are widely used in text classification, spam filtering, and sentiment analysis.
A Convolutional Neural Network (CNN) combined with token embeddings is a type of neural network architecture used in natural language processing (NLP) tasks. In this architecture, the input to the network is a matrix of token embeddings, where each row represents a token in a sentence, and each column represents a dimension of the embedding.
The CNN layer performs a convolution operation on the input embeddings, using a set of filters to learn local patterns in the data. The filters slide over the input embeddings and compute dot products between the filter weights and the embeddings in a local window. The output of the convolution operation is a new feature map for each filter, capturing information about the local patterns in the data.
This information is then passed through a series of non-linear activation functions, such as ReLU, to introduce non-linearity into the model. The resulting feature maps are then usually max-pooled over time to obtain a fixed-length representation of the input sentence. This fixed-length representation can then be used as input to a fully connected layer to produce the final prediction.
CNNs with token embeddings have been used successfully in NLP tasks such as text classification, sentiment analysis, and named entity recognition. The combination of token embeddings and CNNs allows the model to learn both local and global patterns in the data, making it well-suited for NLP tasks where both the individual tokens and the overall context of the sentence are important for making predictions. The model architecture of the model we used looks like the following.
Input (text) -> Tokenize -> Embedding -> Layers -> Output (label probability)
Feature extraction transfer learning models for NLP involve using pre-trained token embeddings as a feature extractor for a new task. In this approach, the pre-trained token embeddings are treated as fixed features, and a new model is trained on top of these features to perform the target task.
Pre-trained token embeddings are learned from a large corpus of text data and represent each token (such as a word or a subword) as a dense vector in a high-dimensional space. These embeddings are typically trained on a task such as language modeling and capture the semantic and syntactic relationships between tokens. Using these pre-trained token embeddings as input features, the feature extraction transfer learning model can leverage the information learned from the large corpus of text data rather than learning the representations from scratch.
The feature extraction transfer learning model can be fine-tuned by training the additional layers added on top of the pre-trained embeddings on the target task while keeping the pre-trained embeddings fixed. This approach is particularly useful when the target task has limited annotated data. The feature extraction transfer learning model can leverage the information learned from the pre-training to help overcome the data scarcity problem.