Feature Extraction in NLP
What is Feature Extraction?
One of the most important parts of data preprocessing is feature extraction, which is a process of reducing data dimensionality by modifying variables describing data such way, that created set of features (feature vector) describe data model accurately and overall in a direct way. Thanks to feature extraction data is easier to compare thus easier to analyze.
Feature extraction is the transformation of original data to a data set with a reduced number of variables, which contains the most discriminatory information. ~ Transient Electromagnetic-Thermal Nondestructive Testing, 2017
Other Data Preprocessing Techniques
Besides feature extraction data preprocessing includes:
- feature engineering - transforming raw data (which often is very redundant) into a dataset with more informative format.
- feature transformation - to improve algorithm performance with normalization, scaling and dealing with missing values.
- feature selection - removing unnecessary features settle a subset of the original features. Unlike feature extraction which transforms attributes (e.g. creates a linear combination of them etc.)
Why we need to use it?
Clarity of data is important to algorithm as it is to humans. Redundancy doesn’t allow us to focus on the important detail, machines work the same. The more concentrated the information in data is the better. It’s worth emphasizing that information is something different from data. It’s an abstract concept of useful value that data brings to the phenomena that we are analyzing.
Not every data is informative, and the role of every data analyst is to draw useful information from the given data. Many believe that well-performed feature extraction is the key to build an effective model.
What are the most common techniques in NLP?
Natural Language Processing requires creating tabular data from the raw text. It’s done many ways but the most common are:
- BoW (Bag of Words/Ngrams) - represent text that describes the frequency of each word in a document. New feature vector has a column for each word in a document, and each row has a number of occurrences of this word in a sample (sentence, paragraph, etc.)
- CBoW (Continuous Bag of Words) - extend BoW, where the model tries to predict learned word by surrounding words disregard the order.
- CSG (Continuous Skip Gram) - this model uses a current word to predict surrounding words, with higher wages for closer words. CBoW is faster, but skip-gram has better results for rare words.
- Word2vec - is a group of related models: CBoW (Continuous Bag of Words) and Continuous skip-gram, used to produce word embeddings
- GloVe (Global Vectors) - is an unsupervised learning model that used to obtain dense word vectors similar to Word2Vec. The basic method of the GloVe model is to first create a huge word-context co-occurence matrix consisting of (word, context) pairs such that each element in this matrix represents how often a word occurs with the context (which is a sequence of words).
- TF-IDF (Term frequency–inverse document frequency) - is a numerical statistic of the importance of each word in a document. TF divides the number of occurrences of each word in a document by the total number of words in the document. IDF is to downscale weights for words that occur in many documents in the corpus and are less informative than those that occur only in a smaller part of the corpus.
- Autoencoders - is an artificial neural network that encodes the input in an unsupervised way that output can be preserved. This algorithm basically learns to ignore the noise in input data. Its hidden form can be used as an encoding of the input with preserved all information that the second part of the network uses to recreate input data.
Python Implementation Essentials
BOW using CountVectorization - sklearn
from sklearn.feature_extraction.text import CountVectorizer corpus = [ 'This is the first document.', 'This document is the second document.', 'And this is the third one.', 'Is this the first document?', ] vectorizer = CountVectorizer() X = vectorizer.fit_transform(corpus) print(vectorizer.get_feature_names()) > ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this'] print(X.toarray()) > [[0 1 1 1 0 0 1 0 1] [0 2 0 1 0 1 1 0 1] [1 0 0 1 1 0 1 1 1] [0 1 1 1 0 0 1 0 1]]
TF-IDF using TfidfTransformer - sklearn
from sklearn.feature_extraction.text import TfidfTransformer tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts) X_train_tf = tf_transformer.transform(X_train_counts) X_train_tf.shape > (2257, 35788)
CBOW and CSG using *Word2Vec - genism
from gensim.models import Word2Vec # CBOW model cbow = Word2Vec(data, min_count = 1, size = 100, window = 5) # Skip Gram model sg = Word2Vec(data, min_count = 1, size = 100, window = 5, sg = 1)
Glove using EmbeddingTransformer - zeugma
from zeugma.embeddings import EmbeddingTransformer glove = EmbeddingTransformer('glove') X_train = glove.transform(corpus)
Autoencoder using layers and models - keras
from keras.layers import Dense, Flatten, Reshape, Input, InputLayer from keras.models import Sequential, Model def build_autoencoder(img_shape, code_size): # The encoder encoder = Sequential() encoder.add(InputLayer(in_shape)) encoder.add(Flatten()) encoder.add(Dense(code_size)) # The decoder decoder = Sequential() decoder.add(InputLayer((code_size,))) decoder.add(Dense(np.prod(in_shape))) decoder.add(Reshape(in_shape)) return encoder, decoder
20 Nov 2019 Mateusz Dorobek UPC - Human Language Engineering