NLP Tool Kit

A method called natural language processing trains a computer to comprehend written or spoken language. Humans talk with one another using a common language so that they may comprehend one another's viewpoints and provide the appropriate response. In an NLP system, a machine rather than a human makes the interaction, comprehension, and response.

Applications of NLP

• Information retrieval & Web Search
• Correction of grammatical errors
• Answering the queries
• Summarization of test
• Machine Translation
• Sentiment Analysis

How to install NLTK?

To install NLTK and use it in our Python programs, follow the below steps:

• Install using the command pip install nltk
• Import nltk
• To install packages use the download() method

Text Processing using NTLK

The first step in processing text using NLTK is Tokenization. Tokenizing is a process of breaking text into smaller parts i.e. paragraphs to sentences, sentences to words. There are two types of tokenizers.

• Sentence Tokenizer
• Word Tokenizer

Sentence Tokenizer

>>> sampletext= “Artificial Intelligence is sometimes called Machine Intelligence. It is intelligence demonstrated by machines”
>>> from nltk.tokenize import sent_tokenize
>>> sent_tokenize(sampletext)

Output: [‘Artificial Intelligence is sometimes called Machine Intelligence’, ‘It is intelligence demonstrated by machines’]

Word Tokenizer

>>> sampletext= “Artificial Intelligence is sometimes called Machine Intelligence”
>>> from nltk.tokenize import word_tokenize
>>> word_tokenize(sampletext)

Output: [‘Artificial’,’ Intelligence’,’ is’, ’sometimes’,’called’, ’Machine’,’ Intelligence’]

Stemming and Lemmatization using nltk

What is Stemming?

Stemming is the process of bringing words into the norm. There will be one root word and several spellings of that term. The root word for play, for instance, has variants such as plays, playing, play-area, etc. We can identify the root word of any variants via stemming.

The "PorterStemmer" algorithm is part of NLTK. This method finds the root word from the collection of tokenized words.

Example:

Output:
call
call
call
call

What is Lemmatization?

Lemmatization is the computational process of determining a word's lemma based on its meaning. The suffix is removed from the word during the stemming process. It removes either the word's beginning or finish. The process of lemmatization is seen as intelligent since the correct form may be determined by consulting a lexicon. Lemmatization hence helps to create better machine learning features.

Click here to learn Data Analytics in Hyderabad

Example to distinguish between Lemmatization and Stemming

Stemming Code

Output:

Stemming for tries is try
Stemming for cries is cry

Lemmatization code

Output:

Lemma for tries is try
Lemma for cries is cry

Click here to learn Artificial Intelligence in Bangalore

Watch Free Videos on Youtube

Find Synonyms From NLTK WordNet

WordNet is an NLP database with a collection of synonyms, antonyms, and brief definitions.

Example:

Antonyms from NLTK WordNet

Stop Words Removal

Stop words can be removed from the text before processing it. Stop words are to be removed from text data to remove noise from the data. It is one of the pre-processing steps in text processing.

Example:

Output: ['Find', 'frequency', 'word', 'text', 'file', '!']

Stop words like ‘of’, ‘from’, and ’a’ are removed from the text data.

Click here to learn Artificial Intelligence in Hyderabad, Machine Learning in Hyderabad, Machine Learning in Bangalore