Tokenization is the process by which string is divided into smaller sub parts called tokens. Tokenization is the first step toward solving the problems like Text classification, sentiment analysis, smart chatbot etc using Natural Language toolkit. Natural Language toolkit has ‘Tokenizer Interface’, now this  tokenize module is further divided into sub parts word tokenize sentence … Continue reading Tokenization of Words and Sentences using NLTK