What is Part-of-speech (POS) tagging ? It is a process of reading the text(corpus) in a language and assigning some specific token (Parts of Speech) to each word such as nouns, pronouns, verbs, adverbs, and so on. It is also called grammatical tagging or word-category disambiguation. What are challenges in Part-of-speech (POS) tagging ? This …
Machine learning is a concept which allows the machine to learn from the experience. In this we train our program using existing examples and data. We keep adjusting our program until our program become near to accurate. The basic of machine learning is to build algorithms that can receive input data and use statistical analysis …
Tokenization is the process by which string is divided into smaller sub parts called tokens. Tokenization is the first step toward solving the problems like Text classification, sentiment analysis, smart chatbot etc using Natural Language toolkit. Natural Language toolkit has ‘Tokenizer Interface’, now this tokenize module is further divided into sub parts word tokenize sentence …
What Is Natural Language Processing? Natural Language Processing (NLP) is defined as “it is the technology by using which we make the software capable to understand the human’s natural language”. It is a branch of artificial intelligence and it deals with the interaction between computers and humans using the natural language. Natural language processing tasks …
This is the second part of IoT An Introduction article, for detail please go to the first part http://mycloudplace.com/iot-an-introduction-part-1/ IoT An Introduction (Part 1) The IoT can change much about the world we live in from how we make purchases online, how we get gas or electricity for our homes and the way how we …
By now in 2019 Internet is an enormous global network which allows people to connect and communicate with each other by sending Email, instant messaging or by use websites to communicate or share data and it is the people who makes the internet, the data which we share basically is generated from client devices such …
Lazy Evaluation means that the execution will not start until an action is triggered. Transformations are lazy in nature meaning when we call some operation in RDD, it does not execute immediately. Now some advantages of this Lazy Evaluation in Spark: Increases Manageability :Using Apache Spark RDD lazy evaluation, users can freely organize their Apache …
Lazy Evaluation in Apache Spark and its Advantage Read More »
For better performance of spark application it is important to understand the resource allocation and the spark tuning process. This article help you to understand how to calculate the number of executors, executor memory and number of cores required for better performance of your application. Below is the sample spark submit command ./bin/spark-submit –class <class_name> …
Executor is a distributed agent that is responsible for executing tasks. Executors are managed by executor backend (ExecutorBackend is a pluggable interface that TaskRunners use to send task status updates to a scheduler). Executors reports heartbeat to HeartbeatReceiver RPC Endpoint on the driver. Executors provide in-memory storage for RDDs using via Block Manager. BlockManager is …
DRIVER : A Spark driver is a JVM process that hosts SparkContext for a Spark application. It is the master node in a Spark application. The driver (an application’s driver process) splits a Spark application into tasks. It also schedules them to run on executors. It’s driver responsibility to coordinated with workers and also manage …
Deep Understanding of SparkContext & Application’s Driver Process Read More »
