Explain Tf-Idf and explain how to compute the same?

Tf-idf or term frequency-inverse document frequency, is a measure of weight often used in information retrieval and text mining techniques. This weight is a statistical measure which is used to evaluate the importance of a word to a document in a collection or corpus. The weight increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.

The computation is often done by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model.

Tf-idf can be successfully used for stop-words filtering in various subject fields including text summarization and classification.

Typically, the tf-idf weight is composed by two terms:

the first computes the normalized Term Frequency (TF), which is the number of times a word appears in a document, divided by the total number of words in that document;

the second term is the Inverse Document Frequency (IDF), defined as the logarithm of the number of documents in the corpus divided by the number of documents where the specific term appears.

TF: Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization:

TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).

IDF: Inverse Document Frequency, which measures how important a term is. While computing TF, all terms are considered equally important. However, it is known that certain terms, such as "is", "of", and "that"may appear a lot of times but have little importance. Thus, we need to weigh down the frequent terms while scale up the rare ones, by computing the following:

IDF(t) = loge(Total number of documents / Number of documents with term t in it).