In [1]:
import lime
import sklearn
import numpy as np
import sklearn
import sklearn.ensemble
import sklearn.metrics
from __future__ import print_function

Fetching data, training a classifier

In the previous tutorial, we looked at lime in the two class case. In this tutorial, we will use the 20 newsgroups dataset again, but this time using all of the classes.

In [2]:
from sklearn.datasets import fetch_20newsgroups
newsgroups_train = fetch_20newsgroups(subset='train')
newsgroups_test = fetch_20newsgroups(subset='test')
# making class names shorter
class_names = [x.split('.')[-1] if 'misc' not in x else '.'.join(x.split('.')[-2:]) for x in newsgroups_train.target_names]
class_names[3] = 'pc.hardware'
class_names[4] = 'mac.hardware'
In [3]:
print(','.join(class_names))

atheism,graphics,ms-windows.misc,pc.hardware,mac.hardware,x,misc.forsale,autos,motorcycles,baseball,hockey,crypt,electronics,med,space,christian,guns,mideast,politics.misc,religion.misc

Again, let's use the tfidf vectorizer, commonly used for text.

In [4]:
vectorizer = sklearn.feature_extraction.text.TfidfVectorizer(lowercase=False)
train_vectors = vectorizer.fit_transform(newsgroups_train.data)
test_vectors = vectorizer.transform(newsgroups_test.data)

This time we will use Multinomial Naive Bayes for classification, so that we can make reference to this document.

In [5]:
from sklearn.naive_bayes import MultinomialNB
nb = MultinomialNB(alpha=.01)
nb.fit(train_vectors, newsgroups_train.target)
Out[5]:
MultinomialNB(alpha=0.01, class_prior=None, fit_prior=True)
In [6]:
pred = nb.predict(test_vectors)
sklearn.metrics.f1_score(newsgroups_test.target, pred, average='weighted')
Out[6]:
0.83501841939981736

We see that this classifier achieves a very high F score. The sklearn guide to 20 newsgroups indicates that Multinomial Naive Bayes overfits this dataset by learning irrelevant stuff, such as headers, by looking at the features with highest coefficients for the model in general. We now use lime to explain individual predictions instead.

Explaining predictions using lime

In [7]:
from lime import lime_text
from sklearn.pipeline import make_pipeline
c = make_pipeline(vectorizer, nb)
In [8]:
print(c.predict_proba([newsgroups_test.data[0]]).round(3))

[[ 0.001  0.01   0.003  0.047  0.006  0.002  0.003  0.521  0.022  0.008
   0.025  0.     0.331  0.003  0.006  0.     0.003  0.     0.001  0.009]]
In [9]:
from lime.lime_text import LimeTextExplainer
explainer = LimeTextExplainer(class_names=class_names)

Previously, we used the default parameter for label when generating explanation, which works well in the binary case.
For the multiclass case, we have to determine for which labels we will get explanations, via the 'labels' parameter.
Below, we generate explanations for labels 0 and 17.

In [10]:
idx = 1340
exp = explainer.explain_instance(newsgroups_test.data[idx], c.predict_proba, num_features=6, labels=[0, 17])
print('Document id: %d' % idx)
print('Predicted class =', class_names[nb.predict(test_vectors[idx]).reshape(1,-1)[0,0]])
print('True class: %s' % class_names[newsgroups_test.target[idx]])

Document id: 1340
Predicted class = atheism
True class: atheism

Now, we can see the explanations for different labels. Notice that the positive and negative signs are with respect to a particular label - so that words that are negative towards class 0 may be positive towards class 15, and vice versa.

In [11]:
print ('Explanation for class %s' % class_names[0])
print ('\n'.join(map(str, exp.as_list(label=0))))
print ()
print ('Explanation for class %s' % class_names[17])
print ('\n'.join(map(str, exp.as_list(label=17))))

Explanation for class atheism
(u'Caused', 0.26601045845449439)
(u'Rice', 0.13658462676578711)
(u'Genocide', 0.13519771973974065)
(u'owlnet', -0.092585962025604027)
(u'certainty', -0.088001257903975866)
(u'Semitic', -0.085734572813977061)

Explanation for class mideast
(u'fsu', -0.056622666266163954)
(u'Luther', -0.051897643027619206)
(u'Theism', -0.049640283384298073)
(u'jews', 0.037819200983811155)
(u'Caused', -0.037513316854574666)
(u'Genocide', 0.027357407069969929)

Another alternative is to ask LIME to generate labels for the top K classes. This is shown below with K=2.
To see which labels have explanations, use the available_labels function.

In [12]:
exp = explainer.explain_instance(newsgroups_test.data[idx], c.predict_proba, num_features=6, top_labels=2)
print(exp.available_labels())

[0, 15]

Now let's see some the visualization of the explanations. Notice that for each class, the words on the right side on the line are positive, and the words on the left side are negative. Thus, 'Caused' is positive for atheism, but negative for christian.

In [13]:
exp.show_in_notebook(text=False)