Python code for constructing word shift graphs is available via pip:
pip install shifterator
Producing Word Shift Graphs
To produce a word shift graph, you need two dictionaries of the form
word2freq where the keys are words and the values are the frequencies of those words in a text. With those in hand, you can produce a number of word shift graphs. Start by importing the shifts.
from shifterator import relative_shift as rs from shifterator import symmetric_shift as ss
The Shifterator package includes a number of lexicons that can be used to quickly compare the sentiment between two texts. Here we use the English labMT sentiment dictionary:
from shifterator import relative_shift as rs # Get a sentiment word shift sentiment_shift = rs.SentimentShift(reference=word2freq_1, comparison=word2freq_2, sent_dict_ref='labMT_English', sent_dict_comp='labMT_English') sentiment_shift.get_shift_graph()
You can also provide score dictionaries where keys are words and values are scores according to a particular dictionary.
We can also use measures of entropy to compare two texts and how the relative abundance and distribution of words affects how they differ.
# Get an entropy shift entropy_shift = rs.EntropyShift(reference=word2freq_1, comparison=word2freq_2, base=2 entropy_shift.get_shift_graph() # Get a Jensen-Shannon divergence shift from shifterator import symmetric_shift as ss jsd_shift = ss.JSDivergenceShift(system_1=word2freq_1, system_2=word2freq_2, base=2) jsd_shift.get_shift_graph()
See the Github page for more details of types of shifts that you can produce and parameters you can use to alter the word shift visualizations.