

whatlies.language.GensimLanguage

This object is used to lazily fetch Embeddings or EmbeddingSets from a keyed vector file. These files are generated by gensim. This object is meant for retreival, not plotting.

Important

The vectors are not given by this library they must be download/created upfront. A potential benefit of this is that you can train your own embeddings using gensim and visualise them using this library.

Here's a snippet that you can use to train your own (very limited) word2vec embeddings.

from gensim.test.utils import common_texts
from gensim.models import Word2Vec
model = Word2Vec(common_texts, size=10, window=5, min_count=1, workers=4)
model.wv.save("wordvectors.kv")

You can also download pre-trained embeddings that are hosted by the gensim project.

import gensim.downloader as api
# To check what models are available
api.info()['models'].keys()
# To download the vectors
wv = api.load('glove-twitter-25')
# This is typically saved in `~/gensim/data` but you can also edit these
# vectors and save them someplace else if you'd like.
wv.save("glove-twitter-25.kv")

Note that if a word is not available in the keyed vectors file then we'll assume a zero vector. If you pass a sentence then we'll add together the embeddings vectors of the seperate words.

Parameters

Name	Type	Description	Default
keyedfile		name of the model to load, be sure that it's downloaded or trained beforehand	required

Usage:

> from whatlies.language import GensimLanguage
> lang = GensimLanguage("wordvectors.kv")
> lang['computer']
> lang = GensimLanguage("wordvectors.kv")
> lang[['computer', 'human', 'dog']]

__getitem__(self, query)

Show source code in language/_gensim_lang.py

    def __getitem__(self, query: Union[str, List[str]]):
        """
        Retreive a single embedding or a set of embeddings.

        Arguments:
            query: single string or list of strings

        **Usage**
        ```python
        > from whatlies.language import GensimLanguage
        > lang = GensimLanguage("wordvectors.kv")
        > lang['computer']
        > lang = GensimLanguage("wordvectors.kv")
        > lang[['computer', 'human', 'dog']]
        ```
        """
        if isinstance(query, str):
            if " " in query:
                return Embedding(
                    query, np.sum([self[q].vector for q in query.split(" ")], axis=0)
                )
            try:
                vec = np.sum([self.kv[q] for q in query.split(" ")], axis=0)
            except KeyError:
                vec = np.zeros(self.kv.vector_size)
            return Embedding(query, vec)
        return EmbeddingSet(*[self[tok] for tok in query])

Retreive a single embedding or a set of embeddings.

Parameters

Name	Type	Description	Default
query	Union[str, List[str]]	single string or list of strings	required

Usage

> from whatlies.language import GensimLanguage
> lang = GensimLanguage("wordvectors.kv")
> lang['computer']
> lang = GensimLanguage("wordvectors.kv")
> lang[['computer', 'human', 'dog']]

embset_similar(self, emb, n=10, lower=False, metric='cosine')

Show source code in language/_gensim_lang.py

    def embset_similar(
        self,
        emb: Union[str, Embedding],
        n: int = 10,
        lower=False,
        metric="cosine",
    ) -> EmbeddingSet:
        """
        Retreive an [EmbeddingSet][whatlies.embeddingset.EmbeddingSet] that are the most similar to the passed query.

        Arguments:
            emb: query to use
            n: the number of items you'd like to see returned
            metric: metric to use to calculate distance, must be scipy or sklearn compatible
            lower: only fetch lower case tokens

        Returns:
            An [EmbeddingSet][whatlies.embeddingset.EmbeddingSet] containing the similar embeddings.
        """
        embs = [
            w[0] for w in self.score_similar(emb=emb, n=n, lower=lower, metric=metric)
        ]
        return EmbeddingSet({w.name: w for w in embs})

Retreive an EmbeddingSet that are the most similar to the passed query.

Parameters

Name	Type	Description	Default
emb	Union[str, whatlies.embedding.Embedding]	query to use	required
n	int	the number of items you'd like to see returned	10
metric		metric to use to calculate distance, must be scipy or sklearn compatible	'cosine'
lower		only fetch lower case tokens	False

Returns

Type	Description
EmbeddingSet	An EmbeddingSet containing the similar embeddings.

score_similar(self, emb, n=10, metric='cosine', lower=False)

Show source code in language/_gensim_lang.py

    def score_similar(
        self,
        emb: Union[str, Embedding],
        n: int = 10,
        metric="cosine",
        lower=False,
    ) -> List:
        """
        Retreive a list of (Embedding, score) tuples that are the most similar to the passed query.

        Arguments:
            emb: query to use
            n: the number of items you'd like to see returned
            metric: metric to use to calculate distance, must be scipy or sklearn compatible
            lower: only fetch lower case tokens

        Returns:
            An list of ([Embedding][whatlies.embedding.Embedding], score) tuples.
        """
        if isinstance(emb, str):
            emb = self[emb]

        queries = self._prepare_queries(lower=lower)
        distances = self._calculate_distances(emb=emb, queries=queries, metric=metric)
        by_similarity = sorted(zip(queries, distances), key=lambda z: z[1])

        if len(queries) < n:
            warnings.warn(
                f"We could only find {len(queries)} feasible words. Consider changing `top_n` or `lower`",
                UserWarning,
            )

        return [(self[q], float(d)) for q, d in by_similarity[:n]]

Retreive a list of (Embedding, score) tuples that are the most similar to the passed query.

Parameters

Name	Type	Description	Default
emb	Union[str, whatlies.embedding.Embedding]	query to use	required
n	int	the number of items you'd like to see returned	10
metric		metric to use to calculate distance, must be scipy or sklearn compatible	'cosine'
lower		only fetch lower case tokens	False

Returns

Type	Description
List	An list of (Embedding, score) tuples.