python - Why can't I load from a specific filepath with gensim downloader?

Question

I'm trying to download then load a pre-trained model with the gensim downloader to/from a specific location. It looks like I can successfully specify the download location using gensim.downloader.BASE_DIR but then I can't figure out how to load from that location.

I'm currently downloading the model using:

import gensim.downloader as api

api.BASE_DIR= 'mnt/project/models'

This has worked so far - I can see the model being downloaded to the correct location - but then I can't access the model. This code:

model = api.load('glove-twitter-25')

Results in an error:

FileNotFoundError: [Errno 2] No such file or directory: '/mnt/user/gensim-data/glove-twitter-25/glove-twitter-25.gz'

Is there a way to specify the directory to load the model from?

question from:https://stackoverflow.com/questions/66056396/why-cant-i-load-from-a-specific-filepath-with-gensim-downloader

Answer 1

When an error reads, "FileNotFoundError: [Errno 2] No such file or directory: '/mnt/user/gensim-data/glove-twitter-25/glove-twitter-25.gz'", the 1st thing I'd check is: are you sure the file /mnt/user/gensim-data/glove-twitter-25/glove-twitter-25.gz exist?s

But more generally, the gensim.downloader:

1. exists chiefly to make it easier to fetch remote resources, & was never envisioned to point at local paths, for which "downloading" is a misnomer

2. is probably a bad idea anyway, for reasons I've presented to the Gensim project in a project Github issue

I think you can and should find the original source of such files, and download the raw data yourself, without any mysterious/unversioned code-shims that weren't in the software release.

You'll then have a better idea where the files land, what formats they're in, and what library routines can further massage them into the formats you need.