If you need python libraries installed that we don’t have, or that are newer than the versions we currently have installed, you should be able to install everything you need on your own using a virtual environment.
Here are the steps needed to set up your own python virtualenv:
All of these assume your CS username is csmajor1
(i.e., when you execute the commands below, please change csmajor1
to your username).
Also, all of these are if you haven’t already done this. If you have, you can skip down to the next section (make the virtual env).
Fall2018 Note: all new accounts as of Fall 2018 already have these preliminary steps done, so you can skip to the next section.
Make a directory in /scratch/csmajor1
for your venvs, so you don’t go over your disk quota. If you use /local
instead of /scratch
, it will be faster, but local to the machine you are on (i.e., only accessible from that lab machine). Either works. Here are the commands for using scratch (again, assuming you are csmajor1):
mkdir /scratch/csmajor1/Envs
ln -s /scratch/csmajor1/Envs ~/Envs
If you already have a ~/.cache/pip
directory, move it to /scratch/csmajor1
and link it back. If not, just make one in /scratch/csmajor1
and link it back. Here are the commands:
mv ~/.cache/pip /scratch/csmajor1 # if you already have one, mv
ln -s /scratch/csmajor1/pip ~/.cache/pip # and link it back
or
mkdir /scratch/csmajor1/pip # if not, make one
ln -s /scratch/csmajor1/pip ~/.cache/pip # and link it back
Here’s how to make a virtual environment, called my_project
, that uses python 3:
export PYTHONPATH=
export WORKON_HOME=~/Envs
source /usr/local/bin/virtualenvwrapper.sh
mkvirtualenv -p /usr/bin/python3 my_project
At this point you have made and are in the virtual environment. Your prompt should have changed to include (my_project)
at the beginning.
Use pip
to install any packages you need! Use pip list
to see what’s installed. Here are some examples:
(my_project) pip install numpy
(my_project) pip install scipy
(my_project) pip install scikit-learn
(my_project) pip list
Package Version
---------------- -------
numpy 1.12.1
scikit-learn 0.18.1
scipy 0.19.0
(my_project) python
>>> import sklearn
>>> sklearn.__version__
'0.18.1'
At this point your virtualenv is ready to use. If you want, start using it and run your programs in it.
deactivate
to end the virtualenvIf you want to get back out of your virtualenv, you can deactivate it or just close/exit the terminal window.
(my_project) deactivate
$
workon
To get back into your virtualenv:
$ export PYTHONPATH=
$ export WORKON_HOME=~/Envs
$ source /usr/local/bin/virtualenvwrapper.sh
$ workon my_project
(my_project)
I sometimes put the above four commands in a file, so I just have to source
that one file to get into my virtual environment (I also added some CUDA environment variables, in case you’re using the GPUs):
$ cat goproj
export CUDA_HOME=/usr/local/cuda-10.2
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64
export PYTHONPATH=
export WORKON_HOME=/scratch/csmajor1/Envs
source /usr/local/bin/virtualenvwrapper.sh
workon my_project
$ source goproj
(my_project)
Note: if you are using cuda, we have different versions available. Some versions of tensorflow only work with specific cuda versions. Check the docs. We have cuda-9.0 and cuda-10.1 installed, in addition to 10.2 (the default), so you’re welcome to change the above “exports” to use cuda 9.0 or 10.1 if needed.
If you want to share your virtualenv with a lab partner, you just need to allow read-execute access to the Envs
directory and any files below that. All that’s stored in there is the software you installed (e.g., with pip
), so it’s OK to open up this directory for others to access. Don’t put your lab work in the Envs
directory!
Here’s a command to allow read and execute permission for everyone on the /scratch/csmajor1/Envs
directory, and any files and directories below that:
$ chmod -R a+rX /scratch/csmajor1/Envs
The R
is for recursive, and the a+rX
says grant all users read and execute access.
And if you made a goproj
file to source, and want to share that with partners, do this:
$ chmod 644 goproj
Here are a few pages with additional info on using python virtual environments: