Virtualenv

If you need python libraries installed that we don’t have, or that are newer than the versions we currently have installed, you should be able to install everything you need on your own using a virtual environment.

Here are the steps needed to set up your own python virtualenv:

preliminaries…

All of these assume your CS username is csmajor1 (i.e., when you execute the commands below, please change csmajor1 to your username).

Also, all of these are if you haven’t already done this. If you have, you can skip down to the next section (make the virtual env).

Fall2018 Note: all new accounts as of Fall 2018 already have these preliminary steps done, so you can skip to the next section.

make a directory for your virtualenvs

Make a directory in /scratch/csmajor1 for your venvs, so you don’t go over your disk quota. If you use /local instead of /scratch, it will be faster, but local to the machine you are on (i.e., only accessible from that lab machine). Either works. Here are the commands for using scratch (again, assuming you are csmajor1):

mkdir /scratch/csmajor1/Envs
ln -s /scratch/csmajor1/Envs ~/Envs

put pip cache on /scratch

If you already have a ~/.cache/pip directory, move it to /scratch/csmajor1 and link it back. If not, just make one in /scratch/csmajor1 and link it back. Here are the commands:

mv ~/.cache/pip /scratch/csmajor1          # if you already have one, mv
ln -s /scratch/csmajor1/pip ~/.cache/pip   # and link it back

or

mkdir /scratch/csmajor1/pip                # if not, make one
ln -s /scratch/csmajor1/pip ~/.cache/pip   # and link it back

make the virtual environment

Here’s how to make a virtual environment, called my_project, that uses python 3:

export PYTHONPATH=
export WORKON_HOME=~/Envs
source /usr/local/bin/virtualenvwrapper.sh
mkvirtualenv -p /usr/bin/python3 my_project

At this point you have made and are in the virtual environment. Your prompt should have changed to include (my_project) at the beginning.

install software with pip

Use pip to install any packages you need! Use pip list to see what’s installed. Here are some examples:

(my_project) pip install numpy
(my_project) pip install scipy
(my_project) pip install scikit-learn
(my_project) pip list 
Package          Version
---------------- -------
numpy           1.12.1     
scikit-learn    0.18.1     
scipy           0.19.0     
(my_project) python
>>> import sklearn
>>> sklearn.__version__
'0.18.1'

At this point your virtualenv is ready to use. If you want, start using it and run your programs in it.

when you are done, run deactivate to end the virtualenv

If you want to get back out of your virtualenv, you can deactivate it or just close/exit the terminal window.

(my_project) deactivate
$ 

when you want to work on it again, use workon

To get back into your virtualenv:

$ export PYTHONPATH=
$ export WORKON_HOME=~/Envs
$ source /usr/local/bin/virtualenvwrapper.sh
$ workon my_project
(my_project) 

I sometimes put the above four commands in a file, so I just have to source that one file to get into my virtual environment (I also added some CUDA environment variables, in case you’re using the GPUs):

$ cat goproj
export CUDA_HOME=/usr/local/cuda-10.2
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64
export PYTHONPATH=
export WORKON_HOME=/scratch/csmajor1/Envs
source /usr/local/bin/virtualenvwrapper.sh
workon my_project

$ source goproj
(my_project) 

Note: if you are using cuda, we have different versions available. Some versions of tensorflow only work with specific cuda versions. Check the docs. We have cuda-9.0 and cuda-10.1 installed, in addition to 10.2 (the default), so you’re welcome to change the above “exports” to use cuda 9.0 or 10.1 if needed.

sharing with partners

If you want to share your virtualenv with a lab partner, you just need to allow read-execute access to the Envs directory and any files below that. All that’s stored in there is the software you installed (e.g., with pip), so it’s OK to open up this directory for others to access. Don’t put your lab work in the Envs directory!

Here’s a command to allow read and execute permission for everyone on the /scratch/csmajor1/Envs directory, and any files and directories below that:

$ chmod -R a+rX /scratch/csmajor1/Envs

The R is for recursive, and the a+rX says grant all users read and execute access.

And if you made a goproj file to source, and want to share that with partners, do this:

$ chmod 644 goproj

Additional Docs

Here are a few pages with additional info on using python virtual environments:

• The Hitchhiker’s Guide to Python: Virtual Environments
• Real Python Virtualenvs Primer
• chmod

Back to SwatCS Help Docs