= How to use GPUs on the ML Department server =

[[PageOutline(2-3)]]

The ML Dept server hostname is `akeso.fi.muni.cz`.

Follow these actions:

1. `ssh akeso`[[br]]
   If you are not currently inside the faculty network, you must [#ConnectingtoFIMUNIVPN connect to the FI MUNI VPN] first.
1. `cd /nlp/projekty/...` or `cd /cyb-trials/...` to your project directory
1. setup !HuggingFace cache as detailed [#ChangethecachelocationofHuggingFaceData below]
1. find available GPUs: run `gpustat` or `nvidia-smi`
2. `export CUDA_DEVICE_ORDER=PCI_BUS_ID`
3. choose a GPU with no running processes
4. limit your process to only selected GPU:
   `export CUDA_VISIBLE_DEVICES=GID` where GID is the ID of the selected GPU (the first column in the `gpustat` or `nvidia-smi` listing).

Contact `kat_server@nlp.fi.muni.cz` if something does not work as expected.


== Connecting to a GPU server ==
==== Connecting to FI MUNI VPN ====
- Download the VPN configuration for FI MUNI from here: https://www.fi.muni.cz/tech/unix/vpn.html
- Use your faculty login `x.....` with your secondary password as the credentials

==== Connect to the server ====
* Run command:
{{{
ssh xsurname@akeso.fi.muni.cz
}}}
* Type in the secondary password again

== Change the cache location of !HuggingFace Data ==

Keep your home directory small, there are quotas active.

* find out your project directory `PROJECT_DIR`. It is located in one of the 
  NFS shares, i.e. `/nlp/projekty/...` or `/cyb-trials/...`. In the following instructions, we assume an environment variable `PROJECT_DIR` is set to your actual project directory, e.g. via
  {{{
export PROJECT_DIR=/nlp/projekty/...  # or
export PROJECT_DIR=/cyb-trials/...
}}}
* You need to change the download location of !HuggingFace models, datasets, .... The default location is in your home and it can easily get full
* prepare a cache directory in your project
{{{
mkdir $PROJECT_DIR/hf_cache
}}}
* To change the HF cache location run:
{{{
export HF_HOME="$PROJECT_DIR/hf_cache"
}}}
* To make this change permanent so you don't need to type it every time you enter the server, add it to your `.bashrc` file in your home directory (including the setting for `PROJECT_DIR`).

== Create virtual environment for your project ==
{{{
ssh akeso
cd $PROJECT_DIR
virtualenv myenv
source myenv/bin/activate
}}}
This `myenv` activation must be done before any package is installed via `pip`/`pip3` or the `python`/`python3` command is run with your program.

Alternatively, you may run these commands with full path to your `myenv` and its `bin` subdirectory:
{{{
$PROJECT_DIR/myenv/bin/pip install ...
$PROJECT_DIR/myenv/bin/python my_program.py
}}}

The created environment may be dependent on the exact server setup and possibly may not work well at another server.

== Run jupyter notebook on a GPU server ==

==== Connect your virtual env with the Jupyter notebook ====
* after [#Createvirtualenvironmentforyourproject activating] your `myenv` environment install `ipykernel`
{{{
pip install ipykernel
python -m ipykernel install --user --name=myenv
}}}
 You should get the output:
{{{
Installed kernelspec myenv in /home/user/.local/share/jupyter/kernels/myenv
}}}
* To check all your connected venvs
{{{
jupyter kernelspec list
}}}

==== Run Jupyter notebook on a personally selected port ====
{{{
jupyter notebook --no-browser --port 2222
}}}
The output should contain one line in the form
{{{
http://localhost:2222/?token=...
}}}
You will use it later, after setting up an SSH tunnel.

==== Create SSH Tunnelling ====
* Open new second terminal and insert the following command
{{{
ssh xsurname@akeso.fi.muni.cz -NL 2222:localhost:2222
}}}
 The second port must match the port selected when starting the jupyter notebook

==== Open Jupyter notebook on the local machine ====
* Point your browser to a the previously obtained address of the notebook 
  http://localhost:2222?token=... [[br]]
  If you have used the same port numbers, as we do, you do not need to change anything in the URL.
  Otherwise, the port must match the first port in the [#CreateSSHTunnelling ssh tunelling] step.
* If you have correctly copied the `token`, then you are logged in. 
  Otherwise, to get the token again run:
  {{{
ssh akeso
jupyter notebook list
}}}
  Copy the jupyter notebook URL and open it in your local browser.

  The notebook is opened with listing the directory of your project.

== References used ==
1. SSH tunell: https://medium.com/@apbetahouse45/how-to-run-jupyter-notebooks-on-remote-server-part-1-ssh-a2be0232c533
2. FI MUNI VPN: https://www.fi.muni.cz/tech/unix/vpn.html
3. Jupyter notebook with virtualenv: https://janakiev.com/blog/jupyter-virtual-envs/
4. !HuggingFace dataset location change: https://huggingface.co/docs/datasets/v1.12.0/cache.html