Changes between Initial Version and Version 1 of en/UsingDeptGPU

Timestamp:

Oct 17, 2024, 1:57:20 PM (9 months ago)

Author:

Ales Horak

Comment:

--

en/UsingDeptGPU

@@ +1 @@
+= How to use GPUs on the ML Department server =
+
+[[PageOutline(2-3)]]
+
+The ML Dept server hostname is `akeso.fi.muni.cz`.
+
+Follow these actions:
+
+1. `ssh akeso`[[br]]
+   If you are not currently inside the faculty network, you must [#ConnectingtoFIMUNIVPN connect to the FI MUNI VPN] first.
+1. `cd /nlp/projekty/...` or `cd /cyb-trials/...` to your project directory
+1. setup !HuggingFace cache as detailed [#ChangethecachelocationofHuggingFaceDatasets below]
+1. find available GPUs: run `gpustat` or `nvidia-smi`
+2. `export CUDA_DEVICE_ORDER=PCI_BUS_ID`
+3. choose a GPU with no running processes
+4. limit your process to only selected GPU:
+   `export CUDA_VISIBLE_DEVICES=GID` where GID is the ID of the selected GPU (the first column in the `gpustat` or `nvidia-smi` listing).
+
+== Connecting to a GPU server ==
+==== Connecting to FI MUNI VPN ====
+- Download the VPN configuration for FI MUNI from here: https://www.fi.muni.cz/tech/unix/vpn.html
+- Use your faculty login `x.....` with your secondary password as the credentials
+
+==== Connect to the server ====
+* Run command (replace `...` with the server name followed by a dot `.`):
+{{{
+ssh xsurname@....fi.muni.cz
+}}}
+* Type in the secondary password again
+
+== Change the cache location of !HuggingFace Datasets ==
+
+Keep your home directory small, there are quotas active.
+
+* find out your project directory `PROJECT_DIR`. It is located in one of the 
+  NFS shares, i.e. `/nlp/projekty/...` or `/cyb-trials/...`. In the following instructions, we assume an environment variable `PROJECT_DIR` is set to your actual project directory, e.g. via
+  {{{
+export PROJECT_DIR=/nlp/projekty/...  # or
+export PROJECT_DIR=/cyb-trials/...
+}}}
+* You need to change the download location of datasets. The default location is in your home and it can easily get full
+* prepare a cache directory in your project
+{{{
+mkdir $PROJECT_DIR/hf_cache
+}}}
+* To change the HF Datasets location run:
+{{{
+export HF_DATASETS_CACHE="$PROJECT_DIR/hf_cache"
+}}}
+* To make this change permanent so you don't need to type it every time you enter the server, add it to your `.bashrc` file in your home directory (including the setting for `PROJECT_DIR`).
+
+== Create virtual environment for your project ==
+{{{
+ssh akeso
+cd $PROJECT_DIR
+virtualenv myenv
+source myenv/bin/activate
+}}}
+This `myenv` activation must be done before any package is installed via `pip`/`pip3` or the `python`/`python3` command is run with your program.
+
+Alternatively, you may run these commands with full path to your `myenv` and its `bin` subdirectory:
+{{{
+$PROJECT_DIR/myenv/bin/pip install ...
+$PROJECT_DIR/myenv/bin/python my_program.py
+}}}
+
+The created environment may be dependent on the exact server setup and possibly may not work well at another server.
+
+== Run jupyter notebook on a GPU server ==
+
+==== Connect your virtual env with the Jupyter notebook ====
+* after [#Createvirtualenvironmentforyourproject activating] your `myenv` environment install `ipykernel`
+{{{
+pip install ipykernel
+python -m ipykernel install --user --name=myenv
+}}}
+ You should get the output:
+{{{
+Installed kernelspec myenv in /home/user/.local/share/jupyter/kernels/myenv
+}}}
+* To check all your connected venvs
+{{{
+jupyter kernelspec list
+}}}
+
+==== Run Jupyter notebook on a personally selected port ====
+{{{
+jupyter notebook --no-browser --port 2222
+}}}
+The output should contain one line in the form
+{{{
+http://localhost:2222/?token=...
+}}}
+You will use it later, after setting up an SSH tunnel.
+
+==== Create SSH Tunnelling ====
+* Open new second terminal and insert the following command
+{{{
+ssh xsurname@akeso.fi.muni.cz -NL 2222:localhost:2222
+}}}
+ The second port must match the port selected when starting the jupyter notebook
+
+==== Open Jupyter notebook on the local machine ====
+* Point your browser to a the previously obtained address of the notebook 
+  http://localhost:2222?token=... [[br]]
+  If you have used the same port numbers, as we do, you do not need to change anything in the URL.
+  Otherwise, the port must match the first port in the [#CreateSSHTunnelling ssh tunelling] step.
+* If you have correctly copied the `token`, then you are logged in. 
+  Otherwise, to get the token again run:
+  {{{
+ssh akeso
+jupyter notebook list
+}}}
+  Copy the jupyter notebook URL and open it in your local browser.
+
+  The notebook is opened with listing the directory of your project.
+
+== References used ==
+1. SSH tunell: https://medium.com/@apbetahouse45/how-to-run-jupyter-notebooks-on-remote-server-part-1-ssh-a2be0232c533
+2. FI MUNI VPN: https://www.fi.muni.cz/tech/unix/vpn.html
+3. Jupyter notebook with virtualenv: https://janakiev.com/blog/jupyter-virtual-envs/
+4. !HuggingFace dataset location change: https://huggingface.co/docs/datasets/v1.12.0/cache.html
+