documentation:ml:berzelius

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
documentation:ml:berzelius [2022/03/23 11:16] – [Working with Singularity] volkerkdocumentation:ml:berzelius [2022/09/02 14:04] (current) – external edit 127.0.0.1
Line 1: Line 1:
 ====== How to use Berzelius ====== ====== How to use Berzelius ======
 +
 +Note that some of the links below might require you to have a password at SNIC
  
 ==== Getting Started ==== ==== Getting Started ====
Line 5: Line 7:
   - below information is from memory and should be updated by the next person going through the process   - below information is from memory and should be updated by the next person going through the process
   - Get a login to SNIC: https://supr.snic.se.    - Get a login to SNIC: https://supr.snic.se. 
-  - You will be walked throught the process.+  - You will be walked through the process.
   - Note that you will have to accept the SNIC user agreements.    - Note that you will have to accept the SNIC user agreements. 
   - After every step you will receive an email from supr@supr.snic.se. Read the emails carefully, they will tell you what to do next.   - After every step you will receive an email from supr@supr.snic.se. Read the emails carefully, they will tell you what to do next.
   - At some point you will have to write a small project proposal: https://supr.snic.se/proposal/   - At some point you will have to write a small project proposal: https://supr.snic.se/proposal/
-  -   - Go to Rounds page +    - Go to Rounds page. 
-  -   - In the sub-menue, select AI/ML and then select LiU Berzelius +    - In the sub-menu, select AI/ML and then select LiU Berzelius. 
-  -   - Create the proposal for getting computation time on Berzelius+    - Create the proposal for getting computation time on Berzelius.
   - You will get a confirmation email from snic.se and NSC Berzelius and later a confirmation email that your project was accepted.   - You will get a confirmation email from snic.se and NSC Berzelius and later a confirmation email that your project was accepted.
-  -  Follow the guidelines in the emails. At some point you will have to accept another user agreement for using Berzelius. Once accepted your account is created.+  - To access Berzelius a login account is needed: 
 +    - Go to Accounts page and request an account for Berzelius. https://supr.snic.se/account/ 
 +    - Accept the Berzelius User Agreement. 
 +    - Wait for your account to be created. 
 +    - When your account is ready you will receive an email instructing you to choose a password.
   - https://www.nsc.liu.se/support/systems/berzelius-getting-started/   - https://www.nsc.liu.se/support/systems/berzelius-getting-started/
-  - before you can log in you need a 2-Factor Authorization (2FA): https://www.nsc.liu.se/support/2fa/migration/ +  - Before you can log in you need a 2-Factor Authorization (2FA): https://www.nsc.liu.se/support/2fa/migration/ 
-  -   - go through the section "How to enable 2FA for your cluster login account - detailed version"+  -   - Go through the section "How to enable 2FA for your cluster login account - detailed version"
   - Finally, you can run ''ssh berzelius.nsc.liu.se''. You get asked for the password and then the 6-digit number from the authenticator account.   - Finally, you can run ''ssh berzelius.nsc.liu.se''. You get asked for the password and then the 6-digit number from the authenticator account.
   - ... and you should be in.   - ... and you should be in.
Line 24: Line 30:
   * https://www.nsc.liu.se/support/systems/berzelius-getting-started/   * https://www.nsc.liu.se/support/systems/berzelius-getting-started/
   * To not have to put in your password on every login, use an SSH key: https://www.nsc.liu.se/support/security/   * To not have to put in your password on every login, use an SSH key: https://www.nsc.liu.se/support/security/
-  * Do not work on the shell you arrive at, it is a shared resource. Use the commant ''interactive -n 1'' go get your own CPU. Work from there. Use ''exit'' to leave the interactive session and free that CPU for others and to return to the login shell. From there, you can log off or request another interactive session. +  * Do not work on the shell you arrive at, it is a shared resource. Use the command ''interactive -n 1'' go get your own CPU. Work from there. Use ''exit'' to leave the interactive session and free that CPU for others and to return to the login shell. From there, you can log off or request another interactive session. 
  
 ==== Working with Conda ==== ==== Working with Conda ====
  
 If you want to use conda:  If you want to use conda: 
-  *   - On your private workstation, use ''conda env export > environment.yml'' to get a description of your conda envionment. Use scp or rsync to copy ''environment.yml'' to berzelius.+  *   - On your private workstation, use ''conda env export > environment.yml'' to get a description of your conda envionment. Use scp or rsync to copy ''environment.yml'' to Berzelius.
   *   - run ''module load Anaconda/2021.05-nsc1'' to load the conda module. It is a good idea to add this to your .bashrc.   *   - run ''module load Anaconda/2021.05-nsc1'' to load the conda module. It is a good idea to add this to your .bashrc.
   *   - ''ln -s ~/.conda /proj/<your_project_dir>/users/$(id -un)'', don't forget to replace <your_project_dir> with your project id. In my case, it is "berzelius-2022-58".   *   - ''ln -s ~/.conda /proj/<your_project_dir>/users/$(id -un)'', don't forget to replace <your_project_dir> with your project id. In my case, it is "berzelius-2022-58".
-  *   - ''conda env create -f environment.yml'' will replicate your home conda environment with the same name, etc. Then, run ''conda activate...'' Note that libraries you have installed with pip are not installed. You have to do that manually. +  *   - ''conda env create -f environment.yml'' will replicate your home conda environment with the same name, etc. Then, run ''conda activate...''  
   *   - ''conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch''   *   - ''conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch''
   *   - If you need ai-gym and atari-games, use ''conda install -c conda-forge gym''.    *   - If you need ai-gym and atari-games, use ''conda install -c conda-forge gym''. 
Line 40: Line 46:
  
 ==== Working with Singularity ==== ==== Working with Singularity ====
 +
 +Install Singularity by following this guide: https://sylabs.io/guides/3.0/user-guide/quick_start.html
 +
 +Creating a simple Singularity image using a recipe file: https://sylabs.io/guides/3.0/user-guide/definition_files.html
 +  - Create and open a recipe file using ''vim Singularity.recipe''
 +  - Choose an bootstrap agent that will create the base OS you want to use and add the corresponding lines to the recipe file:
 +      * Bootstrap: docker
 +      * From: ubuntu:20.04
 +  - Create the %post section which will execute commands within the singularity container:
 +      * %post
 +      * apt -y update
 +      * apt -y install python3
 +      * apt -y install pip
 +      * pip3 install --upgrade pip
 +      * pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
 +  - Add a %files section if any files are to be used by the singularity container:
 +      * %files
 +      * main.py /
 +      * eval.py /
 +      * logs/ /
 +  - Specify in the %runscript section the standard script to be run:
 +      * %runscript
 +      * python3 main.py
 +  - Create a .sif file from the recipe by running ''sudo singularity build image.sif Singularity.recipe''
 +  - The script in the %runscript section can then be run with ''singularity run image.sif''
 +  - Other scripts can be run using ''singularity exec image.sif python3 eval.py''
 +
 +Using .sif file on Berzelius:
 +  - Upload .sif file to Berzelius using ''scp image.sif <username>@berzelius1.nsc.liu.se:/proj/<project name>/users/<username>''
 +  - Also upload all needed files to the same folder on Berzelius.
 +  - Log into Berzelius, request computing resources and change directory to ''/proj/<project name>/users/<username>''
 +  - Run script using, for example ''singularity exec --nv image.sif python3 main.py''
  
   * under investigation...   * under investigation...
  • documentation/ml/berzelius.1648034175.txt.gz
  • Last modified: 2022/09/02 14:04
  • (external edit)