Migration

Introduction

We will soon transition to our new HPC cluster. During this migration, we will follow a structured approach to ensure a smooth transition for everyone. Users will be added in different waves; you will receive a message when it is your turn so that we can manage the increase of new users and provide support where needed.

Your responsibility

You are responsible for critically selecting and transferring your home and scratch folders yourself. Only move what you truly need.
After the migration, check whether you still have access to your project data and whether you are still in the correct groups. Request access in time if needed, or ask to have groups removed.
After the migration, permissions on the old cluster will change to “access only.” You will still be able to view your files, but you will no longer be able to edit them. Additionally, you will no longer be able to start Slurm jobs.

HPC team

The HPC team will handle copying the project folders only. During the initial phase, temporary differences may occur, but at the end of the migration, we will perform a final synchronization to fully align the project folders.

Please note

On the new cluster, each user will have 🚨 10 GB 🚨 available in their home folder.

If you have any questions or encounter any issues, please feel free to contact the HPC team.

Migration preperations

Preserving your Conda environments

On the new cluster you might want to preserve your existing Conda environments. In order to export your environments, follow these instructions:

Info

Run the following commands from your home directory

Show your environments
```
conda env list 
```
Create yml file of your enivorment
```
conda env export --name your-environment-name --file your-environment-name.yml
```
Note

Replace your-environment-name for the name of your environment and repeat this procedure for all your environments

Preparations for transfers

When transferring large datasets, copying can take several days to complete. To prevent the session from dropping during this time, we use tmux or screen. These tools keep the session active even if you disconnect from the server, allowing you to log out and later reattach to check the progress or resume your work.

Below are the steps to start such a session.

TMUX

Tmux is very convenient to keep something active in the background, even if your view session is closed.

Create a tmux session:
```
tmux new -s "my-session"
```
Exit a tmux session:
```
tmux detach
```
Tip

You can also use "CTRL+B" and then "D" (for detach)
List tmux session:
```
tmux ls
```
Attach a previously created session:
```
tmux attach -t "my-session"
```
Delete a tmux session:
```
tmux kill-session -t "my-session"
```

Tip

Enable scrolling up when in a tmux session: type CTRL+B and then [

SCREEN

Similar as tmux, screen allow you to keep session active even during view reboots.

Start new screen session:
```
screen -S "my_session"
```
Detach from session: Ctrl + A + D
List screens session:
```
screen -ls
```
Attach a session:
```
screen -r "my-session"
```

Migration of your data

Home

Archiving

Note

tar -czf home.tar.gz /trinity/home/<username_ares>/ --exclude='.conda'

Transfer

Note

rsync -P /trinity/home/<username_ares>/home.tar.gz <p-number>@helios.umcinfra.nl:/home/<p-number>/

Unpacking

Note

tar --strip-components=3 -xzf home.tar.gz -C /home/<p-number>/

Scratch

Archiving

Note

tar -czf scratch.tar.gz /net/beegfs/scratch/<username_ares>/

Transfer

Note

rsync -P /net/beegfs/scratch/<username_ares>/scratch.tar.gz <p-number>@helios.umcinfra.nl:/net/beegfs/users/<p-number>/

Unpacking

Note

tar --strip-components=3 -xzf scratch.tar.gz -C /net/beegfs/users/<p-number>/

Restoring your conda environments

Note

After having extracted your files from the home folder, also your-environment-name.yml is in your home directory.

To set it up:
```
conda env create --file your-environment-name.yml
```
Tip

Do this for every exported Conda environment file.