1. Pytorch Reproducibility
There is a official page talking about the logic of the reproducibility of PyTorch. Basically, the following snippet support the reproducibility:
1
2
3
4
5
6
7
8
9
10
11
12
13
import numpy as np
import random
import torch
seed = 0
random.seed(seed) # python random generator
np.random.seed(seed) # numpy random generator
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True) # check nondeterministic algorithms
Basically, the CUP version is easier to get reproductive results while GPU reproductibility is not certain.
2. Use latexdiff on Mac
Mac OS already has perl so only the latexdiff needs to be installed.
The eastiest way to install latexdiff is by homebrew:
1
2
3
brew install latexdiff
brew upgrade latexdiff
brew uninstall latexdiff
And use the commend to generate diff.tex:
latexdiff old.tex new.tex --flatten > diff.tex
Useful link:
- https://shannonmlocke.wordpress.com/2020/05/10/a-short-guide-to-using-latexdiff/
3. How to check memory leakage in pytorch
Useful link. Common causes:
- If I create an array that holds tensors and continually add tensors to the array, it will fill up the memory.
- If I compute something from tensors but do not get backpropogation, these tensors will not be cleared and keep growing.
- The issue can be solved by adding
.detach()to any tensor that does not need to be involved in training.
- The issue can be solved by adding
Conclusions:
torch.cuda.empty_cache()is just a bandaid which delay the out of the memory issue instead of solving it.- Use
torch.cuda.memory_allocated()andtorch.cuda.max_memory_allocated()to print the percentage of used memory at the top of the training loop.
4. How to use latexdiff on Mac
Use the following commands to install latexdiff and texlive.
1
2
brew install latexdiff
brew install texlive
Run the simple bash script. Or try this one if necessary: repo. Be careful about the table comparison since there is a bug in latexdiff. Recommand to use a separate file to avoid table comparison.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# latexdiff.sh
#!/bin/bash
latex_file_modify=("main.tex" "appendix.tex")
latex_file_origin=("main.tex" "appendix.tex")
latex_file_num=2
origin_dir="$1"
modify_dir="$2"
diff_dir="$3"
for((i=0;i<${latex_file_num};i++));
do
cp "${origin_dir}/${latex_file_origin[${i}]}" "${diff_dir}/origin_${latex_file_origin[${i}]}"
cp "${modify_dir}/${latex_file_modify[${i}]}" "${diff_dir}/modify_${latex_file_modify[${i}]}"
rm "${diff_dir}/${latex_file_modify[${i}]}"
cd "${diff_dir}"
latexdiff -t UNDERLINE "origin_${latex_file_origin[${i}]}" "modify_${latex_file_modify[${i}]}" > "${latex_file_modify[${i}]}"
rm "origin_${latex_file_origin[${i}]}"
rm "modify_${latex_file_modify[${i}]}"
cd ..
done
Finally, use the ./latexdiff.sh <origin_dir> <modify_dir> <diff_dir> to generate the diff tex files and upload to overleaf to see the results.
Notice that the tex files should be under the root folder instead of multi-level folders.
5. Conda Usage Tips
In Ubuntu environment, if we use conda create -n <env_name> to build a new python environment, the sys.path is still the default local machine and any package installed are in the local python path. If we want an isolated environment, use:
conda create-n <env_name> python instead.
We can use which python or sys.path to check the path.
Otherwise, several issues will appear:
AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'.AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'.
6. Uninstall all the python packages in an environment
1
2
conda activate <env name>
pip freeze | xargs pip uninstall -y
Issue: The problem of the command is that it will uninstall conda either.
After that, I found that conda is uninstalled either. So I copy the conda, conda-env, and conda-content-trust to ~/miniconda3/bin/ and revise the conda configure by editing vi ~/.condarc and update several pathes.
Check conda config: conda config --show.
7.How to use docker in Ubuntu?
- list all running containers:
docker ps - list all containers:
docker ps -a - stop all the containers:
docker rm $(docker ps -aq) - remove all the containers:
docker rm $(docker ps -aq) - remove all the images:
docker rmi $(docker images -q)