Tricks Summary 2021

In the blog, I will summary some tricks I learned and hope it will be helpful for others.

Tensorflow

1. Removes the warnings on Tensorflow v1

I find two solutions while the first one remove all the warnings from the link.

# Method 1
import tensorflow as tf
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)

# Method 2
from tensorflow.python.util import deprecation
deprecation._PRINT_DEPRECATION_WARNINGS = False

2. Set a reproducible environment for tensorflow, numpy and random packages

link

import random
import tensorflow as tf
import numpy as np
SEED = 0
def set_seeds(seed=SEED):
    os.environ['PYTHONHASHSEED'] = str(seed)
    random.seed(seed)
    # tf.random.set_seed(seed)
    tf.random.set_random_seed(seed)# tf.random.set_seed()
    np.random.seed(seed)

def set_global_determinism(seed=SEED):
    set_seeds(seed=seed)

    os.environ['TF_DETERMINISTIC_OPS'] = '1'
    os.environ['TF_CUDNN_DETERMINISTIC'] = '1'

    tf.config.threading.set_inter_op_parallelism_threads(1)

3. Edit static graph in Tensorflow 1

When we have a file to store the pretrained model and we want to attack it directly, we can use graph_editor to change the output or input of the pretrained model. The doc is here. And the following code is an example.

tf.contrib.graph_editor.connect(sgv0, sgv1, disconnect_first=False)
# Connect the outputs of sgv0 to the inputs of sgv1.
inputs = tf.get_default_graph().get_tensor_by_name("input:0")

face_size = 160
ndots = 2
im = tf.placeholder(tf.float32, shape=(face_size, face_size, 3))
expanded = tf.expand_dims(im, 0);
dotted = tf.identity( expanded );
# ge.connect(ge.sgv(dotted.op), ge.sgv(inputs.op), disconnect_first=True)
ge.connect(ge.sgv(dotted), ge.sgv(inputs), disconnect_first=True)

4. Exporting the model to ONNX format

tutorial

5. Remove Warnings each time running the code

Sometimes successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero will show up which is pretty annoying.

for a in /sys/bus/pci/devices/*; do echo 0 | sudo tee -a $a/numa_node; done can be used to remove them.

Python

1. uninstall pip accidentally

As a normal user, the python commands will be stored in /home/$USER/.local/bin/. But this time I remove the pip under the path. So I need to reinstall it by the following commands from the link:

wget https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py --user

# Checkt the installed version
$HOME/.local/bin/pip3 -V

Docker

1. install docker on ubuntu 16.04

This tutorial also point out how to add a normal user to docker group so that he avoids to use sudo when using docker.

2. How to Debug and check log when running docker-compose up -d

# check the end of the log file
docker logs <app name> -ft
# shut down the containers
docker-compose down
# start the containers
docker-compose up -d

If the path is not writable, we need to change the volumes path in docker-compose.yml.

3. some basic commands

docker ps
docker images
docker run -i -t <image id>

Others

1. Mac Bluetooth is not available

Solutions:

• Reset the SMC: link

2. How to use wget to imitate the website download?

Solutions: use Chrome->View->Developer->Developer tools->Network Tab->Headers tab->Copy as cURL

link

3. OneNote is stuck on loading page

Potential solutions:

1. right click the notebook->check sync error->sync

4. Support two branches in the same machine

Used: link

git worktree add ../foo_hotfix

5. VS Code SFTP cannot save the files

Used: Solution

6. Change hostname on Mac

Used: link

7. Imgcat on iTerm2 cannot work on a remote server

Install the python package: pip install --user imgcat from the link.

8. Jupyter runs on the default python environment instead of the conda virtual environment

You can check by the commands:

import sys
sys.path

If the virtual environment pathes are not listed here, it means the environment is the machine default environment. Basically, you can install the jupyter kernel by these commands (link):

conda create -n py36 python=3.6
conda activate py36
conda install notebook ipykernel
ipython kernel install --user

Useful link.

python3 -m ipykernel install --user

9. Add a repository to the current repository with the whole project

We need to remove the cache of the sub repository: git rm -f --cached <path to the submodule>.

10. How to upload overleaf project to Arxiv?

• Add \pdfoutput=1 to the main.tex file. Set the bib file name as the same as the main.bib. Move all the main.tex, main.bib and related files on the root folder.
• Use the submit button and choose the arxiv on the overleaf.
• Remove all the .DS_Store.
• The pathes of each section tex files need to update manually.
• A good way to add authors and affiliations on overleaf: link.
• Important: Arxiv projects should be built alone. Do not combine them with the other conference submissions.

11. What should I do if the model has different outputs each time?

• Check the random seed for tensorflow. torch, numpy.
• Check the model dropout, norm layer.
• Checkout the random seed on layers.

Some useful links:

• Keras model training result is not the same as testing result: link.

12. Manager Jupyter Kernel

Basically, each time we use jupyter to run a jupyter server, we need to figure out which kernel is using.

This is a related blog: link.

Here are some commands that help us manager the jupyter kernels:

# list the kernels
jupyter kernelspec list
# add a kernel
ipython kernel install --name "local-venv" --user
# remove a kernel
ipython kernelspec remove <kernel name>