Tensorflow Learning

1. Start Up

The aim of this chapter is to install IPython and tensorflow packages by Anaconda.

Installing Jupyter methods: There are two methods listing on the page, but the first one which used Anaconda is recommanded because when you want to delete the whole things the pip way just sucks.

When you want to uninstall Jupyter, the conda way uses the following command easily: conda uninstall jupyter notebook. But the pip way needs to use pip list |grep Jupyter and then uninstall the result by hand.

Install Anaconda: Using the curl command is the fastest way to get the installer, or the installer is avaiable on the official website.

curl -O https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-x86_64.sh

You can check the installer by the sha256 with the sum on the official website:

sha256sum Anaconda3-5.2.0-Linux-x86_64.sh

Then run the script:

bash Anaconda3-5.2.0-Linux-x86_64.sh

Once the ouput shows the following information:

 installation finished.
 Do you wish the installer to prepend the Anaconda3 install location
 to PATH in your /home/sammy/.bashrc ? [yes|no]
 [no] >>>   Input `Yes` to use `conda` command everywhere. And activate the configuration with the following command:

source ~/.bashrc

Common usage of conda:

 conda list # list the packages
 conda info -envs # list the environments
 conda create --name my_env python=3.6 # create an environment with python 3.6
 source activate my_env # activate the environment
 source deactivate # deactivate
 conda remove --name my_env --all # delete the environment
 conda update conda/anaconda # update conda and anaconda distribution version
 conda install anaconda-clean # delete anaconda
 anaconda-clean
 rm -rf ~/anaconda3 # need to delete the path in ~/.bashrc as well

Install Jupyter: After created the environment by the command:

conda create -n tensorflow pip python=2.7 # or python=3.6

activate the environment and then install Jupyter and TensorFlow. After that, you need to set the config for running a notebook server so you can just use your browser to control the Jupyter. The information is provided here. For me, the easiest way to configure it is like that:

 $ jupyter notebook --generate-config
 $ vim ~/.jupyter/jupyter_notebook_config.py
 # delete the # before the following lines
     c.NotebookApp.ip = '*'
     c.NotebookApp.notebook_dir = '/home/User/jupyter'
     c.NotebookApp.open_browser = True

After that, you can use <IP address>:8888 on your browser to connect the Jupyter.

Install Tensorflow(After activated the environment):

$ pip install --ignore-installed --upgrade tfBinaryURL

tfBinaryURL is available on the page. The following command may be needed to be updated.

pip install --ignore-installed --upgrade https://download.tensorflow.google.cn/linux/cpu/tensorflow-1.8.0-cp36-cp36m-linux_x86_64.whl
Now when you use jupyter notebook and new your first file try to import tensorflow, it is still not available. The reason is that the jupyter package is installed on conda dirctory but tensorflow is installed in your environment. So the method to solve the problem is to install the jupyter package in your environment and add the environment path to the .bashrc file.
1 ```shell source activate my_env pip install jupyter vim ~/.bashrc
export PATH=”/home/user/anaconda3/bin:$PATH” # replace the line export PATH=”/home/user/anaconda3/envs/my_env/bin:/home/user/anaconda3/bin:PATH”source ~/.bashrc``
Everything has done! Explore the TensorFlow by Jupyter!

Error

If you get the warning like that:

/home/jxu/anaconda3/envs/tensorflow/lib/python3.6/importlib/_bootstrap.py:219: 
RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88

It means the numpy and the tensorflow version are conflict. So the solution is to use the commands to change the numpy version after you activate the anaconda environment.

        
sudo pip uninstall numpy
sudo pip install numpy==1.14.5

Then reactivate the environment and the warning disappear.

2. Tensorboard

Tensorboard is a very useful software which can provide five types of visualizations: scalars, images, audio, histograms and graphs.

Step 1: Serializing the data

Tensorboard operates by tensorflow events files. The following functions can be used to record the information.

tf.summary.scalar: collect the information of the variable changes
tf.summary.histogram: visualize the distributions of gradients or weights
tf.summary.merge_all: manage all the summary nodes
tf.summary.FileWriter: write the summary data to disk

Step 2: Launch the Tensorboard

Before runing Tensorboard, generate summary data in a log directory by the following code snippets:

# sess.graph contains the graph definition
file_writer = tf.summary.FileWriter('/path/to/logs', sess.graph)

And then, run the tensorboard:

tensorboard --logdir=path/to/log-directory

Step 3: Graph Visualization

use tf.name_scope('') to push a name scope in the graph. The better your name scopes, the better your visualization.

Tensorflow graphs have 2 kinds of connections: data dependencies and control dependencies.

data dependencies: solid lines
control dependencies: dotted lines

For networks with long sequences, sequential motifs that nodes whose names differ by a numer at the end and have isomorphic structures are collapsed into a single stack od nodes like range[1-8].

3. A simple example with RNN

Some key terminologies in neural network when reading the code:

batch_size: the number of training examples in one forward/backward pass
epoch: one forward pass and one backward pass of all the training examples
number of iterations: number of passes, each pass using [batch size] number of examples

Example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.

4. Data Preprocessing

Data Normalization

Normalization is an example of preprocessing data to remove or reduce the burden from machine learning (ML) to learn certain invariants, that is, things which make no difference in the meaning of the symbol, but only change the rep- resentation. So the data redundancy is reduced.

5. Cross Validation

Cross validation is primarily used to estimate the skill of a machine learning model on unseen data. That is, to use a limited sample in order to estimate how the model is expected to perform in general when used to make predictions on data not used during the training of the model. It is a popular method because it is simple to understand and because it generally results in a less biased or less optimistic estimate of the model skill than other methods, such as a simple train/test split.

Stanford CS231n

Computer Vision

The whole study materials in the blog are from Stanford official websites.

1. Simple algorithms with basic concept in machine learning

The simple and core problems in Computer Vision is Image Classfication. The image can be reshaped to a matric with three colors channels Red, Green, Blue. But there are a lot of changes of the same object in different pictures based on the viewpoint variation, scale variation, deformation, background clutter and so on. The approach to solve the problem is the using of data-driven.

k-Means Neighbor

The first algorithm is Nearest Neighbor Classifier. It compares the mertics of different images and decides the result on the distance of them. The distance formula which is a reasonable choice is the following formula which is called $L1$ distance: $d_1(I_1, I_2) = \Sigma_p|I_1^p - I_2^p|$

The imporved algorithm is k-Nearest Neighbor Classifier which will find the top k closest images instead of the closest one.

cross-validation

Not just split the data into training set and validation set, the training set is divided into 5 equal folders and use 4 of themfor training and 1 for validation. And iterate over which folder is the validation set.

hyper-parameters

They are associated with the classifier and do not have a good way to choose the bset value.

loss/cost function

The loss function can calcuate the distance between the true aim and the training result which measure the outcomes accurate.

The following formula is the Multiclass Support Vector Machine(SVM) loss.

\[L_i = \Sigma_{j\neq y_i}max(0, s_j - s_{y_i} + 1)\]

The softmax classifier’s cross-entropy loass form:

\[L_i = -log\Big(\frac{e^{f_{}y_i}}{\Sigma_j e^{f_j}}\Big)\]

backpropagation

Backpropagation is a way of computing gradients of expressions through recursive application of chain rule.