![](\"https://github.com/ming-zhao/Optimization-and-Learning/raw/master/figures/feed_forward.png\")
\n",
"\n",
"- We call it two-layer network because it is the number of layers of adaptive weights.\n",
"\n",
"- The network has no closed directed cycles so that outputs are deterministic functions of the inputs.\n",
"\n",
"The corresponding network function is\n",
"\n",
"\\begin{align*}\n",
"y_k(\\mathbf{x},\\mathbf{w}) = f \\overbrace{\\left( \\sum_{j=0}^{\\ell_2} w^{(2)}_{kj} h\\underbrace{\\left( \\sum_{i=0}^{\\ell_1} w^{(1)}_{ji} x_i\\right)}_{a_j}\\right)}^{a_k}\n",
"\\end{align*}\n",
"\n",
"The quantities $a_j$, $a_k$ are *activations*. $h(\\cdot)$ is called *activation* function which is differentiable and nonlinear.\n",
"\n",
"Due to symmetry, multiple distinct choices for the weight vector $\\mathbf{w}$ can give the same mapping function."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Keras\n",
"\n",
"Keras is a deep-learning framework for Python that provides a convenient way to define and train almost any kind of deep-learning model.\n",
"\n",
"Let's look at an example of applying neural network on the MINST dataset to classify handwritten digits.\n",
"\n",
"There is a set of 60,000 training images, plus 10,000 test images, assembled by the National Institute of Standards and Technology (NIST). Each image is a gray scale 28 $\\times$ 28 pixels handwritten digits. we’re trying to classify images into their 10 categories (0 through 9)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"training images:(60000, 28, 28), test images:(10000, 28, 28)\n"
]
}
],
"source": [
"from keras.datasets import mnist\n",
"(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
"print('training images:{}, test images:{}'.format(train_images.shape, test_images.shape))"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" show code\n",
" "
],
"text/plain": [
"\n", "\n", "We provide details on learning the XOR function by using a simple neural network.\n", "\n", "We have\n", "\n", "- input\n", "\n", "\\begin{align}\n", "\\mathbf{x} = \\left( \\begin{array}{l}\n", "0 \\\\\n", "0 \\\\\n", "1 \\\\\n", "\\end{array} \\right) \\text{ or }\n", "\\left( \\begin{array}{l}\n", "0 \\\\\n", "1 \\\\\n", "1 \\\\\n", "\\end{array} \\right) \\text{ or }\n", "\\left( \\begin{array}{l}\n", "1 \\\\\n", "0 \\\\\n", "1 \\\\\n", "\\end{array} \\right) \\text{ or }\n", "\\left( \\begin{array}{l}\n", "1 \\\\\n", "1 \\\\\n", "1 \\\\\n", "\\end{array} \\right) \\nonumber\n", "\\end{align}\n", "\n", "- hidden layer\n", "\n", "\\begin{align}\n", "a^1_{j} = {\\mathbf{w}^1_j}^T \\mathbf{x} \\text{, } \\mathbf{a}^1 = (a^1_{j} :\\forall j)^T \\text{ and } z_{j} = h(a^1_{j}) \\text{, } \\mathbf{z} = (z_{j}: \\forall j)^T \\nonumber\n", "\\end{align}\n", "\n", "- output\n", "\n", "\\begin{align}\n", "a^2_{k} = {\\mathbf{w}^2_{k}}^T \\mathbf{z} \\text{ and } y = y_k = h(a^2_{k}) \\nonumber\n", "\\end{align}\n", "\n", "Note that, in this case, $k = \\{1\\}$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Feed-Forward \n", "\n", "To apply stochastic gradient decent, we consider one instance and its error function\n", "\n", "\\begin{align}\n", "E_i = \\frac{1}{2} \\left( y - t \\right)^2 \\nonumber\n", "\\end{align}\n", "\n", "\\begin{align}\n", "\\mathbf{z} = h\\left({\\mathbf{W}^1}^T \\mathbf{x} \\right) \\text{ and } y = h\\left({\\mathbf{w}^2}^T \\mathbf{z}\\right) \\nonumber\n", "\\end{align}\n", "\n", "where\n", "\n", "\\begin{align}\n", "\\mathbf{W}^{1} = (\\mathbf{w}^1_{j} : \\forall j) \\nonumber\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Derivatives \n", "\n", "Let $h = \\tanh$.\n", "\n", "\\begin{align}\n", "\\sigma^\\prime(x) &= \\sigma(x) (1- \\sigma(x)) \\nonumber\\\\\n", "\\tanh^\\prime(x) &= 1 -\\tanh^2(x) \\nonumber\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Error Backpropagation \n", "\n", "\\begin{align}\n", "\\delta^2_{k} &\\equiv \\frac{\\partial E_i}{\\partial a^2_k} = \\frac{\\partial E_i}{\\partial y} \\frac{\\partial y}{\\partial a^2_k} = (y-t) h^\\prime(a^2_k) = (y-t) (1- h^2(a^2_k)) = (y-t) (1- y^2) \\nonumber \\\\\n", "\\delta^1_{j} &\\equiv \\frac{\\partial E_i}{\\partial a^1_{j}} = \\sum_k \\frac{\\partial E_i}{\\partial a^2_{k}} \\frac{\\partial a^2_{k}}{\\partial z_j} \\frac{\\partial z_j}{\\partial a^1_{j}} = h^\\prime(a^1_{j}) \\sum_{k} \\delta^2_{k} w^2_{k,j} = (1-z_j^2) \\sum_{k} \\delta^2_{k} w^2_{k,j} \\nonumber\n", "\\end{align}\n", "\n", "\\begin{align}\n", "\\frac{\\partial E_i}{\\partial \\mathbf{w}^2_k} & = \\frac{\\partial E_i}{\\partial a^2_k} \\frac{\\partial a^2_k}{\\partial \\mathbf{w}^2_k} = \\delta^2_{k} \\mathbf{z} \\nonumber \\\\\n", "\\frac{\\partial E_i}{\\partial \\mathbf{w}^1_j} & = \\frac{\\partial E_i}{\\partial a^1_j} \\frac{\\partial a^1_j}{\\partial \\mathbf{w}^1_j} = \\delta^1_{j} \\mathbf{x} \\nonumber\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code realize the neural network and backpropagation algorithm. The plot shows that the XOR function is correctly learned by our neural network." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "
![](\"https://github.com/ming-zhao/Optimization-and-Learning/raw/master/figures/k-fold_valid.png\")
\n",
"\n",
"Let's train the network for 300 epochs. To keep a record of how well the model did at each epoch, we save the per-epoch validation score log."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" show code\n",
" "
],
"text/plain": [
"![](\"https://github.com/ming-zhao/Optimization-and-Learning/raw/master/figures/relu_sigmoid.png\")
\n",
"\n",
"**Hidden unit** is a single dimension in the representation space of the layer. Each Dense layer with a relu activation implements the following chain of tensor operations:\n",
"\n",
"\\begin{align*}\n",
"\\text{output = relu (dot($W$, input) + $b$)}\n",
"\\end{align*}\n",
"\n",
"Having 16 hidden units means that the weight matrix W will have shape (input_dimension, 16), i.e. the dot product with W will project the input data onto a 16-dimensional representation space. \n",
"\n",
"Intuitively, the dimensionality of your representation space is **how much freedom you are allowing the network to have when learning internal representations**. \n",
"\n",
"Having more hidden units (a higher-dimensional representation space) allows your network to learn more complex representations, but it\n",
"\n",
"- makes your network more computationally expensive and \n",
"\n",
"- may lead to overfitting"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n", "\n", "The rmsprop optimizer is generally a good enough choice of optimizer whatever your problem.\n", "\n", "Since we are facing a binary classification problem, we end our network with a single-unit layer with a sigmoid activation. It is better to use the binary_crossentropy loss, although you have other viable choices, for instance, you could use loss function: mean_squared_error. But crossentropy is usually the best choice when you are dealing with models that output probabilities.\n", "\n", "If we want to configure the parameters of optimizer, or pass a custom loss function or metric function. This can be done by passing an optimizer class instance as the optimizer argument or passing function objects as the loss or metrics arguments:\n", "\n", "```python\n", "from keras import optimizers\n", "from keras import losses\n", "from keras import metrics\n", "\n", "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n", " loss=losses.binary_crossentropy,\n", " metrics=[metrics.binary_accuracy])\n", "```\n", "\n", "\n", "\n", "In order to monitor during training the accuracy of the model on data that it has never seen before, we will create a validation set by setting apart 10,000 samples from the original training data." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "
![](\"https://ml-cheatsheet.readthedocs.io/en/latest/_images/dropout.png\")
\n",
"\n",
"This technique may seem strange and arbitrary. Why would this help reduce overfitting? Geoff Hinton has said that he was inspired, among other things, by a fraud prevention mechanism used by banks -- in his own words: \"I went to my bank. The tellers kept changing and I asked one of them why. He said he didn’t know but they got moved around a lot. I figured it must be because it would require cooperation between employees to successfully defraud the bank. This made me realize that randomly removing a different subset of neurons on each example would prevent conspiracies and thus reduce overfitting\".\n",
"\n",
"The core idea is that introducing noise in the output values of a layer can break up happenstance patterns that are not significant (what Hinton refers to as \"conspiracies\"), which the network would start memorizing if no noise was present."
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" show code\n",
" "
],
"text/plain": [
"