{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Using TensorFlow backend.\n" ] }, { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "base_dir = 'D:\\\\deep_learning\\\\text'\n", "%run ../initscript.py\n", "# %run ../display.py\n", "import pandas as pd\n", "import numpy as np\n", "import scipy.stats as st\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "from ipywidgets import *\n", "%matplotlib inline\n", "import tensorflow as tf\n", "tf.logging.set_verbosity(tf.logging.ERROR)\n", "\n", "import os\n", "import random\n", "import sys\n", "\n", "from keras import optimizers\n", "from keras import backend as K\n", "from keras import models\n", "from keras import layers\n", "from keras import initializers\n", "from keras import preprocessing\n", "from keras.utils import to_categorical, get_file\n", "\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Text Mining" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Deep learning models don't take raw text as input, they only work with numeric tersors. Vectorizing text is the process of transforming text into numeric tensors." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We break down text into different units. In particular, we convert a sentence into sequence of tokens, i.e., words or bag-of-words. $n$-grams is a group of $n$ or fewer consecutive words. Bag-of-words (or bag-of-$n$-grams) is a set of words (or grams) which are not necessary consecutive.\n", "\n", "To associate a vector with a token, one approach is one-hot encoding of tokens. One-hot encoding is the most basic way to turn a token into a vector which was applied to the IMDB and Reuters examples. It associates with a binary vector of size $N$, the size of the vocabulary, which is all-zeros except 1 for the i-th entry." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Word Embeddings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Word embedding is an approach to provide a dense vector representation of words (e.g. the cat is cute may be represented as [4,100,1,233]) that capture something about their meaning. The geometric relationships between word vectors should reflect the semantic relationships between these words. For example, 4 words are embedded on a 2-dimensional plane:\n", "\n", "With the vector representations we chose here, some semantic relationships between these words can be encoded as geometric transformations. For instance, the same vector allows us to go from cat to tiger and from dog to wolf : this vector could be interpreted as the \"from pet to wild animal\" vector. Similarly, another vector lets us go from dog to cat and from wolf to tiger, which could be interpreted as a \"from canine to feline\" vector.\n", "\n", "There are two ways to obtain word embeddings:\n", "\n", "- Learn word embeddings jointly with the main task such as document classification or sentiment prediction. In this setup, we would start with random word vectors, then learn word vectors as the weights of a neural network by using Embedding layer.\n", "\n", "- Load pre-computed word embeddings package which is obtained from a different machine learning task" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Learning Word Embedding" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider the IMDB movie review sentiment prediction. We\n", "\n", "- restrict the movie reviews to the top 10,000 most common words as we did before, and \n", "\n", "- cut the reviews after only 20 words. \n", "\n", "Our network will simply \n", "\n", "- learn 8-dimensional embeddings for each of the 10,000 words\n", "\n", "- turn the input integer sequences (2D integer tensor with shape (25000, 20) or `(samples, sequence_length)`) into embedded sequences (3D float tensor with shape (25000, 20, 8) or `(samples, sequence_length, embedding_dimensionality)`), \n", "\n", "- flatten the tensor to 2D, and \n", "\n", "- train a single Dense layer on top for classification." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "embedding_1 (Embedding) (None, 20, 8) 80000 \n", "_________________________________________________________________\n", "flatten_1 (Flatten) (None, 160) 0 \n", "_________________________________________________________________\n", "dense_1 (Dense) (None, 1) 161 \n", "=================================================================\n", "Total params: 80,161\n", "Trainable params: 80,161\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "from keras.datasets import imdb\n", "\n", "max_features = 10000\n", "maxlen = 20\n", "np_load_old = np.load\n", "np.load = lambda *a,**k: np_load_old(*a, allow_pickle=True, **k)\n", "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=10000)\n", "np.load = np_load_old\n", "x_train = preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen)\n", "x_test = preprocessing.sequence.pad_sequences(x_test, maxlen=maxlen)\n", "model = models.Sequential()\n", "model.add(layers.Embedding(10000, 8, input_length=maxlen))\n", "model.add(layers.Flatten())\n", "model.add(layers.Dense(1, activation='sigmoid'))\n", "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n", "model.summary()\n", "#history = model.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.2, verbose=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We get to a validation accuracy of ~76%, which is pretty good considering that we only look at the first 20 words in every review. But note that merely flattening the embedded sequences and training a single Dense layer on top leads to a model that treats each word in the input sequence separately, without considering inter-word relationships and structure sentence.\n", "\n", "It would be much better to add recurrent layers or 1D convolutional layers (recognizing local patterns in a sentence or word sequence) on top of the embedded sequences to learn features that take into account each sequence as a whole." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pretrained Word Embeddings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Instead of using the pre-tokenized IMDB data packaged in Keras, we start from scratch by using the original text data." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "imdb_dir = base_dir+'\\\\aclImdb'\n", "train_dir = os.path.join(imdb_dir, 'train')\n", "\n", "labels = []\n", "texts = []\n", "\n", "for label_type in ['neg', 'pos']:\n", " dir_name = os.path.join(train_dir, label_type)\n", " for fname in os.listdir(dir_name):\n", " if fname[-4:] == '.txt':\n", " f = open(os.path.join(dir_name, fname), encoding=\"utf8\")\n", " texts.append(f.read())\n", " f.close()\n", " if label_type == 'neg':\n", " labels.append(0)\n", " else:\n", " labels.append(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are 25000 texts and labels.\n", "\n", "We restrict the training data to its first 200 samples. So we will be learning to classify movie reviews after looking at just 200 examples. The validation samples is 10,000.\n", "\n", "The texts are vectorized into sequences. The length of `sequences` is 25000, and `sequences[i]` is a list of integers corresponding to `texts[i]`.\n", "\n", "The `word_index` is a dictionary with words as keys and integer numbers as values, which is a mapping from words to integers." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found 88582 unique tokens.\n", "Shape of data tensor: (25000, 100)\n", "Shape of label tensor: (25000,)\n" ] } ], "source": [ "maxlen = 100 # We will cut reviews after 100 words\n", "training_samples = 200\n", "validation_samples = 10000\n", "max_words = 10000 # We will only consider the top 10,000 words in the dataset\n", "\n", "tokenizer = preprocessing.text.Tokenizer(num_words=max_words)\n", "tokenizer.fit_on_texts(texts)\n", "sequences = tokenizer.texts_to_sequences(texts)\n", "\n", "word_index = tokenizer.word_index\n", "print('Found %s unique tokens.' % len(word_index))\n", "\n", "data = preprocessing.sequence.pad_sequences(sequences, maxlen=maxlen)\n", "\n", "labels = np.asarray(labels)\n", "print('Shape of data tensor:', data.shape)\n", "print('Shape of label tensor:', labels.shape)\n", "\n", "# Split the data into a training set and a validation set\n", "# But first, shuffle the data, since we started from data\n", "# where sample are ordered (all negative first, then all positive).\n", "indices = np.arange(data.shape[0])\n", "np.random.shuffle(indices)\n", "data = data[indices]\n", "labels = labels[indices]\n", "\n", "x_train = data[:training_samples]\n", "y_train = labels[:training_samples]\n", "x_val = data[training_samples: training_samples + validation_samples]\n", "y_val = labels[training_samples: training_samples + validation_samples]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We use `glove.6B` the pre-computed embeddings from 2014 English Wikipedia." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found 400000 word vectors.\n" ] } ], "source": [ "glove_dir = base_dir+'\\\\glove.6B'\n", "embeddings_index = {}\n", "f = open(os.path.join(glove_dir, 'glove.6B.100d.txt'), encoding=\"utf8\")\n", "for line in f:\n", " values = line.split()\n", " word = values[0]\n", " coefs = np.asarray(values[1:], dtype='float32')\n", " embeddings_index[word] = coefs\n", "f.close()\n", "\n", "print('Found %s word vectors.' % len(embeddings_index))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`glove.6B.100d.txt` has 400,000 rows where each row includes a word and a float array. For example, the first row likes\n", "
\n", "\"the -0.038194 -0.24487 0.72812 -0.39961 0.083172 0.043953 -0.39141 0.3344 ...\"\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We need an embedding matrix to set `Embedding` layer's weight as the pretrained word embeddings. The embedding matrix must have shape `(max_words, embedding_dim)` ((10000, 100)), where each entry $i$ contains the embedding_dim, a dimensional vector for the word of index $i$. Note that `embedding_matrix[0]` needs to be a 0 array as a placeholder.\n", "\n", "We load the GloVe matrix (`embedding_matrix`) into Embedding layer and freeze the embedding layer. We can also try to train the same model without loading the pre-trained word embeddings and without freezing the embedding layer. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "embedding_2 (Embedding) (None, 100, 100) 1000000 \n", "_________________________________________________________________\n", "flatten_2 (Flatten) (None, 10000) 0 \n", "_________________________________________________________________\n", "dense_2 (Dense) (None, 32) 320032 \n", "_________________________________________________________________\n", "dense_3 (Dense) (None, 1) 33 \n", "=================================================================\n", "Total params: 1,320,065\n", "Trainable params: 1,320,065\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyQAAAE/CAYAAAC3hAd1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd5wU9f3H8deHo54ggjSl3KH0IiScomIEO9bDCAoc1iASS9TEXzRiC0piojHGCmeJCieCYo3YFVGxgVKlqAiIoDSpRz2+vz++e7IcV/Zgd2fL+/l47GOZ78zNfHbumJ3PfJs55xAREREREQlClaADEBERERGR9KWEREREREREAqOEREREREREAqOEREREREREAqOEREREREREAqOEREREREREAqOEJMGZWYaZbTSzFtHcNkhm1srMoj7etJmdaGaLwpbnm9lvItl2L471qJnduLc/LyJSkq73ldpv0l/vzewOM3si2vsVSUZVgw4g1ZjZxrDFTGArUBRavsw5V1CZ/TnnioDa0d42HTjn2kZjP2Y2GBjknOsVtu/B0di3iCQvXe8Th673IslNCUmUOed++YIIPZEZ7Jx7u6ztzayqc25HPGITqYj+HkUip+u9iEh0qMlWnIWqaMeZ2Vgz2wAMMrOjzOwTM1trZsvN7D4zqxbavqqZOTPLDi2PCa1/zcw2mNnHZtaystuG1p9qZgvMbJ2Z3W9mH5nZRWXEHUmMl5nZN2b2s5ndF/azGWb2bzNbbWbfAr3LOT83mdkzJcoeNLN7Qv8ebGZzQ5/n29DTrLL2tdTMeoX+nWlmo0OxzQG6lXLchaH9zjGzs0LlnYEHgN+EmkesCju3t4X9/NDQZ19tZi+a2UGRnJvKnOfieMzsbTNbY2Y/mtmfw45zc+icrDezqWZ2cGnNJczsw+Lfc+h8Tg4dZw1wk5m1NrP3Qp9lVei81Q37+azQZ1wZWv8fM6sZirl92HYHmVmhmR1Y1ucVSWW63ut6X971vpTP0CcUz1oze9fM2oatu9HMloWu7/PCPuuRZvZFqPwnM7sr0uOJJBTnnF4xegGLgBNLlN0BbAPOxCeEtYDDge74GqtDgAXAlaHtqwIOyA4tjwFWATlANWAcMGYvtm0EbAByQ+v+CGwHLirjs0QS40tAXSAbWFP82YErgTlAM+BAYLL/0yv1OIcAG4H9wva9AsgJLZ8Z2saA44HNwGGhdScCi8L2tRToFfr33cAkoB6QBXxVYttzgYNCv5OBoRgah9YNBiaViHMMcFvo3yeHYuwK1AQeAt6N5NxU8jzXBX4CrgZqAPsDR4TW/QWYAbQOfYauQH2gVclzDXxY/HsOfbYdwO+BDPzfYxvgBKB66O/kI+DusM8zO3Q+9wtt3yO0Lh8YEXacPwEvBP3/UC+94vFC13td7yt/vb8DeCL07/ahOI4P/Y5uDJ33akBHYDHQJLRtS+CQ0L8/BwaE/l0H6B70/wW99Nqbl2pIgvGhc+4V59xO59xm59znzrlPnXM7nHML8Td2Pcv5+eecc1Odc9uBAvyFsbLbngFMd869FFr3b/yXWakijPHvzrl1zrlF+C+D4mOdC/zbObfUObcauLOc4yzE3/DmhopOAtY656aG1r/inFvovHeBd4BSOzKWcC5wh3PuZ+fcYvxTsPDjjnfOLQ/9Tp7G31zkRLBfgDzgUefcdOfcFuAGoKeZNQvbpqxzs5sKzvNZwPfOuf8457Y659Y75z4LrRsM3Oic+zr0GaY759ZEGP8S59zDzrmi0N/jAufcO865bc65Ffi/jeIYjgIaANc75zaFtv8otO5JYKCZWWj5fGB0hDGIpCpd78s+Tlpf70voD7zsnHs39Du6E//QqTv+oVFNoKP5Zn/fhc4d+MSytZkd6Jzb4Jz7NMLPIZJQlJAE4/vwBTNrZ2avmm+Csx4Yjr/pK8uPYf8upPyOjWVte3B4HM45h3/CVKoIY4zoWPgnPeV5GhgQ+vdA/BdrcRxnmNmn5pssrcU/rSrvXBU7qLwYzOwiM5sRqipfC7SLcL/gP98v+3POrQd+BpqGbRPR76yC89wc+KaMGJoD30YYb0kl/x6bmNl4M/shFMMTJWJY5HyH2t2EEpMdwDFm1gloAby6lzGJpApd78uXttf7Cva7E/87auqcm4+vcR4OrDDfBLBJaNOLgQ7AfDP7zMxOi/BziCQUJSTBKDkE4ij8U6JWzrn9gVvwVdSxtBxfpQ5A6Kl207I336cYl+NvZItVNEzlOODE0BOnXPwXFmZWC3gO+Du+ev0A4M0I4/ixrBjM7BDgYXyzpQND+50Xtt+Khqxchm8WULy/OvimAj9EEFdJ5Z3n74FDy/i5stZtCsWUGVbWpMQ2JT/fP/CjBXUOxXBRiRiyzCyjjDieAgbha0fGO+e2lrGdSLrQ9b586Xy9L2+/VfC/sx8AnHNjnHM98M21MvDnBefcfOdcf3yzvH8BE8ys5j7GIhJ3SkgSQx1gHbDJfKfgy+JwzP8BvzazM82sKr5fQsMYxTgeuMbMmprv4Hx9eRs7537C93P4LzDfOfd1aFUNfL+GlUCRmZ2B7+sQaQw3mtkB5sftvzJsXW38l9BK/Hf1YPwTs2I/Ac0srHN5CWOB35nZYWZWA/9F8YFzrswnkOUo7zy/DLQwsyvNrLqZ7W9mR4TWPQrcYWaHmtfVzOrjv5h/xHemzTCzIYR96ZUTwyZgnZk1B64LW/cxsBr4m/mOo7XMrEfY+tFAX/yTzqf24vOLpDpd78Ok+fW+ZMxnmVmv0LH/D9/v51Mza29mx4WOtzn0KsJ/gPPNrEGoRmVd6LPt3MdYROJOCUli+BNwIf7iMwr/xCimQl8C5wH34G8wDwW+xD8Zj3aMD+Pb/s7Cd8B7LoKfeRrfafHpsJjXAtcCL+A7CvbFf9FG4lb8k7tFwGuE3Sw752YC9wGfhbZpB4S3w30L+Br4yczCq+KLf/51fFX6C6Gfb4FvZ7w3yjzPzrl1+DbW5+A7VS5gV7vuu4AX8ed5Pb7Nd81Q04xL8R0kV+E7uVfUxvhW4Aj8l9vLwISwGHbg26O3x9eWLMH/HorXL8L/nrc556ZU8rOLpANd7/eUrtf78P3OwZ/zh/HJUm/grFB/khrAP/HX8B/xNTI3hX70NGCu+VHc7gbOc85t29d4ROLN/P2KpLtQE5xlQF/n3AdBxyPJy8yeAhY6524LOhYR2ZOu9yKSaFRDksbMrLeZ1Q1VA9+M75D8WQU/JlKmUPvsXODxoGMRkV10vReRRKaEJL0dAyzEVwP3BvqoE7LsLTP7O34ulL8555YEHY+I7EbXexFJWGqyJSIiIiIigVENiYiIiIiIBEYJiYiIiIiIBKZqUAdu0KCBy87ODurwIiL7ZNq0aaucc+XN5RAYM3scPzzzCudcpzK26QXcC1QDVjnnepa2XTFds0UkmSXyNVsCTEiys7OZOnVqUIcXEdknZrY46BjK8QTwAGVMTmlmBwAPAb2dc0vMrFFFO9Q1W0SSWYJfs9OemmyJiKQY59xk/GRyZRkIPF88GppzbkVcAhMRESmFEhIRkfTTBqhnZpPMbJqZXRB0QCIikr4Ca7IlIiKBqQp0A04AagEfm9knzrkF4RuZ2RBgCECLFi3iHqSIiKQH1ZCIiKSfpcDrzrlNzrlVwGSgS8mNnHP5zrkc51xOw4bqCyoiIrGhhEREJP28BPzGzKqaWSbQHZgbcEwiIpKm1GRLRCTFmNlYoBfQwMyWArfih/fFOTfSOTfXzF4HZgI7gUedc7ODildERNKbEhIRkRTjnBsQwTZ3AXfFIRwREZFyVdhky8weN7MVZlbq0zPz7jOzb8xsppn9OvphiqS2ggLIzoYqVfx7QUHQEcVHkJ87Xc95vOj8iohIpCKpIXmCcibYAk4FWode3YGHQ+8iEoGCAhgyBAoL/fLixX4ZIC8vuLhiLcjPna7nPF50fkVEpDLMOVfxRmbZwP+cc51KWTcKmOScGxtang/0cs4tL2+fOTk5TrP+ivinx4tLmT82KwsWLYp3NPET5OeOxrHNbJpzLieacSWyylyz0/VvWkQSV7pds5NNNEbZagp8H7a8NFS2BzMbYmZTzWzqypUro3BokeS3ZEnlylNFkJ87Xc95vOj8iohIZUQjIbFSykqtdtGY9iJ7Kmu+uVSfhy7Iz52u5zxedH5FRKQyopGQLAWahy03A5ZFYb8iaWHECMjM3L0sM9OXp7IgP3e6nvN40fkVEZHKiEZC8jJwQWi0rSOBdRX1HxGRXfLyID/ft6838+/5+anf+TfIz52u5zxedH5FRKQyKuzUHj7BFvATJSbYMjPDj8LVGygELnbOVdjzUZ3aRSSZpVsHSV2zRSSZpds1O9lUOOxvRRNsOZ/RXBG1iCTtFRTAsGG+A2yLFr6Zh56sioiIiKQmzdQuCUXzF4iIiIikl2j0IRGJmmHDdiUjxQoLfbmIiIiIpB4lJJJQNH+BiIiISHpRQiIJRfMXiIiIiKQXJSSSUDR/gYiIiEh6UUIiCUXzF4iIiIikF42yJQknL08JiIiIiEi6UA2JiIiIiIgERgmJiIiIiIgERgmJiIiIiIgERgmJiIiIiIgERgmJiIiIiIgERgmJiIiIiIgERgmJiIiIiIgERgmJiEiKMbPHzWyFmc2uYLvDzazIzPrGKzYREZGSlJCIiKSeJ4De5W1gZhnAP4A34hGQiIhIWZSQiIikGOfcZGBNBZtdBUwAVsQ+IhERkbIpIRERSTNm1hQ4GxgZdCwiIiJKSERE0s+9wPXOuaLyNjKzIWY21cymrly5Mk6hiYhIuqkadAAiIhJ3OcAzZgbQADjNzHY4514M38g5lw/kA+Tk5Li4RykiImlBCYmISJpxzrUs/reZPQH8r2QyIiIiEi9KSEREUoyZjQV6AQ3MbClwK1ANwDmnfiMiIpJQlJCIiKQY59yASmx7UQxDERERqZA6tYuIiIiISGCUkIiIiMi++fFHeP31oKMQkSSlhERERET2zXXXwWmnwbJlQUciIklICYmIiIjsvdWr4bnnwDl45ZWgoxGRJKSERERERPbe6NGwdSvUqwcvvRR0NCKShJSQiIiIyN5xDvLz4cgj4eKL4Z13YMOGoKMSkSSjhERERET2zkcfwdy5MGQI5ObCtm3q3C4ilaaERCRMQQFkZ0OVKv69oCDoiEREElh+Puy/P5x7Lhx9NBx4oJptiUilaWJEkZCCAv+Qr7DQLy9e7JcB8vKCi0tEJCGtWQPjx8Pvfgf77efLzjwTXnwRtm+HatWCjU9EkoZqSERChg3blYwUKyz05SIiUsKYMb4ze/GTG/DNttauhcmTg4tLRJKOEhKRkCVLKlcuIpK2ijuzH3EEdOmyq/ykk6BmTTXbEpFKUUIiEtKiReXKRUTS1scfw5w5u9eOgG+6dfLJvtmWc8HEJiJJRwmJSMiIEZCZuXtZZqYvFxGRMPn5UKcOnHfenutyc+H772H69PjHJSJJSQmJSEhenv+OzcoCM/+en68O7SIiu/n5Zxg3zl8ca9fec/0ZZ/iLqJptiUiElJCIhMnLg0WLYOdO/65kRESkhIIC2LJlz+ZaxRo1gh49fLMtEZEIKCERERGRyBR3Zs/JgV/9quztcnNhxgz/ZEdEpAIRJSRm1tvM5pvZN2Z2Qynr65nZC2Y208w+M7NO0Q9VREREAvXppzBrVtm1I8Vyc/37yy/HPiYRSXoVJiRmlgE8CJwKdAAGmFmHEpvdCEx3zh0GXAD8J9qBioiISMDy832/kf79y9+udWvo0EHNtkQkIpHUkBwBfOOcW+ic2wY8A+SW2KYD8A6Ac24ekG1mjaMaqYiIiARn3Tp45hkYONCPsFWR3Fw/QeKaNbGPTUSSWiQJSVPg+7DlpaGycDOA3wKY2RFAFtAsGgGKiIhIAigogM2bK26uVaxPHygqgokTYxuXiCS9SBISK6Ws5GxHdwL1zGw6cBXwJbBjjx2ZDTGzqWY2deXKlZUOVkRERALgHIwaBb/+NXTrFtnP5OTAQQep2ZaIVCiShGQp0DxsuRmwLHwD59x659zFzrmu+D4kDYHvSu7IOZfvnMtxzuU0bNhwH8IWERGRuPn8c5g5M/LaEYAqVeCss+D11/0wwSIiZYgkIfkcaG1mLc2sOtAf2G3YDDM7ILQOYDAw2Tm3PrqhioiISCDy82G//WDAgMr9XJ8+sGkTvPNObOISkZRQYULinNsBXAm8AcwFxjvn5pjZUDMbGtqsPTDHzObhR+O6OlYBi4iISBytXw9jx/pkZP/9K/ezxx3nO8Br1nYRKUfVSDZyzk0EJpYoGxn274+B1tENTURE9oaZPQ6cAaxwzu0xL5SZ5QHXhxY3Ar93zs2IY4iSTJ5+GgoLK9dcq1iNGtC7t5+PZORI34xLRKQEXRlERFLPE0DvctZ/B/QMzR11O5Afj6AkCRV3Zu/a1XdS3xt9+sBPP/lJFUVESqGEREQkxTjnJgNlTv7gnJvinPs5tPgJGqZdyjJtGkyf7mtHrLRBNyNw2mlQtaqabYlImZSQiIikt98BrwUdhCSo/HzIzPSTIe6tAw6Anj2VkIhImZSQiIikKTM7Dp+QXF/Ges0dlc42bPD9R/r3h7p1921fffrAvHkwf350YhORlKKEREQkDZnZYcCjQK5zbnVp22juqDT3zDN+yN5LL933fZ11ln9XLYmIlEIJiYhImjGzFsDzwPnOuQVBxyMJKj8fOneG7t33fV8tWvhZ3pWQiEgplJCIiKQYMxsLfAy0NbOlZva7EnNH3QIcCDxkZtPNbGpgwUpi+uILmDp13zqzl5SbCx9/7EfcEhEJE9E8JCIikjycc+VOp+2cGwwMjlM4koweeQRq1oRBg6K3z9xcuPVWeOUVGKw/PxHZRTUkIiIissvGjVBQAOed50fIipbDDoPsbHjxxejtU0RSghISERER2WXcOD/C1t7MzF4eM19L8vbbPukREQlRQiIiIiK75OdDx45w1FHR33duLmzdCm++Gf19i0jSUkIiIiIi3vTp8Nln0e3MHu43v4F69dRsS0R2o4REREREvFh0Zg9XtSqccQa8+irs2BGbY4hI0lFCIiIiIn4SxDFjoF8/qF8/dsfJzYU1a+DDD2N3DBFJKkpIpFQFBX4wlCpV/HtBQdARiYhITI0fD+vXR78ze0mnnAI1aqjZloj8QgmJ7KGgwH8fLV4Mzvn3IUOUlIiIpLT8fGjfHnr0iO1xateGE0/0s7Y7F9tjiUhSUEIiexg2DAoLdy8rLPTlIiKSgmbOhE8+iV1n9pJyc2HRIpg1K/bHEpGEp4RE9rBkSeXKRUQkyT3yiG9Gdf758TnemWf6xEfNtkQEJSRSihYtKlcuIiJJrLAQRo+Gvn3hwAPjc8wmTeDII32zLRFJe0pIZA8jRkBm5u5lmZm+XEREUsyzz8K6dbHvzF5Snz7wxRfw/ffxPa6IJBwlJLKHvDzftzEry9eoZ2X55by8oCMTEZGoy8+Htm39pIXxlJvr31VLIpL2lJBIqfLyfH/DnTv9u5IREZEUNHs2TJkSv87s4dq29S8lJCJpTwmJiIhIunrkEaheHS64IJjj9+kDkybB2rXBHF9EEoISEhERkXS0eTM89RSccw40aBBMDLm5sGMHTJwYzPFFJCEoIRFJcwUFkJ0NVar4d02AKZImnnvO10zEuzN7uO7doXFjNdsSSXNKSETSWEGBvxdZvNhPmLx4sV9WUiKSBvLzoXVr6NkzuBiqVIGzzvI1JFu3BheHiARKCYlIGhs2zE9BEK6w0JeLSAr76iv48MNgOrOXlJsLGzfCe+8FG4eIBEYJiUgaW7KkcuUikiIeeQSqVYMLLww6EjjhBNhvPzXbEkljSkhE0liLFpUrF5EUsGULPPkk/Pa30LBh0NFAzZrQu7dPSHbuDDoaEQmAEhKRNDZiBGRm7l6WmenLRSRFTZgAP/8cbGf2knJzYflymDo16EgqZ8UKJVEiUaCERCSN5eX5fq1ZWb4ZeVaWX9ZEmCIpLD8fWrWCXr2CjmSX00+HjIzkarY1a5avTh40yI8KIiJ7TQmJSJrLy4NFi/xDvkWLlIykAjN73MxWmNnsMtabmd1nZt+Y2Uwz+3W8Y5SAzJsHkyfDpZf6Ea4SRf36cOyx8OKLQUcSme3bff+boiIYOxZGjQo6IpGklkBXIxERiZIngN7lrD8VaB16DQEejkNMkgiKO7NfdFHQkewpN9eP/vX110FHUrE774Qvv/TJSO/ecPXV8MUXQUclkrSUkIiIpBjn3GRgTTmb5AJPOe8T4AAzOyg+0Ulgijuz5+ZCo0ZBR7On3Fz/nujNtmbMgNtvh/79oW9fGD3an89+/fxEkyJSaUpIRETST1Pg+7DlpaEySWUvvACrVydWZ/Zw2dnQpUtiJyTbt/vapXr14P77fVmDBjBunB8v/ZJL1J9EZC8oIRERST+lzYS3x12UmQ0xs6lmNnXlypVxCEtiKj8fWrb0834kqtxcmDLFj16ViP72N5g+HUaO9IlIsaOPhn/8wyd9994bXHwiSUoJiYhI+lkKNA9bbgYsK7mRcy7fOZfjnMtpmAjzVcjeW7AAJk1KvM7sJfXp40fY+N//go5kT9Onwx13wIABcPbZe66/9lof/5//DB9/HP/4RJJYAl+VREQkRl4GLgiNtnUksM45tzzooCSGHnkEqlaFiy8OOpLyde3qh9JNtGZb27b5ploHHrirqVZJZvDf/0Lz5nDuubBqVVxDFElmSkhERFKMmY0FPgbamtlSM/udmQ01s6GhTSYCC4FvgEeAywMKVeJh61Z44gk46yxo0iToaMpn5uN86y0oLAw6ml1GjPCd2UeN8klJWQ44AJ591jc5u+ACTZooEiElJCIiKcY5N8A5d5Bzrppzrplz7jHn3Ejn3MjQeuecu8I5d6hzrrNzLsmmx5ZKefFF/7Q+UTuzl9SnD2zeDG++GXQk3pdf+r4jeXm7RgIrT7duvh/Ja6/5fiUiUqGIEhIz621m80OTaN1Qyvq6ZvaKmc0wszlmluB1wiIiImkiPx+ysuCkk4KOJDLHHutrGhKh2VZxU60GDeC++yL/uaFD/bDAN90E778fs/BEUkWFCYmZZQAP4ifS6gAMMLMOJTa7AvjKOdcF6AX8y8yqRzlWERERqYyvv4Z33038zuzhqlWD006DV16BHTuCjeWOO2DmTJ/U1a8f+c+Z+Z9p1conJj/9FLsYRVJAJFenI4BvnHMLnXPbgGfwk2qFc0AdMzOgNn5CroCvIiIiImnu0UchIyPxO7OX1KePnzNlypTgYpg2zTfVOv98OPPMyv98nTrw3HOwbh0MHAhFRdGPUSRFRJKQRDKB1gNAe/ywkbOAq51z6sklIiISlG3b/KhPZ54JBx8cdDSV07s3VK8eXLOtrVt9U61GjeA//9n7/XTuDA895Gup/vrXqIUnkmoiSUgimUDrFGA6cDDQFXjAzPbfY0eaZEtERCQ+XnoJVq5Mns7s4erU8RM4vvhiMDOf3347zJ7th0uuV2/f9nXRRb6G6o474I03ohKeSKqJJCGJZAKti4HnQyO3fAN8B7QruSNNsiUiIhIn+fl+To+TTw46kr2TmwsLF8KcOfE97tSpcOedcOGFcPrp0dnnAw9Ax44waBAsXRqdfYqkkEgSks+B1mbWMtRRvT9+Uq1wS4ATAMysMdAWP8a9iIiIxNu338Lbb8Pgwb4PSTIq7rcRz2ZbxU21Gjf2Q/dGS2am70+yZYvv5L59e/T2LZICKkxInHM7gCuBN4C5wHjn3JwSk2zdDhxtZrOAd4DrnXOaolRERCQIjz3mR9W65JKgI9l7Bx8M3bv7Zlvx8te/+hqZRx/1Qw9HU9u2vtbqo49g2LDo7lskyVWNZCPn3ET8zL7hZSPD/r0MSNI6YRERkRSyfTs8/jiccQY0LTkGTZLJzYUbb4Qffoj9Z/n8cz+R4cUXw6mnxuYYAwbABx/AXXfBMcf4WelFRDO1J7qCAsjO9g+6srP9soiISJleecXPe5GMndlLKp4Z/eWSLcWjbMsW32fk4IPhnntie6x77oFf/9of77vvYnsskSShhCSBFRT475PFi/0gI4sX+2UlJSIiUqb8fGjWzA+dm+zat4fWrWPfbOu222Du3Ng01SqpZk149ln/xX7uub7fikiaU0KSwIYNg8LC3csKC9X0VEREyvDdd/Dmm8ndmT2cma8lee89P8FgLHz6qW9C9bvfwSmnxOYYJR1yCDzxhB/R67rr4nNMkQSmhCSBLVlSuXIREUlzjz3mb+KTuTN7Sbm5vl/M669Hf99btvhRtQ4+GP71r+jvvzx9+sAf/+iHBB4/Pr7HFkkwSkgSWIsWlSsXEZE0VtyZ/bTToHnzirdPFkcdBQ0bxqbZ1i23wLx5PpGrWzf6+6/InXf6zzd4MCxYEP/jiyQIJSQJbMQIP3R5uMxMXy4iIvKLoiL4/e9h+XK47LKgo4mujAw/J8nEibBtW/T2+8knvlbk0kuDmzyyWjUYNw6qV4d+/WDz5mDiEAmYEpIElpfn+yZmZfka+Kwsv5yXF3RkIiKSMLZuhfPO80/5b745erOLJ5I+fWD9enj//ejsb/Nm31SrWTO4++7o7HNvNW8Oo0fDzJlw1VXBxiISECUkCS4vDxYtgp07/buSERER+cXGjX6+kQkT4N//huHD/ROsVHPiib6JQLSabd18M8yf75O4/fePzj73xamn+hFrHnsMnnwy6GhE4k4JiYiISDJas8bfqL/3nh+x6Zprgo4odmrV8s2qXnrJD5e7L6ZM8XOBXHaZP3+J4rbboFcv3/Ru9uygoxGJKyUkIiIiyWbZMjj2WJg+3deOXHhh0BHFXp8+fsb2adP2fh+Fhb6pVosWfqjfRFK1Kjz9tK+x6dfP136JpAklJCIiIsnkm2+gRw8/W+5rr+2azTzVnX46VKnia0n21k03wddf+6ZRdepEL7ZoOeggGDvWj7h12WX7XhskkiSUkIiIiCSLmTPhmGNgwwZ491047rigI4qfBg38Z9/bhOTDD+Hee32TqBNOiG5s0XTccfDXv/rakkceCToakbhQQiIiIpIMpkyBnj19054PPoDDDznC0bQAACAASURBVA86ovjr0wdmzYKFCyv3c4WFcPHFfrjKf/4zNrFF0403+lnj//AH+OKLoKMRiTklJCIiIonu9dd9B+yGDeGjj6B9+6AjCkZx87TK1pIMG+abuj3+ONSuHf24oq1KFT8UcIMGvj/JunVBRyQSU0pIRERSjJn1NrP5ZvaNmd1Qyvq6ZvaKmc0wszlmdnEQcUqExo2Ds86Ctm19zUhWVtARBeeQQ6BTp8olJB98AP/5D1xxRXI1cWvYEMaPhyVL4JJL1J9EUpoSEhGRFGJmGcCDwKlAB2CAmXUosdkVwFfOuS5AL+BfZlY9roFKZPLzYcAAOPJImDQJGjcOOqLg9enjk4xVqyredtMm31QrOxvuvDPmoUXd0Uf7uJ9/Hu67L+hoRGJGCYmISGo5AvjGObfQObcNeAYoOQyTA+qYmQG1gTXAjviGKeVyzt+IXnaZnzTv9dehbt2go0oMubl+tuBXX6142xtvhG+/TZ6mWqX54x/9Z77uOvjkk6CjEYkJJSQiIqmlKfB92PLSUFm4B4D2wDJgFnC1c25nfMKTCjkH118Pf/kLDBzoZyfPzAw6qsTRrRs0bVpxs6333/e1Cldd5SccTFZm8N//QrNmcN55sHp10BGJRJ0SEhGR1GKllJVsfH4KMB04GOgKPGBm+++xI7MhZjbVzKauXLky+pHKnoqK4NJL/aR9V1zhOzZXqxZ0VInFzNcYvPEGbN5c+jabNvl+F4ceCn//e3zji4V69eDZZ+HHH+GCC3wNkUgKUUIiIpJalgLNw5ab4WtCwl0MPO+8b4DvgHYld+Scy3fO5Tjncho2bBizgCVk61b/BPyxx+Dmm+H++/1oS7Kn3Fw/lO/bb5e+/oYb/NDAjz8O++0X39hiJScH7rkHJk5MjqGLo2nzZj+h5Tvv+PlkJOVUDToAERGJqs+B1mbWEvgB6A8MLLHNEuAE4AMzawy0BSo5sUOC2rTJ37BNmOBrFoYO9R2DrbSKowSycSOcfba/wf73v+Gaa4KOKLH16gX77++bs5155u7rJk2CBx6Aq6+GY48NIrrYufxy36F/2DA46ig/L02y27nT1/x8/70fUaz4Fb4cXkN70knw5pvBxSsxoYRERCSFOOd2mNmVwBtABvC4c26OmQ0NrR8J3A48YWaz8E28rnfORTBkUYIqTkLGj/fvhYXQqJGvcRgzBn71K9+PoH9/qFUr6Gj3tGYNnHYaTJ0KTzwBF14YdESJr3p1f85eecU3c8vI8OUbN/pRtVq1gr/9LdgYY8HMz97+5Zdw7rlw+ulwwAG7v+rV27Osdu3gkvL16/dMMMKXly6F7dt3/5nataFFC//q1g2aN9+1fMghwXwOiSklJCIiKcY5NxGYWKJsZNi/lwEnxzuuqApPQl591TfpaNTI38yfey785jewZYtPSO6/3/cn+L//8/0zfv97f2OTCJYtg5NP9s1RnnvOD2krkcnNhWee8SNP9ejhy66/HhYvhsmTU3cggDp1fA3g4MHw1luwdq1PxMqTkbFnklJW8lJaWa1apSc027fDDz+Un3CUnNQxI8N30G/e3NfyFCca4UlH3bqJX6spUWUuoIl2cnJy3NSpUwM5tojIvjKzac65nKDjiJeEuGZv2uSTj2ef3T0JOeecXUlI8ZPycM75Zjz3379rZKbc3F2jLwV14/Ptt3729VWrfFzHHx9MHMlq3To/eeDVV/tBAN59F044Aa691ve1SCc7dvjz8fPPPkEp+aqovKzBAYpVr757glKlik84li3bc8LGAw8sPckoXj7ooNL/n8ZYul2zk40SkggUFPjmmkuW+P9PI0ZAXl7QUYlIkNLtyy2wa3ZpSUjjxj4J6dev7CSkLIsXw8MP+2Yva9b4Wb+vvBIGDYpv5+eZM+GUU/wT5tdeg8MPj9+xU8kpp8B338G0aXDYYb7f0PTpqVs7Eitbt/qEpqIEpnhdUdGuZCM86WjePGHPfbpds5ONmmxVoKAAhgzxTZLBf5cNGeL/raRERCQGykpCLr5475KQcFlZfsLBW2/1zX3uv993fL/hBt+s64orYt9GfcoU3/Z/v/38U/327WN7vFSWm+t/Z337+i/oDz5I2BvihFajhq9tbNQo6EgkTamGpALZ2f4aV1JWFixaFO9oRCRRpNvTtphfs6NdExIp53yCcP/9vl1+UZHvLH3VVX40n2gPu/vGG340rWbNfPv/rKzo7j/dLF3qn8oD/OlPcPfdwcYjCSvdrtnJRjUkFViypHLlIiISoeIkpHh0rGjWhETKzHeI7tHDt4cfORJGjYLevaFNG9+c68IL/RCz+2r8eN80rGNHeP11/1ll3zRr5od1Xr0abr896GhEZC9pxqUKlDUQS6IM0CIiklQ2boRx43wTm4YN/USAH37ok5D33vMj9jz4oO9sHu+OrwcfDMOH+ydOo0f7zrt/+AM0beprTObP3/t95+f7YYePPNJ3sFcyEj2vvAIff5yYQzqLSESUkFRgxIg9m6NmZvpyERGJQHgS0qiRvzEvTkImTQo2CSlNjRq+JuPTT/3r7LN9QtGune9E/b//+aZdkbrzTrjsMjj1VF8zUrdu7GJPR/Xr+6FqRSRpKSGpQF6e/x7KyvI1+1lZflkd2kVEKvDeexUnIT17JkYSUpYjjoCnnvK1JrffDrNn+5nB27TxQ8v+/HPZP+sc/PnP8Je/wMCBflZxdbgWEdmDEpII5OX5Duw7d/p3JSMiIhGYMyc5k5DSNG4MN93kvwTGjfPNu/70J9+H4bLLfKISrqjIT8J4111+FKjRo/2QtCIisgclJCIiEhuXXprcSUhpqlXzkzB+8AF8+SUMGOBrUDp3huOOg+ef9+PEn3cePPYY3HyzH8Er2qN1iYikEF0hRUQkNmrUSI0kpCxdu8Kjj/qhZ//xDz9B3znnQIMGfgjhf//bd5IPaiZ4EZEkoYRERERkXxx4oO8r8u238MILfv6S0aPhmmuCjkxEJCloHhIREZFoyMiAPn38S0REIqYaEhERERERCYwSEhERERERCYwSEhERERERCYwSEhERERERCUxECYmZ9Taz+Wb2jZndUMr6/zOz6aHXbDMrMrP60Q9XRERERERSSYUJiZllAA8CpwIdgAFm1iF8G+fcXc65rs65rsBfgPedc2tiEbCIiIiIiKSOSGpIjgC+cc4tdM5tA54BcsvZfgAwNhrBiYiIJJOCAsjO9hOzZ2f7ZRERKV8kCUlT4Puw5aWhsj2YWSbQG5iw76GJiIgkj4ICGDIEFi8G5/z7kCFKSkREKhJJQmKllLkytj0T+Kis5lpmNsTMpprZ1JUrV0Yao4iISMIbNgwKC3cvKyz05SIiUrZIEpKlQPOw5WbAsjK27U85zbWcc/nOuRznXE7Dhg0jj1JERCTBLVlSuXIREfEiSUg+B1qbWUszq45POl4uuZGZ1QV6Ai9FN0QREZHE16JF5cpFRMSrMCFxzu0ArgTeAOYC451zc8xsqJkNDdv0bOBN59ym2IQqIiKRqGio9tA2vUJDtc8xs/fjHWMqGjECMjN3L8vM9OUiIlK2qpFs5JybCEwsUTayxPITwBPRCkxERCovbKj2k/BNbj83s5edc1+FbXMA8BDQ2zm3xMwaBRNtasnL8+/DhvlmWi1a+GSkuFxEREoXUUIiIiJJ45eh2gHMrHio9q/CthkIPO+cWwLgnFsR9yhTVF6eEhARkcqKaKZ2ERFJGpEM1d4GqGdmk8xsmpldUNqONDKiiIjEgxISEZHUEslQ7VWBbsDpwCnAzWbWZo8f0siIIiISB2qyJSKSWiIZqn0psCo0CMkmM5sMdAEWxCdEERGRXVRDIiKSWiIZqv0l4DdmVtXMMoHu+FEURURE4k41JCIiKcQ5t8PMiodqzwAeLx6qPbR+pHNurpm9DswEdgKPOudmBxe1iIikMyUkIiIpJsKh2u8C7opnXCIiIqVRky0REREREQmMEhIREREREQmMEhIREREREQmMEhIREUkpBQWQnQ1Vqvj3goKgIxIRkfKoU7uIiKSMggIYMgQKC/3y4sV+GSAvL7i4RESkbKohERGRlDFs2K5kpFhhoS8XEZHEpIRERERSxpIllSsXEZHgKSEREZGU0aJF5cpFRCR4SkhERCRljBgBmZm7l2Vm+nIREUlMSkhERCRl5OVBfj5kZYGZf8/PV4d2EZFEplG2REQkpeTlKQEREUkmqiEREREREZHAKCEREREREZHAKCEREREREZHAKCEREREREZHAKCEREREREZHAKCEREREREZHAKCEREREREZHAKCEREREREZHAKCEREREREZHAKCEREREREZHAKCEREREREZHAKCEREREREZHAKCEREUkxZtbbzOab2TdmdkM52x1uZkVm1jee8YmIiIRTQiIikkLMLAN4EDgV6AAMMLMOZWz3D+CN+EYosVJQANnZUKWKfy8oCDoiEZHIKCEREUktRwDfOOcWOue2Ac8AuaVsdxUwAVgRz+AkNgoKYMgQWLwYnPPvQ4YoKRGR5KCEREQktTQFvg9bXhoq+4WZNQXOBkbGMS6JoWHDoLBw97LCQl8uIpLolJCIiKQWK6XMlVi+F7jeOVdU7o7MhpjZVDObunLlyqgFKNG3ZEnlykVEEokSEhGR1LIUaB623AxYVmKbHOAZM1sE9AUeMrM+JXfknMt3zuU453IaNmwYq3glClq0qFy5iEgiUUIiIpJaPgdam1lLM6sO9AdeDt/AOdfSOZftnMsGngMud869GP9QJVpGjIDMzN3LMjN9uYhIolNCIiKSQpxzO4Ar8aNnzQXGO+fmmNlQMxsabHQSK3l5kJ8PWVlg5t/z8325iEiiqxp0ACIiEl3OuYnAxBJlpXZgd85dFI+YJPby8pSAiEhyUg2JiIiIiIgEJqKEJJJZf82sl5lNN7M5ZvZ+dMMUEREREZFUVGGTrbBZf0/Cj97yuZm97Jz7KmybA4CHgN7OuSVm1ihWAYuIiIiISOqIpIYkkll/BwLPO+eWADjnNPOviIiIiIhUKJKEpMJZf4E2QD0zm2Rm08zsgmgFKCIiIiIiqSuSUbYimfW3KtANOAGoBXxsZp845xbstiOzIcAQgBaarUlEREREJO1FUkMSyay/S4HXnXObnHOrgMlAl5I70qy/IiIiIiISLpKEpMJZf4GXgN+YWVUzywS64yfkEhEREZEoKSiA7GyoUsW/FxQEHZHIvquwyZZzboeZFc/6mwE8Xjzrb2j9SOfcXDN7HZgJ7AQedc7NjmXgIiIiIumkoACGDIHCQr+8eLFfBk2KKcnNnCvZHSQ+cnJy3NSpUwM5tojIvjKzac65nKDjiBdds0WCl53tk5CSsrJg0aJ4R5Nc0u2anWw0U7uIiIhIEliypHLlIslCCYmIiIhIEihrgFINXCrJTgmJiIiISBIYMQIyM3cvy8z05SLJLJJ5SEREYmfHDti0CbZu3fO1bVvp5fvyKrnP666DK64I+iyIiFSouOP6sGG+mVaLFj4ZUYd2SXZKSERk7znnk4m1a2HdurLfy1u3aVN0YsnIgBo1yn5Vr+7f69TZvVxtHUQkieTlKQGR1KOERERg+3aYNQt+/rn85KG0RKOoqPx9V68OBxwAdevuem/adPfl/fYrP5koK7kIf2VkxOdciYiISFQpIREp6bvvoFEjf5OcDj74AC67DOaWMZdp3bp7JhMdO+5eVt57zZrx/TwiIiKSVJSQJLq33oK774a//Q26dQs6mtQ3ZQr06uUTkn/9C849F8yCjio21qyB66+HRx/1g9g/+aQf5D48mahTx08HLCIiIhIjutNIVJs3w9VXw8knw5tv+vfZs4OOKrUtXw7nnOP7FDRqBP37w/HHp955dw7GjIF27eC//4U//xnmzIELLoBjj4XDDvMJSt26SkZEREQk5lRDkoi++AIGDfJNaK6+Gi69FE46yb8mT4bWrYOOMPVs2wb9+sH69b5Wqn17eOQRuPFG6NoVrroKbrvN36Qns6+/ht//Ht55B7p3h7ff9gmIiIiIADBt2rRGVatWfRTohB7eR8NOYPaOHTsGd+vWbUVpGyghSSRFRfDPf8Ktt0LDhr5m5KST/Lq334aePeHEE32bf40MFF1/+hN89BE88wx06uTLhg6Fvn3hppvgP/+Bp5/2v5/zz0++moOtW33sI0b4Ph0PPwxDhiTf5xAREYmxqlWrPtqkSZP2DRs2/LlKlSou6HiS3c6dO23lypUdfvzxx0eBs0rbRncjieK773zfhRtvhLPP9iMeFScjAB06wBtv+FGNTjwRfvwxsFBTzlNPwQMP+KTkvPN2X9egAYwcCZ9/DoccAhddBMcc42uxksX77/tanltugT59fM3b0KFKRkRERErXqWHDhuuVjERHlSpVXMOGDdfha5xK3yaO8UhpnIMnnoAuXWDmTBg92j+lr19/z21//WuYOBGWLfPJyurVcQ835Xz5pR9h6rjj4M47y96uWzdfg/Lf/8K330JOjm/6lMi/g9Wr4ZJLfKK7dSu89pr/2zrooKAjExERSWRVlIxEV+h8lpl3KCEJ0qpVvknQxRf7ZGPmTN93pLxRnY4+Gl56yfcF6N3b93mQvbN6Nfz2t7553LhxULWCFoxVqvgakvnz4Q9/8H1M2rSBUaMqnosjnpzzI2a1a+cT3Btu8B3ze/cOOjKJEzPrbWbzzewbM7uhlPV5ZjYz9JpiZl2CiFNERASUkATntdegc2f43//grrt8J+OsrMh+9oQT4NlnYfp0OOMMKCyMbaypqKgIBg70tU0TJvikJFIHHAD33utrVzp39s2fjjgCPv44dvFGav58PzLYRRf5ZOmLL+Dvf4fMzKAjkzgxswzgQeBUoAMwwMw6lNjsO6Cnc+4w4HYgP75RiohIWX788ceMdu3adWjXrl2HBg0adGnUqNFhxctbtmyJaC6Cvn37Zs+YMaNGedv8/e9/b/jwww+X0iQn/pSQxFthIVxxBZx2mu+f8PnncN11lZ9l+swz/dCtH33k+5xs3RqbeFPVLbf4QQMeeggOP3zv9tG5M7z3Howd6/v0HH20r+366afoxhqJrVvhr3/1I2ZNn+5rbT74wMco6eYI4Bvn3ELn3DbgGSA3fAPn3BTn3M+hxU+AZnGOUUQkZYwcSf2DD6ZzlSp0O/hgOo8cyT7d5Ddp0qRo3rx5X82bN++rCy64YOXQoUN/Kl6uWbOmA9i5cydF5bTOeO655xZ16dKl3JvDv/zlLyt///vfr9mXWKNFCUk8TZ3qm2Y99BD88Y8+GdmXIVfPO883G3rzTRgwAHbsiF6sqeyFF/xEk0OGwO9+t2/7MvPzlcyf7ycZLCjwNRP/+U/8fh/vvef/jm67zc+jMneuRtBKb02B78OWl4bKyvI74LWYRiQikqJGjqT+tdeStXw51Z2D5cupfu21ZO1rUlKa2bNn12jdunXHgQMHtujYsWOHJUuWVBswYEBWp06d2rdq1arjdddd90sn0W7durWdMmVKre3bt1OnTp2ul19+edO2bdt26Nq1a7sffvihKsAf/vCHg4cPH96oePvLL7+8aefOndtnZ2d3euutt/YDWL9+fZVTTjnl0LZt23Y488wzW3bq1Kn9lClTakX7s+mOJR527IA77oCjjoJNm3zzrH/9yw+/uq8uucTf/L7wgm+ms3Pnvu8zlc2bBxde6JtY3Xdf9PZbu7bvFD9rFhx5JFxzDfzqVzBpUvSOUdKqVf53fvzx/m/sjTf80MRNmsTumJIMSqvOL7Vzppkdh09Iri9j/RAzm2pmU1euXBnFEEVEUsPw4TTdsmX3++ktW6gyfHi5D4L22rffflvzsssuWzV37tyvWrZsuf3ee+9dOnv27Llz586d89577+0/bdq0PW4uN27cmNGrV68N8+fP/yonJ2fjgw8+2KC0fTvnmDVr1twRI0Z8P3z48IMB7rzzzkaNGjXaPn/+/K9uvPHGH+fOnRuTNuBKSGLt22/97Nc33+wn3ps5099ARtMf/uDnlygogMsv952aZU8bNvjmbTVr+n4jNcptWrl32raF11/3CeKGDX70rgEDYOnS6B3DOT/aV9u2/nd+442+0/rJJ0fvGJLMlgLNw5abActKbmRmhwGPArnOuVKHi3PO5TvncpxzOQ0r089KJE4KCiA721cIZ2f7ZZF4+vFHqlemfF81b958a8+ePX/pPPz444/X79ChQ/uOHTt2WLhwYc2ZM2fuUXtRs2bNneeee+56gG7duhUuWrSo1Nj69eu3FuDoo48uXLp0aXWAjz/+uHZeXt4agKOOOmrzoYceujkWn0sJSaw4B48+6ofznTvXP7l++mmoVy82x7vxRj+a0qhR8H//p6SkJOd8bcLXX8P48dAshk3mzXbN93HrrT45adcO/vGPfe/rM2+eT3IuucTPJj99uk9Ga0W99lSS1+dAazNraWbVgf7Ay+EbmFkL4HngfOfcggBiFNlnBQW+derixf4Sv3ixX1ZSIvHUpAnbKlO+r2rVqvVLU5hZs2bVGDVqVOPJkycvWLBgwVfHHnvs+s2bN+9RS161atVfbgozMjJcUVFRqR3ja9asubPkNi5O95NKSGJhxQr/JP7SS6F7d18rMmBA7I/7t7/BlVf65mDDh8f+eMnkn/+E55/3I5r16hWfY9aq5ft1fPWVn8zyhht8X4833qj8vrZs8R3xDzsMZszwfYcmT4aOHaMetiQ359wO4ErgDWAuMN45N8fMhprZ0NBmtwAHAg+Z2XQzmxpQuCJ7bdiwPQeZLCz05SLxcsst/FCzJru1l69Zk5233MIPsT722rVrM/bbb7+ievXqFS1evLja5MmT94/2MY466qiNY8eOrQfw2Wef1Vq4cGFMnoBWMPGCVNqrr/qn1+vWwT33wNVXx69zsZnvT7Jxo78Rrl3bzz6e7t56y9cg9e/v+3bE2yGHwIsv+qGer77azwfSp4//+2jZsuKff+cdPwnj119DXp5POBs3jn3ckrSccxOBiSXKRob9ezAwON5xiUTTkiWVKxeJhaFDWQO+L8mPP1K9SRO23XILPxSXx1KPHj0KW7duvaVNmzYdW7RosbVbt24bo32MG264YUW/fv1atmnTpkPnzp0LW7Vqtbl+/fpRn3zN4lUVU1JOTo6bOjWFHspt2uRv/keN8k+xCwqgU6dgYimeY2P8eBg50s9Enq4WLfKzqh90EHzyCey3X7DxbN0K//433H67H4Dghhvgz38uvcnVihX+b2rMGGjVCh5+2Ne0SEIws2nOuZyg44iXlLtmS9LLzvbNtErKyvKXfpFwlblmz5gxY1GXLl1WxTqmZLB9+3a2b99umZmZbtasWTV69+7dZtGiRbOqVatW6X3NmDGjQZcuXbJLW6cmW9Hw6ad+RKX8fN9/47PPgktGwM9pMno0nH66f7I+ZkxwsQRp82Y/E/uOHb4fR9DJCPiO9Dfc4IcJzs31NVkdOvgalOKHAzt3wmOP+X4n48bBTTf5Zn9KRkREfjFixJ5zvmZm+nIRiY5169ZlHH744e3atm3b4Zxzzjn0/vvvX7w3yUhFkiYhSciRNHbs8DeUPXr4J9/vvef7KsRi9KbKql7dz+beq5fvzP3CC0FHFF/O+RnUv/zS/7G0ahV0RLtr1gyeeQbefdcnSmefDaee6pt19eoFgwf7pHbGDF+bok7rIpKggvp+zsvzzwGzsnyL5awsv5yXF5/ji6SDBg0aFM2ZM2fu/Pnzv1qwYMFXv/3tb9fH4jhJ0YekeCSN4s5rxSNpQIAXnq+/hkGDfG3I+efD/fdD3boBBVOGWrXg5Zf9cLDnnQevvAKnnBJ0VPHx8MPw1FM+YTz99KCjKdtxx/mk6cEH/Yhcb7wB9ev7GpKLLtLkhiKS0IL+fs7LUwIikgqS4m4noUbScM73E+naddcQsk89lXjJSLHatWHiRD8a09ln+5GZUt2UKb7z+Bln+PlfEl21ar6z/fz5flCCefP8wAhKRkQkwSXU97OIJK2kuONJmJE0fvoJzjrLNwXq0cPPyt2vX5yD2AsHHABvvunrs884Az7/POiIYmf5cujb13/W0aOT66a+SRM/yaUmoBORJJEw388iktSS4m6tRYvKlUfdtm2+c3HnzvD223DffX427qZN4xRAFDRs6GNv0MA325o1K+iIom/bNp8grlvnO4kfcEDQEYmIpLTAv59FJCUkRUISyEgaW7f6OUUuvtjP+dC/v++IPG0aXHVVcj15L9a0qZ/TIjMTTjoJFqTYBM3XXQcffQSPPx7sKGciImlCI12JRN8RRxzRdsKECbtNcjh8+PBGgwYNKjPVz8zM/BXAokWLqvXu3fuQsvY7efLkzNLWhR9nw4YNv9zk9uzZs9WqVasyKvcJKi8p7qrjNpLGli2+E/gFF/gk5Iwz/JP23FzfIfzTT/0QrcmsZUtfU7Jzpx9GtrRB3JPR6NF+YIE//cl34BcRkZjTSFci0devX7/VY8eOrR9eNmHChPqDBg2qcLLF7Ozs7a+//vrCvT32qFGjGm/cuPGX/OD999//pkGDBlGfCLGkpEhIwF/cFi3y99GLFkXxYrd5sx8SNy8PGjXyycerr8I55/jO4D/9BE884ZOTGIy7HIh27Xyfkg0b4IQTfL+LZPbll35Yl1694M47g45GRCStxOz7WSRNnX/++T+/8847dTdv3mwA8+fPr75ixYpq3bt3LzzqqKPadOjQoX2bNm06jBkzZo+26fPnz6/eunXrjgAbN260M84445A2bdp0OP300w/ZsmWLFW+Xl5fXolOnTu1btWrV8dprrz0Y4I477mi0YsWKaj179mzTvXv3NgBNmzbtvHz58qoAt912W+PWrVt3bN26dcfhw4c3Kj7eIYcc0rF///5ZrVq16tijR4/WGzdutJJxVSQphv2Nuk2b/HwPzz0H//ufX27QwDfL6tvXD8WaXQSWagAADfdJREFUKslHWbp29efgxBP96/33/TlINmvW+MkPGzTw/XyqpueftIiIiMTAJZc0Z/bscps5VVqnToU8/vj3Za1u0qRJUZcuXTZNmDCh7qBBg9Y++eST9c8666yfa9euvfPVV1/9pn79+juXL19etXv37u0GDhy4tkoZ3QjuvvvuRrVq1dq5YMGCrz799NNaPXr0+KWZzz333PND48aNi3bs2MHRRx/d9tNPP6110003rXj44Ycbv//++wsOOuigHeH7+uCDDzKffvrpA6dNmzbXOUe3bt3an3DCCRsaNGhQtGTJkppjxoxZePTRRy8+7bTTDnnqqafqXX755RXW5oRLmhqSfbZxo79h7dfP14T06+cnMjz/fN+EaflyX8988smpn4wUO/JI3xRt4ULf0X3duqAjqpyiIhg4EJYtgwkT/O9VREREJMmde+65a8aNG1cP4Pnnn69//vnnr9m5c6ddc801zdq0adPhuOOOa7NixYrqS5cuLfNJ7Icfflj7/PPPXw3QvXv3zW3atPllkO4nn3yyfocOHdp36NChw9dff11zxowZNcuLZ9KkSbVPO+20tfvvv//OunXr7jz99NN/fu+99+oANG3adOvRRx+9GeBXv/pV4aJFiyo9Q3hqP05ev97XgDz3nK8N2LLFD6168cW+JuQ3v4GMmPfTSWzHHedv5vv08RMIvvGGnzk8GRRPJPjII3DEEUFHIyIiIqmmnJqMWMrLy1t70003Nf/www8zt2zZUuWYY44pvO+++w5cvXp11VmzZs2tUaOGa9q0aefNmzeXW7lgtmfrqXnz5lV/4IEHGk+bNm1uw4YNi84555zsLVu2lLsf51yZ66pXr/7LyoyMDFdRTKVJvRqStWt9B+fcXP/EPC/Pd0YfMsRPCrh0KTzwgO9vkO7JSLHTToOnn4aPP/aJyZYtQUdUsRdf9MO4XHopDB4cdDQiIiIiUVO3bt2dRx555IbBgwdn//a3v10DsG7duowGDRpsr1GjhnvllVfqLFu2rHp5+zjmmGM2jhkzpj7A559/XnPBggWZAD///HNGrVq1dtavX7/o+++/rzpp0qRfZvfeb7/9itatW7dHfnD88cdvnDhx4gEbNmyosn79+ioTJ06sd9xxx22I1udNjRqSNWv86FjPPgtvvQXbt0Pz5nD55b4m5Mgjk3OY3njq29cPl3vRRX6UqueeS9yma/Pn+5HQjjjCj6wlIiIikmL69++/5sILLzx07NixCwEGDx685tRTT23VqVOn9h07dixs2bJluU+Qr7vuuhX9+/dv2eb/27v3GCnLK47j37PQpkW8tBEqogteVrlmgV1FxDSmtkEUpVFJFEq3psZUqbWmTVH5Q+MlNahNTayYDdrauNE0dBNIQ2wVm7T/lDiL3RS1WqNAERYWm1p1SetmTv94ZmRdF3d2mXmfnff5fZLJzLzzsnseZnPmPfPczjln1uzZs/vmzp37IcCiRYsOz5kzp6+pqWl2Y2Pjf1taWj4o/5u2trZDS5cubZo8efJH27dv/3h/iIsuuqhv5cqV7y5YsGAmwOrVq3sXL158+PXXX//MoqhS9lldMLXU2trqhUJh9D/g0CHYvDkUIdu2QX9/WG9wxYpwcX3eeSpCRuOxx2DNGrjuutDTNNZ6kd5/HxYuDO9/V1coPEUiMLMud2+NHUdWjjlni+RMRwesWxd2pW9sDJ32WmFs7BpJzu7u7t7V3Nx8qNYxpaa7u/vk5ubm6UO9Vl89JAcPhqE6mzbBiy+GSc1nnhn2nrjmGmhpCQuhy+jdfHO46L/99jCXpL197PyfuocenDfeCD1hKkZERCSCjo4wEryvNEV49+7wHFSUiIxGfRUkV10VduJuaoK1a0MRMm/e2Llgzou1a8OqZPfdFxaWb2uD+fPh+OPjxrV+PXR2wsMPh8n4IiIiEaxbd6QYKevrC8dVkIiMXEUFiZldCjwCjAM2uvsDg16/GNgMvF061Onu91QxzmD9epg4EebOVRFSa/fcEya3P/RQmFtiBueeG3qhWlvD/fz54f3IwvPPw513hvktt92Wze8UEREZwp49IzsudadYLBatoaEhzryGHCoWiwYUj/b6sAWJmY0DfgF8A9gLvGRmW9z91UGn/tndlx1LsMO68MKa/ngZwAwefDAMh+vqCrdCIezd0tFx5JwZM0JxUi5U5s2rfpGya1eY0zJrFjzxhIpRERGJqrExDNMa6rjkws7e3t5ZkyZNek9FybErFovW29t7IrDzaOdU0kNyPvCmu78FYGbPAsuBwQWJ5NEpp4T9SS6//Mix/fuPFCldXWFRgaefDq+ZwcyZny5SRru3yeHDcPXVYdGCzs762SNFRERy6/77PzmHBGDChHA8z1KZyN/f339DT0/Pxp6enjnkcYuM7BWBnf39/Ufdp6GSgmQqMHBTmL3AwiHOW2Rm3cA+4Mfu/spIIpU6MmUKLFsWbmXlIqVQCPcvvBBW6YKw2tmMGUeGerW0VFakuMNNN8GOHWFH+aam2rVJRESkQuWL8BQuzstiT+TPshhqaWk5CFxZm58uQxl22V8zWwEscfcbSs9XA+e7+y0DzjkBKLr7B2Z2GfCIu3/q6tHMbgRuBGhsbGzZPVR/p+THvn2f7EkpFKCnJ7zW0HCkJ6VcqMybF75iKtuwIaz6ddddcPfdUZogcjRa9ldEUjJ9+tDD1KZNCyOra2lwMQThcqG9vfKiJLWcXW8qKUgWAXe7+5LS8zsA3P2nn/FvdgGt7n7UNZz14ZaocpFS7kkpFODAgfBaQ0OYJ9LSAmedBffeC0uWhP1mtKeMjDGpfbgpZ4ukraEhDFwYzCwsyFlL1SiGUsvZ9aaSIVsvAU1mdgbwDnAtsHLgCWZ2CnDA3d3MzieMt3u32sFKDpx6arhdcUV47v7JnpRCAZ57LhQpZ58dhn2pGBEZkQpWRrTS65cBfcB33H1H5oGKSN2IOZFfq5rl37AFibv3m9n3gd8TPtyedPdXzOx7pdcfB64BbjKzfuAwcK3H2gJe6osZTJ0ableWhmuWi5SJE+HEE+PGJ1JnKlwZcSnQVLotBDYw9NxAEREg7kR+rWqWfxXtQ+LuW4Gtg449PuDxo8Cj1Q1NklUuUkRkNCpZGXE58OvSF0d/MbOTzGyKu+/PPlwRqQcxJ/KnuqpZSjQWRkQkX4ZaGXFwhV/JOZjZjWZWMLNCb29v1QMVkfqyalWYs1EshvusVhVbtSpMYJ82LXxnOW3ayCa0y9hXUQ+JiIjUjaF2Dh08hLaSc3D3dqAdwqT2Yw9NRGR0Vq1SAZJn6iEREcmXvcDpA56fRtgfaqTniIiIZEIFiYhIvny8MqKZfZ6wMuKWQedsAb5twQXAe5o/IiIisWjIlohIjlS4MuJWwpK/bxKW/b0+VrwiIiIqSEREcqaClREdWJN1XCIiIkPRkC0REREREYlGBYmIiIiIiESjgkRERERERKJRQSIiIiIiItFYmNsY4Reb9QK7o/zy0TsZOBQ7iIyl2GZIs90pthlG3+5p7j6p2sGMVcrZdSXFdqvN6VDOzqFoBUk9MrOCu7fGjiNLKbYZ0mx3im2GdNudglTf2xTbrTanI9V2552GbImIiIiISDQqSEREREREJBoVJCPTHjuACFJsM6TZ7hTbDOm2OwWpvrcptlttTkeq7c41zSEREREREZFo1EMiIiIiIiLRqCAZhpmdbmZ/NLPXzOwVM7s1dkxZMrNxZvaymf0udixZMLOTzGyTmf299J4vih1TFszsttLf904ze8bMvhA7pmozsyfN7KCZ7Rxw7Mtm9ryZ/aN0/6WYMUp1pJy3U8vZkGbeTiFng/J2SlSQDK8f+JG7zwQuANaY2azIMWXpVuC12EFk6BHgOXefATSTQNvNbCrwA6DV3ecA44Br40ZVE78CLh107HZgm7s3AdtKz6X+pZy3U8vZkFjeTihng/J2MlSQDMPd97v7jtLj9wmJbmrcqLJhZqcBlwMbY8eSBTM7Afgq8ASAu//P3f8dN6rMjAe+aGbjgQnAvsjxVJ27/wn416DDy4GnSo+fAr6ZaVBSE6nm7dRyNiSdt3Ofs0F5OyUqSEbAzKYD84HtcSPJzM+BnwDF2IFk5EygF/hlacjDRjM7LnZQtebu7wAPAXuA/cB77v6HuFFl5ivuvh/CRSwwOXI8UmWJ5e3UcjYkmLcTz9mgvJ1LKkgqZGYTgd8CP3T3/8SOp9bMbBlw0N27YseSofHAAmCDu88HPiSBruDS+NvlwBnAqcBxZvatuFGJHLuU8naiORsSzNvK2ZJHKkgqYGafI3yodbh7Z+x4MrIYuNLMdgHPAl8zs6fjhlRze4G97l7+JnUT4YMu774OvO3uve7+EdAJXBg5pqwcMLMpAKX7g5HjkSpJMG+nmLMhzbydcs4G5e1cUkEyDDMzwtjU19z9Z7HjyYq73+Hup7n7dMJkuRfdPdffwLh7D/BPMzu3dOgS4NWIIWVlD3CBmU0o/b1fQs4nhQ6wBWgrPW4DNkeMRaokxbydYs6GZPN2yjkblLdzaXzsAOrAYmA18Dcz+2vp2J3uvjViTFI7twAdZvZ54C3g+sjx1Jy7bzezTcAOwupEL5PDnXDN7BngYuBkM9sL3AU8APzGzL5L+JBfES9CqSLl7bQklbdTydmgvJ0S7dQuIiIiIiLRaMiWiIiIiIhEo4JERERERESiUUEiIiIiIiLRqCAREREREZFoVJCIiIiIiEg0KkhERERERCQaFSQiIiIiIhKNChIREREREYnm/9kNse5/qMogAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "embedding_dim = 100\n", "\n", "embedding_matrix = np.zeros((max_words, embedding_dim))\n", "for word, i in word_index.items():\n", " embedding_vector = embeddings_index.get(word)\n", " if i < max_words:\n", " if embedding_vector is not None:\n", " # Words not found in embedding index will be all-zeros.\n", " embedding_matrix[i] = embedding_vector\n", " \n", "model = models.Sequential()\n", "model.add(layers.Embedding(max_words, embedding_dim, input_length=maxlen))\n", "model.add(layers.Flatten())\n", "model.add(layers.Dense(32, activation='relu'))\n", "model.add(layers.Dense(1, activation='sigmoid'))\n", "model.summary()\n", "\n", "model.layers[0].set_weights([embedding_matrix])\n", "model.layers[0].trainable = False\n", "\n", "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n", "history = model.fit(x_train, y_train, epochs=10, batch_size=32, validation_data=(x_val, y_val), verbose=0)\n", "\n", "acc = history.history['acc']\n", "val_acc = history.history['val_acc']\n", "loss = history.history['loss']\n", "val_loss = history.history['val_loss']\n", "epochs = range(1, len(acc) + 1)\n", "\n", "plt.figure(figsize=(12, 5))\n", "plt.subplot(1, 2, 1)\n", "plt.plot(epochs, acc, 'bo', label='Training')\n", "plt.plot(epochs, val_acc, 'r', label='Validation')\n", "plt.title('Training and validation accuracy')\n", "\n", "plt.subplot(1, 2, 2)\n", "plt.plot(epochs, loss, 'bo', label='Training')\n", "plt.plot(epochs, val_loss, 'r', label='Validation')\n", "plt.title('Training and validation loss')\n", "plt.legend(bbox_to_anchor=(1.02, 0.2), loc=2, borderaxespad=0.5)\n", "plt.show()\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The model quickly starts overfitting and validation accuracy stalls around 50% with only 200 samples. If you increase the number of training samples, this will quickly stop being the case." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "25000/25000 [==============================] - 1s 55us/step\n" ] }, { "data": { "text/plain": [ "[0.7815723500633239, 0.5814]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_dir = os.path.join(imdb_dir, 'test')\n", "\n", "labels = []\n", "texts = []\n", "\n", "for label_type in ['neg', 'pos']:\n", " dir_name = os.path.join(test_dir, label_type)\n", " for fname in sorted(os.listdir(dir_name)):\n", " if fname[-4:] == '.txt':\n", " f = open(os.path.join(dir_name, fname), encoding=\"utf8\")\n", " texts.append(f.read())\n", " f.close()\n", " if label_type == 'neg':\n", " labels.append(0)\n", " else:\n", " labels.append(1)\n", "\n", "sequences = tokenizer.texts_to_sequences(texts)\n", "x_test = preprocessing.sequence.pad_sequences(sequences, maxlen=maxlen)\n", "y_test = np.asarray(labels)\n", "model.evaluate(x_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We tokenize the test data and evaluate the model on the test data. The test accuracy is around 50%." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Recurrent Neural Networks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Simple RNN" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Feedforward networks has no memory. However, a recurrent neural network (RNN) processes sequences by iterating through the sequence elements and maintaining a state containing information relative to what it has seen so far. In effect, an RNN is a type of neural network that has an internal loop.\n", "\n", "\n", "\n", "The hidden state acts as the neural networks memory. It holds information on previous data the network has seen before.\n", "\n", "\n", "\n", "The hidden state is calculated as follows\n", "\n", "\n", "**Pseudocode RNN**\n", "```python\n", "state_t = 0\n", "for input_t in input_sequence:\n", " output_t = activation(dot(W, input_t) + dot(U, state_t) + b)\n", " state_t = output_t\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The final output is a 2D tensor of shape `(timesteps, output_features)`, where each timestep is the output of the loop at time t. Each timestep t in the output tensor contains information about timesteps 0 to t in the input sequence—about the entire past. For this reason, in many cases, we don't need this full sequence of outputs; we just need the last output (output_t at the end of the loop), because it already contains information about the entire sequence." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "embedding_3 (Embedding) (None, None, 32) 320000 \n", "_________________________________________________________________\n", "simple_rnn_1 (SimpleRNN) (None, 32) 2080 \n", "=================================================================\n", "Total params: 322,080\n", "Trainable params: 322,080\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model = models.Sequential()\n", "model.add(layers.Embedding(10000, 32))\n", "model.add(layers.SimpleRNN(32))\n", "model.summary()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "embedding_4 (Embedding) (None, None, 32) 320000 \n", "_________________________________________________________________\n", "simple_rnn_2 (SimpleRNN) (None, None, 32) 2080 \n", "=================================================================\n", "Total params: 322,080\n", "Trainable params: 322,080\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model = models.Sequential()\n", "model.add(layers.Embedding(10000, 32))\n", "model.add(layers.SimpleRNN(32, return_sequences=True))\n", "model.summary()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyQAAAE/CAYAAAC3hAd1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXhU5fn/8c8dIktQUWQVJaDsoKhQFXHBfYe6VTSg0CJaaxe/ta17/VGptVqtigVTRFRS1LpVW9zqhkurQgVBEEQkEQQBlc2AkOT+/fFMJIQsE5jMycy8X9c1VzLnnDnnPifwzLnPs5m7CwAAAACikBV1AAAAAAAyFwkJAAAAgMiQkAAAAACIDAkJAAAAgMiQkAAAAACIDAkJAAAAgMiQkDRwZtbIzDaYWcdEbhslM+tiZgkfb9rMTjCzJRXeLzCzo+LZdgeONdHMrt3RzwNAZZT3ddpvypf3ZnazmU1O9H6BVJQddQDpxsw2VHibI+lbSaWx95e6e0Fd9ufupZJ2TfS2mcDduydiP2Y2StIwdx9UYd+jErFvAKmL8r7hoLwHUhsJSYK5+3dfELEnMqPc/d/VbW9m2e5ekozYgNrw7xGIH+U9ACQGTbaSLFZF+6iZTTWz9ZKGmdkAM/uvma0xs+VmdreZ7RLbPtvM3Mw6xd5Pia1/zszWm9l/zKxzXbeNrT/VzBaa2Vozu8fM3jKzEdXEHU+Ml5rZIjP72szurvDZRmZ2p5l9aWafSDqlhutzvZk9UmnZvWZ2R+z3UWY2P3Y+n8SeZlW3r6VmNij2e46ZPRyL7UNJ/ao47uLYfj80s8Gx5QdIGifpqFjziNUVru1NFT5/WezcvzSzp82sfTzXpi7XuTweM/u3mX1lZivM7NcVjnND7JqsM7MZZrZ3Vc0lzOzN8r9z7HpOjx3nK0nXm1lXM3s1di6rY9etRYXP58bOcVVs/V1m1jQWc88K27U3s2Iz26u68wXSGeU95X1N5X0V5/D9WDxrzOwVM+teYd21ZvZ5rHz/qMK5Hm5m/4st/8LMbov3eECD4u686uklaYmkEyotu1nSZklnKiSEzSR9T9JhCjVW+0laKOmK2PbZklxSp9j7KZJWS+ovaRdJj0qasgPbtpG0XtKQ2Lr/k7RF0ohqziWeGP8hqYWkTpK+Kj93SVdI+lDSPpL2kjQ9/NOr8jj7SdogqXmFfa+U1D/2/szYNibpOEkbJR0YW3eCpCUV9rVU0qDY77dLek3SnpJyJc2rtO0PJLWP/U0ujMXQNrZulKTXKsU5RdJNsd9PisV4kKSmkv4i6ZV4rk0dr3MLSV9I+rmkJpJ2l3RobN01kmZL6ho7h4MktZTUpfK1lvRm+d85dm4lkn4sqZHCv8duko6X1Dj27+QtSbdXOJ+5sevZPLb9wNi6fEljKxznl5Keivr/IS9eyXiJ8p7yvu7l/c2SJsd+7xmL47jY3+ja2HXfRVJvSYWS2sW27Sxpv9jv70m6IPb7bpIOi/r/Ai9eO/KihiQab7r7s+5e5u4b3f09d3/H3UvcfbHCjd0xNXz+cXef4e5bJBUoFIx13fYMSbPc/R+xdXcqfJlVKc4Yb3H3te6+ROHLoPxYP5B0p7svdfcvJf2hhuMsVrjhHRJbdKKkNe4+I7b+WXdf7MErkl6WVGVHxkp+IOlmd//a3QsVnoJVPO5j7r489jf5m8LNRf849itJeZImuvssd98k6WpJx5jZPhW2qe7abKOW6zxY0mfufpe7f+vu69z93di6UZKudfePY+cwy92/ijP+Incf7+6lsX+PC939ZXff7O4rFf5tlMcwQFIrSb9x929i278VW/egpAvNzGLvh0t6OM4YgHRFeV/9cTK6vK9kqKRn3P2V2N/oDwoPnQ5TeGjUVFJvC83+Po1dOykkll3NbC93X+/u78R5HkCDQkISjc8qvjGzHmb2LwtNcNZJGqNw01edFRV+L1bNHRur23bvinG4uys8YapSnDHGdSyFJz01+ZukC2K/X6jwxVoexxlm9o6FJktrFJ5W1XStyrWvKQYzG2Fms2NV5Wsk9Yhzv1I4v+/25+7rJH0tqUOFbeL6m9VynfeVtKiaGPaV9Emc8VZW+d9jOzN7zMyWxWKYXCmGJR461G4jlpiUSDrSzPpI6ijpXzsYE5AuKO9rlrHlfS37LVP4G3Vw9wUKNc5jJK200ASwXWzTkZJ6SVpgZu+a2WlxngfQoJCQRKPyEIj3KTwl6uLuu0u6UaGKuj4tV6hSlyTFnmp3qH7znYpxucKNbLnahql8VNIJsSdOQxS+sGRmzSQ9LukWher1PSS9GGccK6qLwcz2kzReodnSXrH9flRhv7UNWfm5QrOA8v3tptBUYFkccVVW03X+TNL+1XyuunXfxGLKqbCsXaVtKp/frQqjBR0Qi2FEpRhyzaxRNXE8JGmYQu3IY+7+bTXbAZmC8r5mmVze17TfLIW/2TJJcvcp7j5QoblWI4XrIndf4O5DFZrl/UnSE2bWdCdjAZKOhKRh2E3SWknfWOgUfGkSjvlPSYeY2Zlmlq3QL6F1PcX4mKRfmFkHCx2cf1PTxu7+hUI/hwckLXD3j2Ormij0a1glqdTMzlDo6xBvDNea2R4Wxu2/osK6XRW+hFYpfFePUnhiVu4LSftYhc7llUyV9CMzO9DMmih8Ubzh7tU+gaxBTdf5GUkdzewKM2tsZrub2aGxdRMl3Wxm+1twkJm1VPhiXqHQmbaRmY1WhS+9GmL4RtJaM9tX0lUV1v1H0peSfm+h42gzMxtYYf3Dks5VeNL50A6cP5DuKO8ryPDyvnLMg81sUOzYv1Lo9/OOmfU0s2Njx9sYe5UqnMBwM2sVq1FZGzu3sp2MBUg6EpKG4ZeSLlYofO5TeGJUr2JfAudLukPhBnN/Se8rPBlPdIzjFdr+zlHogPd4HJ/5m0Knxb9ViHmNpCslPaXQUfBchS/aePxW4cndEknPqcLNsrt/IOluSe/GtukhqWI73JckfSzpCzOrWBVf/vnnFarSn4p9vqNCO+MdUe11dve1Cm2sz1HoVLlQW9t13ybpaYXrvE6hzXfTWNOMSxQ6SK5W6OReWxvj30o6VOHL7RlJT1SIoUShPXpPhdqSIoW/Q/n6JQp/583u/nYdzx3IBJT328vU8r7ifj9UuObjFZKlUyQNjvUnaSLpjwpl+AqFGpnrYx89TdJ8C6O43S7pfHffvLPxAMlm4X4FmS7WBOdzSee6+xtRx4PUZWYPSVrs7jdFHQuA7VHeA2hoqCHJYGZ2ipm1iFUD36DQIfndWj4GVCvWPnuIpElRxwJgK8p7AA0ZCUlmO1LSYoVq4FMkfZ9OyNhRZnaLwlwov3f3oqjjAbANynsADRZNtgAAAABEhhoSAAAAAJEhIQEAAAAQmeyoDtyqVSvv1KlTVIcHgJ0yc+bM1e5e01wOaYUyG0Aqy7QyO9VElpB06tRJM2bMiOrwALBTzKww6hiSiTIbQCrLtDI71dBkCwAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkak1IzGySma00s7nVrDczu9vMFpnZB2Z2SOLDBNJbQYHUqZOUlRV+FhRw7HQ+dibg+gIA4hXPxIiTJY2T9FA160+V1DX2OkzS+NhPAHEoKJBGj5aKi8P7wsLwXpLy8jh2uh07E3B9AQB1Ye5e+0ZmnST90937VLHuPkmvufvU2PsFkga5+/Ka9tm/f39n1l8gPD0urGL+2NxcackSjt1Qj21mM929fyLjasjqUmZH+bcFgKpkWpmdahLRh6SDpM8qvF8aW7YdMxttZjPMbMaqVasScGiko0xr6lFUVLflHDu1j50MZnaKmS2INaW9uor1vzKzWbHXXDMrNbOWiTp+ul9fAEBiJSIhsSqWVVnt4u757t7f3fu3bt06AYdGuilv6lFYKLlvbeqRzklJx451W86xU/vY9c3MGkm6V6E5bS9JF5hZr4rbuPtt7n6Qux8k6RpJr7v7V4mKIZ2vLwAg8RKRkCyVtG+F9/tI+jwB+0UGuu66re3OyxUXh+XpauxYKSdn22U5OWE5x06/YyfBoZIWuftid98s6RFJQ2rY/gJJUxMZQJpfXwBAgiUiIXlG0kWx0bYOl7S2tv4jQHUysalHXp6Unx/a15uFn/n5yen8y7GTf+wkqEsz2hxJp0h6IpEBlF/fbvtuVLZK0u36AgASrNZO7WY2VdIgSa0kfSHpt5J2kSR3n2BmpjAK1ymSiiWNdPdaez7SqR1VoTMsUkVD7SBpZudJOtndR8XeD5d0qLv/tIptz5c0zN3PrGZfoyWNlqSOHTv2K6zqP2dNbrtNuvZaqXNnqUuX7V+dOkmNG9dtnwCwAxpqmY2g1mF/3f2CWta7pJ8kLCJktLFjtx0uVKKpB1BHdWlGO1Q1NNdy93xJ+VJ4iFTnSAYOlH79a2nRovB6801p/fqt67OywtOGqpKV/faTmjat8yEBAKknnnlIgKQpb9Jx3XWhmVbHjiEZoakHELf3JHU1s86SlikkHRdW3sjMWkg6RtKweovkiCPCq5y7tHq19PHHW5OU8tfUqdKaNRUDlPbZp+pkZf/9pebN6y1sAEBykZCgwcnLIwEBdpS7l5jZFZJekNRI0iR3/9DMLoutnxDb9CxJL7r7N0kLzkxq3Tq8KiYq5b76avtEZdEi6emnpcpDxbdvX3Wy0qWLtPvuyTmfTLdypfT++9L//hdeGzZIw4dL55wjNWkSdXTpa/Vq6aGHpFdflU4+WRo2TNpjj6ijAnZKXBMj1gf6kABIZZnWHjnyMnvtWumTT6pOWJZXGkeldeuQmOy9t7TbbnV77bqrlM2zum24S59/vjXxKH8tXbp1my5dpNJS6dNPpVatpB/+MLS/3X//6OJOJ+7Sa6+F0SGefFLavDnUIC5dKjVrJv3gB+F6DxgQEn9sJ9PK7FRDqQsAaPhatJAOOSS8KtuwYftk5eOPpXnzQp+V8ldZWXzHatas6kSlLolNu3ahpibVbg7dwwgilZOPlSvD+qwsqUcPadCgrX+Pgw4Kf5+yMumVV6Tx46U//Un64x+lk06SLrtMOvNMEr0dsXKlNHmyNHFi+De9xx7hel5yidSnT/jb/PWvYbKuBx+UevcOicmwYVLLhM11CtQ7akgAYAdk2tO2lC+z3aWNG7cmJxs2bJus1PW1YUPtx2zePDzFrurVoUP42apVdElLWVm4ya2cfJT35cnODje4hxwi9esXfh54YHz9dz7/XLr//vBEf+nSUFt1ySXSqFHhvFG9sjLp5ZfDtfvHP6QtW6SjjgrX79xzQ8Jc2YYN0qOPhs+8+25oMnfeeSE5OfLI1EuM60GmldmphoQEAHZApn25UWZXUlYmffPN9knK+vXSunXSihXhRrz8tWxZuEkvLd12P02abE1OKicr5a+2baVGjXYu3pIS6aOPpJkztyYes2ZtTayaNAnJRnmtxyGHhCfwOzvSWUmJNG2aNGGC9Pzz4cb4zDOlH/9YOvHEUOOCYPly6YEHQm3Ip59Ke+0lXXxxSOJ69ox/P7NmhVqTKVPCv8UePUJictFFYZ8ZKtPK7FRDQgJUUFDACF+IT6Z9uVFmJ0BpqfTFF9smKuXJSsX3mzdv+7lGjUINQ1XJSvmrffutc7p8+6304Yfb1nrMni1t2hTWN28emllVTD569pR22aV+z//TT8MT/PvvD4MUdO4sXXqpNHKk1KZN/R67oSotlV58MVyXZ58N7489NtSGnHXWziWE33wjPfZY2Pd//xuSznPOCcnJ0UdnXK1JppXZqYaEBIgpKKh6DhRmmEZVMu3LjTI7ScqHRq4uWVm6VPrss20LKincXLZpI+25Z+hPs2VLWF6x7035q2vXna9x2RmbN0tPPRVqTV57LSRC554b+kYcdVRm3CgvXSpNmhSSs6KiMBDDyJGhNqRr18Qf74MPQq3Jww+HASK6dw9Jz8UXh2aDGSDTyuxUQ0ICxDBLPOoi077cKLMbEPdwU1lVwvLll+Fmszz52G+/hn2DP3++dN99oUP2mjWhpuayy0LzonQbyrakRHruufCUa9q00OzvxBPDk7DBg7fWcNWn4mLp738PMbz9djjm2WeHGAYNatj/VnZSppXZqYaEBIjJygrf85WZxT84DzJHpn25UWajXhUXh+ZF48eHTtnNmkkXXBCSk+99L+rodk5hYagJuf/+0I+oXbswLPKPfhQSxqjMnRtqTR56KCSDXbturTVJwyZ0mVZmpxp6kwExHTvWbTkAIEFycqQRI6R33gkd74cPD6NGHXqo1L9/6Oj9TfLm8NxpW7aEZmmnnhr6ytx8s9S3b1hWVBQ6KEaZjEhh0IK77gpJ0kMPhcETfv3r0Cfp/PPDSF88jUOSkJAAMWPHhu/EinJywnIAQJIcckhoxrVsmXTvvaGT/iWXhI79V1wRnuw3VIsXS9deG55knX22NGeOdMMNoUP/tGnS979f/4MH1FWzZiEBfOONMBjCFVdI//63dMIJUrdu0q23hsEYgHpEQgLE5OWFZrW5uaGZVm4uHdoBIDItWkiXXx46ZL/1VuhnMXGidMABofN7QcHWkcOitHlz6Jdx4olhZvpbbw3NzJ59NnRA/H//L3yhpIJevaQ77gjJYEFBGNXt6qtDrcl550kvvUStCeoFfUgAYAdkWntkymw0CKtXhw7wEyZIixaFeTVGjgwdst3DzXJpadU/62PdN9+EyQtXrQq1IqNGhXjSafLHjz4KieDkyWHQhM6dpSFDQof4srKt173iz6qW1WWbmtb16yfddFOdTyPTyuxUQ0ICADsg077cKLPRoJSVSa+8EhKTp5/efsLJRMvKCkMlZ2Vt+3t2dpjT49JLQw1JlMMp17dvvw19YO67L8xrIoXmBFlZVf+sr3UDB0r33FPn8DOtzE412VEHAAAAUCdZWaGPwwknhP4NhYXbJwtVJRA7sozZ5IMmTaShQ8MLSDASEgAAkLratg0vACmLtB8AAABAZEhIUKWCgjBzeVZW+FlQEHVEAAAASEc02cJ2Cgqk0aPDxLlSaJo7enT4nSFwAQAAkEjUkGA71123NRkpV1wclgMAAACJREKC7RQV1W05AAAAsKNISLCdjh3rthwAAADYUSQkDVwUncvHjpVycrZdlpMTlgMAAACJRELSgJV3Li8slNy3di6v76QkL0/Kz5dyc8OkqLm54T0d2gEAAJBoJCQNWJSdy/PypCVLpLKy8JNkBAAAAPWBhKQBo3M5AAAA0h0JSQNG53IAAACkOxKSBozO5QAAAEh3JCQNGJ3LAQAAkO6yow4ANcvLIwEBAABA+qKGBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARCauhMTMTjGzBWa2yMyurmL9nmb2lJl9YGbvmlmfxIcKAEDtCgqkTp2krKzws6Ag6ogAADWpNSExs0aS7pV0qqReki4ws16VNrtW0ix3P1DSRZLuSnSgAADUpqBAGj1aKiyU3MPP0aNJSgCgIYunhuRQSYvcfbG7b5b0iKQhlbbpJellSXL3jyR1MrO2CY0UABCX2mq1Y9sMMrNZZvahmb2e7Bjry3XXScXF2y4rLg7LAQANUzwJSQdJn1V4vzS2rKLZks6WJDM7VFKupH0SESAAIH7x1Gqb2R6S/iJpsLv3lnRe0gOtJ0VFdVsOAIhePAmJVbHMK73/g6Q9zWyWpJ9Kel9SyXY7MhttZjPMbMaqVavqHCwAoFbx1GpfKOlJdy+SJHdfmeQY603HjnVbDgCIXjwJyVJJ+1Z4v4+kzytu4O7r3H2kux+k0IektaRPK+/I3fPdvb+792/duvVOhA0AqEY8tdrdFB4ivWZmM83soqRFV8/GjpVycrZdlpMTlgMAGqZ4EpL3JHU1s85m1ljSUEnPVNzAzPaIrZOkUZKmu/u6xIYKAIhDPLXa2ZL6STpd0smSbjCzbtvtKAVrtfPypPx8KTdXMgs/8/PDcgBAw5Rd2wbuXmJmV0h6QVIjSZPc/UMzuyy2foKknpIeMrNSSfMk/ageYwYAVK/WWu3YNqvd/RtJ35jZdEl9JS2suJG750vKl6T+/ftXTmoarLw8EhAASCW1JiSS5O7TJE2rtGxChd//I6lrYkMDAOyA72q1JS1TqNW+sNI2/5A0zsyyJTWWdJikO5MaJQAAMXElJACA1BBPrba7zzez5yV9IKlM0kR3nxtd1ACATEZCAgBpprZa7dj72yTdlsy4AACoSjyd2gEAAACgXpCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAACAyJCQAAAAAIgMCQkAAAlSUCB16iRlZYWfBQVRRwQADV921AEAAJAOCgqk0aOl4uLwvrAwvJekvLzo4gKAho4aEgAAEuC667YmI+WKi8NyAED1SEgAAEiAoqK6LQcABCQkAAAkQMeOdVsOAAhISAAASICxY6WcnG2X5eSE5QCA6sWVkJjZKWa2wMwWmdnVVaxvYWbPmtlsM/vQzEYmPlQAQDziKLMHmdlaM5sVe90YRZzpJi9Pys+XcnMls/AzP58O7QBQm1pH2TKzRpLulXSipKWS3jOzZ9x9XoXNfiJpnrufaWatJS0wswJ331wvUQMAqhRnmS1Jb7j7GUkPMM3l5ZGAAEBdxVNDcqikRe6+OJZgPCJpSKVtXNJuZmaSdpX0laSShEYKAIhHPGU2AAANRjwJSQdJn1V4vzS2rKJxknpK+lzSHEk/d/eyyjsys9FmNsPMZqxatWoHQwYA1CCeMluSBsSa2T5nZr2TExoAANuLJyGxKpZ5pfcnS5olaW9JB0kaZ2a7b/ch93x37+/u/Vu3bl3nYAEAtYqnzP6fpFx37yvpHklPV7kjHiIBAJIgnoRkqaR9K7zfR6EmpKKRkp70YJGkTyX1SEyIAIA6qLXMdvd17r4h9vs0SbuYWavKO+IhEgAgGeJJSN6T1NXMOptZY0lDJT1TaZsiScdLkpm1ldRd0uJEBgoAiEutZbaZtYv1+ZOZHarwXfBl0iMFAEBxjLLl7iVmdoWkFyQ1kjTJ3T80s8ti6ydI+p2kyWY2R6G5wG/cfXU9xg0AqEKcZfa5kn5sZiWSNkoa6u6Vm3UBAJAUtSYk0ndV+tMqLZtQ4ffPJZ2U2NAAADsijjJ7nMJgJAAARI6Z2gEAAABEhoQEAAAAQGRISAAAAABEhoQEAAAAQGRISAAAAABEhoQEAAAAQGRISAAAAABEhoQEAAAAQGRISAAAAABEhoQEAAAAQGRISAAAAABEhoQEAAAAQGRISAAAAABEhoQEAAAAQGRISAAAAABEhoQkDgUFUqdOUlZW+FlQEHVEAAAAQHrIjjqAhq6gQBo9WiouDu8LC8N7ScrLiy4uAAAAIB1QQ1KL667bmoyUKy4OywEAaCiozQeQqqghqUVRUd2WAwCQbNTmA0hl1JDUomPHui0HACDZqM0HkMpISGoxdqyUk7PtspycsBwAgIaA2nwAqYyEpBZ5eVJ+vpSbK5mFn/n5VIEDABoOavMBpDISkjjk5UlLlkhlZeEnyQgAoCGhNh9AKiMhAQAgxVGbDyCVMcoWAABpIC+PBARAaqKGBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAIM2Y2SlmtsDMFpnZ1TVs9z0zKzWzc5MZHwAAFcWVkNT25WZmvzKzWbHX3NgXXMvEhwsAqImZNZJ0r6RTJfWSdIGZ9apmu1slvZDcCAEA2FatCUk8X27ufpu7H+TuB0m6RtLr7v5VfQQMAKjRoZIWuftid98s6RFJQ6rY7qeSnpC0MpnBAQBQWTw1JPF+uZW7QNLURAQHAKizDpI+q/B+aWzZd8ysg6SzJE1IYlwAAFQpnoSk1i+3cmaWI+kUhaduAIDksyqWeaX3f5b0G3cvrXFHZqPNbIaZzVi1alXCAgQAoKLsOLaJ58ut3JmS3qquuZaZjZY0WpI6duwYV4AAgDpZKmnfCu/3kfR5pW36S3rEzCSplaTTzKzE3Z+uuJG750vKl6T+/ftXV+4DALBT4qkhiefLrdxQ1dBcy93z3b2/u/dv3bp1/FECAOL1nqSuZtbZzBorlMvPVNzA3Tu7eyd37yTpcUmXV05GAABIlngSklq/3CTJzFpIOkbSPxIbIgAgXu5eIukKhdGz5kt6zN0/NLPLzOyyaKMDAGB7tTbZcvcSMyv/cmskaVL5l1tsfXmnyLMkveju39RbtACAWrn7NEnTKi2rsgO7u49IRkwAAFQnnj4kcX25uftkSZMTFRgAAACA9MdM7QAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICNBRffy3dcIO0fn3UkQAAACQNCQnQUPz0p9LNN0uTJkUdCQAAQNKQkAANwTPPSAUF0i67SA88EHU0AAAASUNCAkTt66+lyy6TDjxQuvVWafZsadasqKMCAABIChISIGpXXimtXBlqRi6+WGrcmFoSAIhDQYHUqZOUlRV+FhREHRGAHUFCAkTpueekBx+Urr5aOuQQqWVLaciQ8K26eXPU0QFAg1VQII0eLRUWSu7h5+jRJCVAKiIhAaKydq10ySVS795hdK1yI0dKX34p/fOf0cUGAA3cdddJxcXbLisuDssBpApwwtIAACAASURBVBYSEiAqV10lLV8emmc1abJ1+YknSu3b02wLQEqIqtlUUVHdlgNouEhIgCi8+KI0caL0q19J3/vetuuys6WLLgrNuVasiCY+AIhDlM2mOnas23IADRcJCZBs69eHplo9ekg33VT1NiNHSqWl0pQpSQ0NAOoiymZTY8dKOTnbLsvJCcsBpBYSEiDZfv1r6bPPwgSITZtWvU337tKAAaHZlnty4wOAOEXZbCovT8rPl3JzJbPwMz8/LAeQWkhIgGR65RVpwoQw1O+AATVvO2KENG+e9N57SQkNAOoq6mZTeXnSkiVSWVn4STICpCYSEiBZNmyQRo2SunaVfve72rc//3ypWTM6twNosGg2BSARSEiAZLnmmvAIb9Kk7b/Bq9KihXT22dLUqdKmTfUeHgDUFc2mACQCCQmQDNOnS+PGST/9qXTkkfF/bsSIMF/J00/XW2gAsDNoNgVgZ5GQAPWtuFj64Q+l/faTfv/7un32uONCY2yabQEAgDRFQgLUt+uvlz75RLr/fql587p9NitLuvhi6aWXpKVL6yc+AACACJGQAPXprbekP/9ZuvxyadCgHdvHxReHoX8feiihoQEAADQEJCRAfdm4MTTV6thRuvXWHd/P/vtLRx/NnCQAACAtkZAA9eW3v5UWLpQmTpR23XXn9jVypLRokfT224mJDWnNzE4xswVmtsjMrq5i/RAz+8DMZpnZDDOrw0gLAAAkFgkJUB/eeUf605+kSy6RTjhh5/d37rmh/wmd21ELM2sk6V5Jp0rqJekCM+tVabOXJfV194Mk/VDSxORGCQDAVnElJLU9bYttMyj2tO1DM3s9sWECKWTTptBUa++9pdtuS8w+d91VOu886dFHpW++Scw+ka4OlbTI3Re7+2ZJj0gaUnEDd9/g/l37v+aSaAsIAIhMrQlJPE/bzGwPSX+RNNjde0s6rx5iBVLDmDHSvHnSX/8aJjdMlJEjw2zvTzyRuH0iHXWQ9FmF90tjy7ZhZmeZ2UeS/qVQS7IdMxsda9I1Y9WqVfUSLAAA8dSQ1Pq0TdKFkp509yJJcveViQ0TSBEzZ0p//GNIHk45JbH7PuqoMJfJ5MmJ3W+m2rxZevLJ0M8nvVgVy7arAXH3p9y9h6TvS/pdVTty93x37+/u/Vu3bp3gMAEACOJJSOJ52tZN0p5m9pqZzTSzi6raEU/bkNY2bw6JSNu20h13JH7/ZmHm9ldflT79NPH7zxSzZ0u/+IXUoYN0zjnp2C9nqaR9K7zfR9Ln1W3s7tMl7W9mreo7MAAAqhJPQhLP07ZsSf0knS7pZEk3mFm37T7E0zaks7FjpTlzpPvuk/bYo36OcfHFITF58MH62X+6Wr1auvtu6eCDpYMOksaPl449Vpo2TfpdlZUDqew9SV3NrLOZNZY0VNIzFTcwsy5mZrHfD5HUWNKXSY8UAACFRKI28TxtWypptbt/I+kbM5suqa+ktGsLAVRp1izp97+Xhg2Tzjij/o7TsaN0/PEhIbnxxjCTO6pWUiK98EKoAXnmGWnLFumQQ6R77pEuuEDaa6+oI6wX7l5iZldIekFSI0mT3P1DM7sstn6CpHMkXWRmWyRtlHR+hU7uAAAkVTwJyXdP2yQtU3jadmGlbf4haZyZZSs8aTtM0p2JDBRosLZsCU219tpLuuuu+j/eiBEh8Xn99fCUH9uaPz8kIQ8/LK1YIbVuLV1xRbhuBx4YdXRJ4e7TJE2rtGxChd9vlbQTs3UCAJA4tSYk8Txtc/f5Zva8pA8klUma6O5z6zNwoMH4wx9CDclTT0ktW9b/8c46S9p993DTTUISrFkThkR+4IEwB0yjRtLpp4dE8bTTpMaNo44QAABUI54aklqftsXe3yYpQZMuAClizpzQB2HoUOn730/OMXNywvEeflgaNy4kJ5motFR65ZWQhDz1VJj/pU+fMCFlXl4YXAAAADR4NEAHdlRJSXgCv8ceoV9CMo0YIW3cKP3978k9bkOwaJF0ww1S587SSSdJzz0XJqJ87z3pgw+k//s/khEAAFJIXDUkAKpw++1h3pHHHpNaJXnE1MMPl7p3D7UDP/pRco8dhQ0bQvL1wAPSG2+EkcZOOkm67TZpyBCpadOoIwQAADuIGhJgR8ybJ/32t9K550rnnZf845uF2pm33krHif0C99Bxf+RIqV27UAuyYkUYzayoSHr+een880lGAABIcSQkQF2Vloab4912k+69N7o4hg8Pw/6m25wkRUWhX06XLtKgQdLjj4c+M2++KS1YIF1zjbTPPlFHCQAAEoSEBKirO+8MIzndc4/Upk10cey9t3TyySEhKS2NLo5E2LhRKiiQTjxR6tQpzLGSmxvObcUKaeJEaeDAUDMEAADSCgkJUBcLFoQO1UOGhKf2URs5Ulq2TPr3v6OOZMf96U+hSdawYdLHH4dkZPHiMILWRRdJzZtHHSEAAKhHJCRAvMqbajVrJo0f3zCe1g8eLO25pzR5ctSR7Jg33pCuuip00n/55ZCI3HRTGEELAABkBEbZAuJ1zz3S229LDz0ktW8fdTRBkybShReGJk1ffx2Sk1RRXBwSvM6dpSefpCYEAIAMRQ0JEI9Fi6Rrrw2zfw8bFnU02xo5Uvr2W+mRR6KOpG5uuCFc1/vvJxkBACCDkZAAtSkrC3N9NG4s3Xdfw2iqVdEhh0gHHJBazbbefjsMDnDZZdKxx0YdDQAAiBAJCVCb8eOl6dOlO+6QOnSIOprtmYWZ2999N8yP0tBt3Biaau27r/THP0YdDQAAiBgJCVCTTz+VfvObMLzuyJFRR1O9YcOk7Owwk3lDd9NNYbSyiRPDXC4AACCjkZAA1XGXRo0Kkw/m5ze8ploVtWkT+rc8/LBUUhJ1NNV7913p9tvDdT3xxKijAQAADQAJCVCd/PwwF8btt0sdO0YdTe1GjpS++EJ6/vmoI6nat9+GGPfeO1xTAAAAkZAAVSsqkn71K+n446VLLok6mvicdprUunXDbbY1Zkzo45KfL7VoEXU0AJCSCgqkTp1C5X2nTuE9kOpISIDK3EMSUlYW+jk05KZaFe2yS+hL8uyz0urVUUezrZkzpVtvDZ3vTz016mgAICUVFEijR0uFheGrqrAwvCcpQapjYsRUsHZtGJkoCrvuGl6ZZNIk6cUXpXHjwuOnVDJiRBhO929/k372s6ijCTZvDk212rQJI5UBAHbIddeFOWUrKi4Oy/PyookJSAQSkoastDSMSDR2bHgUEoXsbOmMM8IN5amnhqfw6aikJPS9eOCBUMNwzDHSj38cdVR1d+CBYV6SBx5oOAnJ738vzZkjPfNMas0kDwANTFFR3ZYDqYKEpKFavly68ELptdek4cOlI46IJo6PP5amTJGefjo84R4+PDyF79MnmngSbd68cPP+8MOhQ3ibNtJPfxqG+s1K0RaNI0eGc5g1SzrooGhjmT07JNR5edKZZ0YbCwCkuI4dQzOtqpYjcWbOnNkmOzt7oqQ+ontDIpRJmltSUjKqX79+K6vagISkIXrllZCMrFsXZt+++OJo4/nDH6Tnngs37nfdJf3pT1L//uHG94ILUu+p95o10tSp4dq+++7WWqARI0LH8FSvBbrgAumXvwzn9+c/RxfHli3hmu61V/h3AwBppKAgNJUqKgoJQfmzl/o0dmzoM1Kx2VZOTliOxMnOzp7Yrl27nq1bt/46KysroiYq6aOsrMxWrVrVa8WKFRMlDa5qG7K+hqS0VPrd78L8DC1bSu+9F30yIoUb9MGDpaeekj7/PPRR2LxZ+slPpPbtpaFDQ3On0tKoI61eaWnoF3LBBVK7dtLll4d+OXfcIS1bFs5tyJDUT0akkAAMHhy+LTdvji6OW28NtTTjx4eYACBNRNW5PC8vDFSYmxvGW8nNDe/pP5JwfVq3br2OZCQxsrKyvHXr1msVapyq3iaJ8aAmK1eGPho33hhqR959V+rdO+qotte6tfSLX4QbzZkzw2hUL70UYs/Nla69Vlq4MOoot1q0SLr++tA5/eSTpRdeCJPyzZgRmhNdeWVoppVuRo4MI23985/RHH/u3DDM7/nnS2edFU0MAFBPaupcXt/y8qQlS8JAkEuWkIzUkyySkcSKXc9q8w4SkobgjTekgw+Wpk+X/vpX6aGHGv7IVmah8/Q994Rak7//XerbNzwV795dOvLIMGTuunXJj239+jBS1lFHSV27SrfcIh1wgPTYYyHWceOkfv1SZzjfHXHSSaH2avLk5B+7pCQkRHvsEf59AECaoXM5kFgkJFEqKws38MceGxqB/ve/4el9qt0oN2kinXuu9K9/SZ99FvqcrF4dak/at5cuukh69dVwvvWlrEx6/fXQZ6F9e+lHPwq1TrfcEr4hpk2TzjtPatq0/mJoSLKzwwAE06ZJK1Yk99h/+lOogRo3LtSoAUCaqa4TOZ3LkQgrVqxo1KNHj149evTo1apVq75t2rQ5sPz9pk2b4rpJPPfcczvNnj27SU3b3HLLLa3Hjx/fMjFR7xzziIaT7d+/v8+YMSOSYzcIX34Z+of861/hRnniRGn33aOOKnHcpXfeCR3hH3kk1JR06hTOecSIxM3vUVgoPfhgeC1eLO22W+jTMnKkdPjhqZfcJdJHH0k9e0q33SZddVVyjjl/fhjZ64wzpMcfT+vrb2Yz3b1/1HEkS8aX2UAF5X1IKncupz9Hw1WXMnv27NlL+vbtG/cMwxMmqOWYMeqwYoUat2unzTfeqGWXXaavdjzarf7v//5v71133bV0zJgxX1RcXlZWJndXo0aNEnGYpJg9e3arvn37dqpqHTUkUfjvf0Nzp5deCk+RH300vZIRKdyIHn64dN99YQjjKVOkLl1Cv4LOnaXjjgtD7VZuhBuP4uLwbXDCCWFfv/1tSHAefjjUBuTnSwMGpPXNcFx69Ah/gwceSM48NqWl0g9/GJob/uUvXH8AaYvO5Sg3YYJaXnmlcpcvV2N3aflyNb7ySuVOmKCE1zzMnTu3SdeuXXtfeOGFHXv37t2rqKholwsuuCC3T58+Pbt06dL7qquual++bb9+/bq//fbbzbZs2aLddtvtoMsvv7xD9+7dex100EE9li1bli1JP/vZz/YeM2ZMm/LtL7/88g4HHHBAz06dOvV56aWXmkvSunXrsk4++eT9u3fv3uvMM8/s3KdPn55vv/12s0SfGwlJMrmHYViPOirMcfHWW2GkqnS/ccvJCaX0Sy9Jn34akpLCwtCUq1270LTr7bdrvml2l/7zn/BIqn17adiwUCNy002hV9/LL4dlOTnJOqvUMHJkmGslGU+2//znkGzfc4/Utm39Hw8AIkTnckjSmDHqsGnTtvfTmzYpa8wYdaiP433yySdNL7300tXz58+f17lz5y1//vOfl86dO3f+/PnzP3z11Vd3nzlz5nZt0zds2NBo0KBB6xcsWDCvf//+G+69995WVe3b3TVnzpz5Y8eO/WzMmDF7S9If/vCHNm3atNmyYMGCeddee+2K+fPn18uNFglJsqxZI51zThjV6fTTpf/9L8zlkWlyc6UbbggTLr72WrgmU6dKAweGJ/q33BKG4S33+eehn02vXmFyyIIC6fvfD31SFi0Ko5Ll5kZ2Og3e+eeHfjMPPFC/x1m4MIxmNnhwGFoZAIAMsGKFGtdl+c7ad999vz3mmGO+a14yadKklr169erZu3fvXosXL276wQcfbFd70bRp07If/OAH6ySpX79+xUuWLKkytvPOO2+NJB1xxBHFS5cubSxJ//nPf3bNy8v7SpIGDBiwcf/9999YH+eVWgnJY4+F/ggb6+Va1J+ZM0MTrWefDfNePPVU6k0mmGhZWdIxx4Qb5eXLw6hYbduGYYM7dgzDCJ9+urTvvtLVV0utWoV+NitWhP4igwal7kzqydSihXT22SHp27Spfo5R3lSraVNpwoT0r/EDACCmXTtVOeFXdct3VrNmzb4bIWjOnDlN7rvvvrbTp09fuHDhwnlHH330uo0bN273JZydnf1dE5RGjRp5aWlplV/UTZs2Lau8TbL6mqfWHV1+fnj6uvfe0o9/HObqiKhTflzcQ1v6I44IQ6FOnx5qSLhh29Zuu4WmRdOnh5qTa68NzYw++CAkIwsXhqGRf/SjsC3qZuTIUEP3j3/Uz/7HjQvND++6KzSnAwAgQ9x4o5Y1bapthhFt2lRlN96oZdV9JlHWrFnTqHnz5qV77rlnaWFh4S7Tp09PeIfkAQMGbJg6deqekvTuu+82W7x4ccL7j0iplpC8+KL073+HJ+eTJ0uHHSb16SPdfnvyhzatzfr1IXn6yU9C5+v33w8drVGzLl3CbPWFhWEI4bFjw1wi2HHHHhtqmuqj2dYnn0jXXCOddloYZhgAgAxy2WX66s47Vdi+vTabSe3ba/Odd6owUaNs1WTgwIHFXbt23dStW7feI0aMyO3Xr9+GRB/j6quvXrl8+fLG3bp163Xrrbe27dKly8aWLVuWJvo4qTvs79q1oQnXAw+Ezs6NGoVmPiNHhiFHG9dL0734fPBBGMp30aJwQ/3rX9O8CNG64Ybwb7GoSNpnn8Tss6wsjJb2/vvShx8mbr8pgmF/ASB11Oewv+lsy5Yt2rJli+Xk5PicOXOanHLKKd2WLFkyZ5dddqnzvtJz2N8WLbaOzjR/fphnYebM0Em6QwfpF7+QZs9Obkzu0v33h5qb9etDx+urryYZQfRGjAj/Ph9+OHH7nDAhTEZ5xx0Zl4wAAJAJ1q5d2+h73/tej+7du/c655xz9r/nnnsKdyQZqU163Cn36BFmBy8qChMNHnusNH58mKDt4IOlu+8OM4fXp2++CZP+jRolHXmkNGuWdPTR9XtMIF777x+Gm07UnCRLloSav5NOCh3aAQBA2mnVqlXphx9+OH/BggXzFi5cOO/ss89eVx/HSY+EpFx2dmjL/thjYbjYe+4JtRM//3noCH/uuSFhKSlJ7HHnzZMOPTRM/nfTTdLzz0tt2iT2GMDOGjkyDBrw9ts7tx/3kHhnZUl//SuDNDRAZnaKmS0ws0VmdnUV6/PM7IPY620z6xtFnABSS0FBmIc4Kyv8LCiIOiKki/RKSCraay/piitCM67Zs8Pv06eH/iX77hue7s6fv/PHmTJF+t73Qg3MSy+FWcMbNdr5/QKJdt55UvPmYUCInfHXv4aJKG+7LQzRjAbFzBpJulfSqZJ6SbrAzHpV2uxTSce4+4GSficpP7lRAkg1BQVhbuLCwvBcqrAwvCcpQSKkb0JS0YEHhnbuS5eGOUAOPTS879VLOvxw6b77Qif5uti4MfRhGT48JCTvvy8df3z9xA8kwq67hqTk0UdDE8MdUVQU+msdd1z4JkJDdKikRe6+2N03S3pE0pCKG7j72+7+deztfyXRCQhAja67Tiou3nZZcXFYDuysuBKSOKr/B5nZWjObFXvdmPhQE6Bx4zDL9z/+EWYDv/12acMG6bLLpHbtpLy8UMtRVlbzfhYuDInMxIlhzox//zs0CQMauhEjwoALTz5Z98+6hySkrCz826epVkPVQdJnFd4vjS2rzo8kPVfVCjMbbWYzzGzGqlWrEhgigFRTVFS35UBd1JqQxFn9L0lvuPtBsdeYBMeZeG3bSr/8pTRnjvTee6Fj7rRpoZNup05hmNRPPtn+c489JvXvHxKaadPCUKrZ2UkPH9ghRx8t7bffjs1JMnmy9MILYQCJzp0THhoSpqpMscqRDMzsWIWE5DdVrXf3fHfv7+79W7duncAQAaSa6lro0nI38Q499NDuTzzxxDaTHI4ZM6bNsGHDqr3aOTk5B0vSkiVLdjnllFP2q26/06dPz6np2GPGjGmzfv367/KDY445psvq1avrvS9CPDUktVb/pzSzkGDce6+0fLn0yCOhKdfYsWGSvmOOCTdvX34Z+qGcf36YjPH998O8J0AqMQu1JK++GkbKiteyZdKVV4aE5vLL6ys6JMZSSftWeL+PpM8rb2RmB0qaKGmIu3+ZpNgApKixY6WcSreyOTlhORLrvPPO+3Lq1KktKy574oknWg4bNqzWyRY7deq05fnnn1+8o8e+77772m7YsOG7/OD1119f1KpVq4RPhFhZPAlJvNX/A8xstpk9Z2a9ExJdsjVtGhKO558PdZBjx4Yk5Yc/lFq3DknLL38Z5l7Yd9/a9wc0RBddFBKTBx+Mb3t36dJLpc2bwzw7zKvT0L0nqauZdTazxpKGSnqm4gZm1lHSk5KGu/vCCGIEkGLy8qT8fCk3N3yF5OaG93l5UUeWfoYPH/71yy+/3GLjxo0mSQsWLGi8cuXKXQ477LDiAQMGdOvVq1fPbt269ZoyZcoelT+7YMGCxl27du0tSRs2bLAzzjhjv27duvU6/fTT99u0adN3Neh5eXkd+/Tp07NLly69r7zyyr0l6eabb26zcuXKXY455phuhx12WDdJ6tChwwHLly/PlqSbbrqpbdeuXXt37dq195gxY9qUH2+//fbrPXTo0NwuXbr0HjhwYNcNGzbUuU13PG2N4qn+/5+kXHffYGanSXpaUtftdmQ2WtJoSerY0Ov49tkn9A+55powTOpTT4X5TU4/PerIgJ2Tmxs6pU+eHJom1pZgTJkShsu+885Qa4gGzd1LzOwKSS9IaiRpkrt/aGaXxdZPkHSjpL0k/cVCX6CSTJp1HsCOycvLwATkhz/cV3Pn1tjMqc769CnWpEmfVbe6Xbt2pX379v3miSeeaDFs2LA1Dz74YMvBgwd/veuuu5b961//WtSyZcuy5cuXZx922GE9LrzwwjVZ1XyP33777W2aNWtWtnDhwnnvvPNOs4EDB37X5eKOO+5Y1rZt29KSkhIdccQR3d95551m119//crx48e3ff311xe2b99+mzky3njjjZy//e1ve82cOXO+u6tfv349jz/++PWtWrUqLSoqajplypTFRxxxROFpp52230MPPbTn5ZdfXmttTkXxPOqstfrf3de5+4bY79Mk7WJmrSrvKCXbI5tJAweGDvAkI0gXI0eGJlvTp9e83fLlYR6fI46QfvrTpISGnefu09y9m7vv7+5jY8smxJIRufsod9+zQr8/khEAaEB+8IMffPXoo4/uKUlPPvlky+HDh39VVlZmv/jFL/bp1q1br2OPPbbbypUrGy9durTayoU333xz1+HDh38pSYcddtjGbt26fTdO2oMPPtiyV69ePXv16tXr448/bjp79uymNcXz2muv7Xraaaet2X333ctatGhRdvrpp3/96quv7iZJHTp0+PaII47YKEkHH3xw8ZIlS5rU9XzjqSH5rvpf0jKF6v8LK25gZu0kfeHubmaHKiQ6tEkGGqqzzpJ23z30jxo0qOpt3KUf/zgMcT1pEvPrAAAyTw01GfUpLy9vzfXXX7/vm2++mbNp06asI488svjuu+/e68svv8yeM2fO/CZNmniHDh0O2LhxY42VC1bFiJgfffRR43HjxrWdOXPm/NatW5eec845nTZt2lTjftyrHBtFktS4cePvVjZq1Mhri6kqtX7A3UsklVf/z5f0WHn1f3kTAEnnSpprZrMl3S1pqNcUOYBo5eSE/lKPPx6GAa7KI4+EIbJ/9zupe/fkxgcAQAZr0aJF2eGHH75+1KhRnc4+++yvJGnt2rWNWrVqtaVJkyb+7LPP7vb55583rmkfRx555IYpU6a0lKT33nuv6cKFC3Mk6euvv27UrFmzspYtW5Z+9tln2a+99lqL8s80b968dO3atdvlB8cdd9yGadOm7bF+/fqsdevWZU2bNm3PY489tpobiLqLa7zaWDOsaZWWTajw+zhJ4xIVFIAkGDkyzLr+97+HgRsq+uKL0ETrsMPC6FoAACCphg4d+tXFF1+8/9SpUxdL0qhRo7469dRTu/Tp06dn7969izt37rypps9fddVVK4cOHdq5W7duvXr37l18wAEHfCNJAwYM2NinT5/irl279u7YseO3/fr121D+mYsvvnj1qaee2rVNmzZb3nnnne8GPTnyyCOLL7zwwi8POeSQnpI0fPjwVQMHDty4YMGCGpOieFlUFRn9+/f3GTNmRHJsAApNsnr2DCPIvfHGtuvOO0965pkwvHWvqqYdgpnNzKS+F5TZAFJZXcrs2bNnL+nbt+/q+o4p08yePbtV3759O1W1jvE7gUxVPifJm29KH3+8dfnjj4fXTTeRjAAAgHpHQgJksosuCsP+Tp4c3q9eHSY+7NdP+tWvIg0NAABkBhISIJPtvbd08snSQw9JpaWh38iaNSFByY6rixkAAOmmrKysrM6T+6F6setZVt16EhIg040YIS1dGuYbeeSRMFlinz5RRwUAQFTmrlq1qgVJSWKUlZXZqlWrWkiaW902PAIFMt3gwdKee0r33isddJB09dVRRwQAQGRKSkpGrVixYuKKFSv6iIf3iVAmaW5JScmo6jZImYSkoEC67jqpqEjq2FEaO1bKy4s6KiANNG0qDR8u/eUvYaLEXXaJOiIAALaRzPvAfv36rZQ0uH72jqqkRNZXUCCNHi0VFoaRSgsLw/uCgqgjA9LE738vzZkTakgAAGhAuA9MfymRkFx3nVRcvO2y4uKwHEACNG8u9egRdRQAAGyH+8D0lxIJSVFR3ZYDAAAgPXAfmP5SIiHp2LFuywEAAJAeuA9MfymRkIwdK+XkbLssJycsBwAAQPriPjD9pURCkpcn5edLubmSWfiZn88oWwAAAOmO+8D0lzLD/ubl8Q8PAAAgE3EfmN5SooYEAAAAQHoiIQEAAAAQGRISAAAAAJEhIQEAAAAQGRISAAAAAJEhIQEAAAAQGRISAAAAAJEhIQEAAAAQGXP3aA5stkpSYSQH33GtJK2OOogky8RzljLzvDPxnKUdP+9cd2+d6GAaKsrslJKJ5805Zw7K7DQUWUKSisxshrv3jzqOZMrEc5Yy87wzlB78RQAAA8NJREFU8ZylzD3vTJCpf9tMPG/OOXNk6nmnO5psAQAAAIgMCQkAAACAyJCQ1E1+1AFEIBPPWcrM887Ec5Yy97wzQab+bTPxvDnnzJGp553W6EMCAAAAIDLUkAAAAACIDAlJLcxsXzN71czmm9mHZvbzqGNKJjNrZGbvm9k/o44lGcxsDzN73Mw+iv3NB0QdUzKY2ZWxf99zzWyqmTWNOqZEM7NJZrbSzOZWWNbSzF4ys49jP/eMMkYkRiaX25lWZkuZWW5nQpktUW5nEhKS2pVI+qW795R0uKSfmFmviGNKpp9Lmh91EEl0l6Tn3b2HpL7KgHM3sw6Sfiapv7v3kdRI0tBoo6oXkyWdUmnZ1ZJedveukl6OvUfqy+RyO9PKbCnDyu0MKrMlyu2MQUJSC3df7u7/i/2+XqGg6xBtVMlhZvtIOl3SxKhjSQYz213S0ZLulyR33+zua6KNKmmyJTUzs2xJOZI+jziehHP36ZK+qrR4iKQHY78/KOn7SQ0K9SJTy+1MK7OljC63077Mlii3MwkJSR2YWSdJB0t6J9pIkubPkn4tqSzqQJJkP0mrJD0Qa/Iw0cyaRx1UfXP3ZZJul1Qkabmkte7+YrRRJU1bd18uhZtYSW0ijgcJlmHldqaV2VIGltsZXmZLlNtpiYQkTma2q6QnJP3C3ddFHU99M7MzJK1095lRx5JE2ZIOkTTe3Q+W9I0yoCo41v52iKTOkvaW1NzMhkUbFbDzMqncztAyW8rAcpsyG+mIhCQOZraLwpdagbs/GXU8STJQ0mAzWyLpEUnHmdmUaEOqd0slLXX38iepjyt80aW7EyR96u6r3H2LpCclHRFxTMnyhZm1l6TYz5URx4MEycByOxPLbCkzy+1MLrMlyu20REJSCzMzhbap8939jqjjSRZ3v8bd93H3Tgqd5V5x97R+AuPuKyR9ZmbdY4uOlzQvwpCSpUjS4Wb2/9u7Y5QIgiAKoL9ADIy9jYngGQxlEWMvYGLqRUQQEw28g4kK5kbeow12Ag+wS+H0e9HQUQXDbz70TB8t7/tZVv5R6B8vSTbL8ybJc+Ms7MiMuT1jZifT5vbMmZ3I7VU66B7gHzhJcpHkq6o+l7WbMcZr40zsz3WS+6o6TPKd5LJ5nr0bY7xV1VOS92z/TvSRFd6EW1UPSU6THFfVT5LbJHdJHqvqKttN/rxvQnZIbs9lqtyeJbMTuT0TN7UDAABtHNkCAADaKCQAAEAbhQQAAGijkAAAAG0UEgAAoI1CAgAAtFFIAACANgoJAADQ5heyY0S0K36ongAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = models.Sequential()\n", "model.add(layers.Embedding(10000, 32))\n", "model.add(layers.SimpleRNN(32))\n", "model.add(layers.Dense(1, activation='sigmoid'))\n", "\n", "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n", "history = model.fit(x_train, y_train, epochs=10, batch_size=128, validation_split=0.2, verbose=0)\n", "\n", "acc = history.history['acc']\n", "val_acc = history.history['val_acc']\n", "loss = history.history['loss']\n", "val_loss = history.history['val_loss']\n", "epochs = range(1, len(acc) + 1)\n", "\n", "plt.figure(figsize=(12, 5))\n", "plt.subplot(1, 2, 1)\n", "plt.plot(epochs, acc, 'bo', label='Training')\n", "plt.plot(epochs, val_acc, 'r', label='Validation')\n", "plt.title('Training and validation accuracy')\n", "\n", "plt.subplot(1, 2, 2)\n", "plt.plot(epochs, loss, 'bo', label='Training')\n", "plt.plot(epochs, val_loss, 'r', label='Validation')\n", "plt.title('Training and validation loss')\n", "plt.legend(bbox_to_anchor=(1.02, 0.2), loc=2, borderaxespad=0.5)\n", "plt.show()\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The validation accuracy is about 60%." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### LSTM" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Although `SimpleRNN` should theoretically be able to retain at time t information about inputs seen many timesteps before, in\n", "practice, such long-term dependencies are impossible to learn. This is due to the vanishing gradient problem, an effect that is similar to what is observed with non-recurrent networks (feedforward networks) that are many layers deep: as you keep adding layers to a network, the network eventually becomes untrainable.\n", "\n", "For example, consider a language model trying to predict the next word based on the previous ones. If we are trying to predict the last word in \"the clouds are in the sky,\" it's pretty obvious the next word is going to be sky. In such cases, where the gap between the relevant information and the place that its needed is small, RNNs can learn to use the past information.\n", "\n", "But there are also cases where we need more context. Consider trying to predict the last word in the text “I grew up in France… I speak fluent French.” Recent information suggests that the next word is probably the name of a language, but if we want to narrow down which language, we need the context of France, from further back. It’s entirely possible for the gap between the relevant information and the point where it is needed to become very large.\n", "\n", "Long Short-Term Memory (LSTM) saves information for later, thus preventing older signals from gradually vanishing during processing.\n", "\n", "**Pseudocode of the LSTM architecture**\n", "```python\n", "output_t = activation(dot(state_t, Uo) + dot(input_t, Wo) + dot(c_t, Vo) + bo)\n", "i_t = activation(dot(state_t, Ui) + dot(input_t, Wi) + bi)\n", "f_t = activation(dot(state_t, Uf) + dot(input_t, Wf) + bf)\n", "k_t = activation(dot(state_t, Uk) + dot(input_t, Wk) + bk)\n", "```\n", "We obtain the new carry state (the next c_t) as follows\n", "```python\n", "c_t+1 = i_t * k_t + c_t * f_t\n", "```\n", "The multiplying `c_t` and `f_t` is a way to deliberately forget irrelevant information in the carry dataflow. Meanwhile, `i_t` and `k_t` provide information about the present, updating the carry track with new information." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyQAAAE/CAYAAAC3hAd1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXhU5fn/8fdNwmJEZREEQZYiqKAFJcW6a60VV7RVq/J1R0TFWpf+VKxaqah1qVsVREvFNmqtS0VFrVUrblWDRQQRRQoY2QLKIqBs9++P56QMwySZJJOcyczndV1zzZznbPcZwknu82zm7oiIiIiIiMShSdwBiIiIiIhI/lJCIiIiIiIisVFCIiIiIiIisVFCIiIiIiIisVFCIiIiIiIisVFCIiIiIiIisVFCkuXMrMDMvjGzLpncNk5mtrOZZXy8aTP7sZnNSVieaWYHpLNtLc71oJmNqO3+IiLJdL+v0XEb/f3ezG4ws4cyfVyRxqgw7gByjZl9k7BYBHwHbIiWz3P3kpocz903AC0zvW0+cPddMnEcMxsC/J+7H5xw7CGZOLaINF6632cP3e9FGjclJBnm7v/7BRE9kRni7v+sbHszK3T39Q0Rm0h19PMokj7d70VEMkNNthpYVEX7VzN71MxWAv9nZvuY2b/NbJmZLTCzu82sabR9oZm5mXWLlv8SrX/BzFaa2Ttm1r2m20brjzCzT81suZndY2ZvmdmZlcSdToznmdksM/vazO5O2LfAzO4ws6Vm9jkwsIrv59dm9lhS2b1m9vvo8xAzmxFdz+fR06zKjlVmZgdHn4vM7M9RbNOB/inOOzs67nQzOzYq3wP4A3BA1DxiScJ3+5uE/YdF177UzP5uZh3T+W5q8j1XxGNm/zSzr8xsoZn9v4TzXBN9JyvMrNTMdkzVXMLM3qz4d46+z0nReb4Cfm1mPc3stehalkTf23YJ+3eNrrE8Wn+XmbWIYt4tYbuOZrbazNpWdr0iuUz3e93vq7rfp7iG46J4lpnZq2a2S8K6EWY2P7q/f5JwrT80sw+i8kVmdmu65xPJKu6uVz29gDnAj5PKbgDWAscQEsKtgB8AexNqrL4HfAoMj7YvBBzoFi3/BVgCFANNgb8Cf6nFtu2BlcCgaN2lwDrgzEquJZ0YnwG2A7oBX1VcOzAcmA50BtoCk8KPXsrzfA/4Btg64diLgeJo+ZhoGwN+BKwBvh+t+zEwJ+FYZcDB0efbgH8BrYGuwMdJ254EdIz+TU6NYtghWjcE+FdSnH8BfhN9/kkUYz+gBXAf8Go6300Nv+ftgEXAxUBzYFtgQLTuKuBDoGd0Df2ANsDOyd818GbFv3N0beuB84ECws9jL+BQoFn0c/IWcFvC9UyLvs+to+33i9aNBUYlnOcy4Om4/x/qpVdDvND9Xvf7mt/vbwAeij7vFsXxo+jfaET0vTcF+gBzgQ7Rtt2B70Wf3wdOiT5vA+wd9/8FvfSqzUs1JPF4092fdfeN7r7G3d9393fdfb27zyb8YXdQFfs/4e6l7r4OKCHcGGu67dHAFHd/Jlp3B+GXWUppxniTuy939zmEXwYV5zoJuMPdy9x9KXBzFeeZTfiDd1BUdBiwzN1Lo/XPuvtsD14FXgFSdmRMchJwg7t/7e5zCU/BEs/7uLsviP5NHiH8cVGcxnEBBgMPuvsUd/8WuBI4yMw6J2xT2XezmWq+52OBL9z9Lnf/zt1XuPt70bohwAh3/yy6hinu/lWa8c9z99HuviH6efzU3V9x97Xuvpjws1ERwz7A9sAV7r4q2v6taN144FQzs2j5NODPacYgkqt0v6/8PHl9v09yMjDB3V+N/o1uJjx02pvw0KgF0MdCs7//Rt8dhMSyp5m1dfeV7v5umtchklWUkMTji8QFM9vVzJ630ARnBTCS8EdfZRYmfF5N1R0bK9t2x8Q43N0JT5hSSjPGtM5FeNJTlUeAU6LPpxJ+sVbEcbSZvWuhydIywtOqqr6rCh2risHMzjSzD6Oq8mXArmkeF8L1/e947r4C+BrolLBNWv9m1XzPOwGzKolhJ+DzNONNlvzz2MHMHjezL6MYHkqKYY6HDrWbiRKT9cD+ZrY70AV4vpYxieQK3e+rlrf3+2qOu5Hwb9TJ3WcSapxHAostNAHsEG16FtAbmGlm75nZkWleh0hWUUISj+QhEO8nPCXa2d23Ba4lVFHXpwWEKnUAoqfanSrfvE4xLiD8IVuhumEq/wr8OHriNIjwCwsz2wp4AriJUL3eCvhHmnEsrCwGM/seMJrQbKltdNxPEo5b3ZCV8wnNAiqOtw2hqcCXacSVrKrv+QugRyX7VbZuVRRTUUJZh6Rtkq/vd4TRgvaIYjgzKYauZlZQSRwPA/9HqB153N2/q2Q7kXyh+33V8vl+X9VxmxD+zb4EcPe/uPt+hOZaBYTvBXef6e4nE5rl3Q48aWYt6hiLSINTQpIdtgGWA6ssdAo+rwHO+Rywl5kdY2aFhH4J7eopxseBX5pZJwsdnK+oamN3X0To5/AnYKa7fxatak7o11AObDCzowl9HdKNYYSZtbIwbv/whHUtCb+Eygm/q4cQnphVWAR0toTO5UkeBc4xs++bWXPCL4o33L3SJ5BVqOp7ngB0MbPhZtbMzLY1swHRugeBG8yshwX9zKwN4RfzQkJn2gIzG0rCL70qYlgFLDeznYDLE9a9AywFbrTQcXQrM9svYf2fgRMITzofrsX1i+Q63e8T5Pn9PjnmY83s4OjcvyL0+3nXzHYzs0Oi862JXhsIF3CamW0f1agsj65tYx1jEWlwSkiyw2XAGYSbz/2EJ0b1Kvol8HPg94Q/MHsA/yE8Gc90jKMJbX8/InTAeyKNfR4hdFp8JCHmZcAlwNOEjoInEH7RpuM6wpO7OcALJPyx7O5TgbuB96JtdgUS2+G+DHwGLDKzxKr4iv1fJFSlPx3t34XQzrg2Kv2e3X05oY31zwidKj9lU7vuW4G/E77nFYQ23y2iphnnEjpILiF0cq+ujfF1wADCL7cJwJMJMawntEffjVBbMo/w71Cxfg7h33mtu79dw2sXyQe6328pX+/3icedTvjORxOSpYHAsVF/kubALYR7+EJCjcyvo12PBGZYGMXtNuDn7r62rvGINDQLf69Ivoua4MwHTnD3N+KORxovM3sYmO3uv4k7FhHZku73IpJtVEOSx8xsoJltF1UDX0PokPxeNbuJVCpqnz0IGBd3LCKyie73IpLNlJDkt/2B2YRq4IHAceqELLVlZjcR5kK50d3nxR2PiGxG93sRyVpqsiUiIiIiIrFRDYmIiIiIiMRGCYmIiIiIiMSmsLoNzGwcYZjPxe6+e4r1BtxFGHpuNXCmu39Q3XG3335779atW40DFhHJBpMnT17i7lXN5ZBTdM8WkcYs3+7ZjU21CQnwEPAHKp/k7AigZ/TamzCG9t7VHbRbt26UlpamF6WISJYxs7lxx9CQdM8WkcYs3+7ZjU21TbbcfRJhUqLKDAIe9uDfQCsz65ipAEVEREREJHdlog9JJ8KMzRXKorItmNlQMys1s9Ly8vIMnFpERERERBqzTCQklqIs5VjC7j7W3YvdvbhdOzXjExERERHJd5lISMqAnRKWOwPzM3BcERERERHJcZlISCYAp1vwQ2C5uy/IwHFFRERERCTHpTPs76PAwcD2ZlYGXAc0BXD3McBEwpC/swjD/p5VX8GKiIiIiEhuqTYhcfdTqlnvwIUZi0hERERERPKGZmoXyXMlJdCtGzRpEt5LSuKOSBKZ2UAzm2lms8zsyhTrf2VmU6LXNDPbYGZtqtrXzNqY2ctm9ln03rohr0lERCSREhKRPFZSAkOHwty54B7ehw5VUpItzKwAuJcwAW1v4BQz6524jbvf6u793L0fcBXwurt/Vc2+VwKvuHtP4JVoWUREJBZKSETy2NVXw+rVm5etXh3KJSsMAGa5+2x3Xws8RpiMtjKnAI+mse8gYHz0eTxwXMYjF5HctHgxvPVW3FFIjlFCIpLH5s2rWXmmqblYtWoy8WwRMBB4Mo19d6gYDTF6b5/BmEUkVy1aBPvtBwccAG+8EXc0kkOUkIjksS5dalaeSWoulpa0J54FjgHecvevarFv6pObDTWzUjMrLS8vr8muIpJrli2Dww+H+fOhUyc46yxYtSruqCRHKCERyWOjRkFR0eZlRUWhvL6puVhaajLx7Mlsaq5V3b6LzKwjQPS+ONUB3X2suxe7e3G7du1qEb6I5ITVq+Hoo+Hjj+Hpp8OTo9mz4Yor4o5McoQSEpE8NngwjB0LXbuCWXgfOzaU17e4m4s1Eu8DPc2su5k1IyQdE5I3MrPtgIOAZ9LcdwJwRvT5jKT9REQ2WbsWfvYzeOcdeOQR+MlP4MAD4eKL4d574ZVX4o5QcoASEpE8N3gwzJkDGzeG94ZIRiDe5mKNhbuvB4YDLwEzgMfdfbqZDTOzYQmbHg/8w91XVbdvtPpm4DAz+ww4LFoWEdnchg1w+unw4otw//1wwgmb1o0aBT17wtlnw4oV8cUoOaHaiRFFROrDqFGhz0his62Gai7WmLj7RGBiUtmYpOWHgIfS2TcqXwocmsk4RSTHuMMFF8Bf/wq33AJDhmy+vqgIxo+H/feHyy6DBx6IJ07JCaohEZFYxNlcTOqfRlATaeRGjAg35auugl/9KvU2++wDl18ODz4IL7zQsPFJTjH3Gg26kjHFxcVeWloay7lFROrKzCa7e3HccTSUmtyzK0ZQS679UsIp0kjcckvosH7eeTB6dHhqVJlvv4X+/cMoXNOmQevWDRdnDeTbPbuxUQ2JiIhklEZQE2nEHnggJCM//3notF5VMgLQogU8/HCYo+TiixsmRsk5SkhEsoCat0gu0QhqIo3U3/4WakWOOCIkGQUF6e3Xv39o4vXnP8MzGrRPak4JiUjMNEGg5BqNoCbSCL30UmhTue++8MQT0KxZzfb/9a+hX7/wC2zJkvqJUXKWRtkSiVlVzVvU3l4ao4oR1M5ffRvX8FvW0ox11oxtVzeDXZqFP3RSvZo2rXxdda+EfV95oxljH2rG7EVbU7RjKy66phUnDGkFhfqVJ5LS22/DT38KvXvDc89tOWNuOpo1C6NuFRfDhReG0blE0qS7s0jM1LxFck1FIv3CpX350+KzadNyLfsWr2XHLmvDJGvJr9WrQ4fYVOvWroV16za9p+FQEsY0ng+cH71atoRWrcKrdevUnytbt802oU2lSK6ZOhWOOgo6dQq1JK1a1f5Y3/8+XHddqC352c/gpJMyF6fkNI2yJRKzbt1CM61kXbuGiQolO+XbiC1Zcc9235ScVPIadMRali5cS3O+o4jVtGIZrfmabtst49Kzl4XE5+uvw3vi5+XLqz63GWy3XfqJTNu20K5deG27bfUdg0XiMGtWmEeksBDeeiv84qmr9etDs6/Zs2H6dNhhh7ofMwPy7Z7d2KiGRCRmmiBQJE1mm5poVeLZRZDqMZutgEt/X8WxN2yAlStTJyuVLc+cuenzqlWVH7tp05CYbL/9piQl1XJFWdu26XcmFqmtL7+Eww4LCcRrr2UmGYGQ3IwfD3vuGTrIP/20EnKplhISkZhVNG+5+urQTKtLl5CMqP+ISM116ZK6xrHaDvUFBZtqN2pj7VpYvpwJDy/jj7d9zbqFS9m1bTmDDy+n/07lUF4eOvqWl0NpaXivrFbGDNq0qTppSS5r3rx2cUt+WroUfvKT8P7qq7Dbbpk9/m67wQ03hAkV//IXOO20zB5fco6abImI1EK+Vf83lnt2nJMy1vjca9eGPwjLyzd/VSQuyctLlsDGjalP3rLlpuSkbdtNNS0V76nKlMTkp5Ur4dBDQ9+RF1+Egw+un/Ns2AAHHRQmS5w+PfRRiVG+3bMbG9WQiIhIzoizxrHGI+Y1awYdO4ZXOjZuDM3Gqkpaysth8WL4+OOQ7HzzTeXHa9my8mSlsrKiIjW/acy+/RaOOw4++ACeeqr+khEItY5/+hP07QtDhsDEifrZkUopIRFJUFKiplMijd3gwfH8v633EfOaNNmUGKTru+9CYrJ0aUhakj8nln3+eXhftqzy47VoUXni0r59uHF27RreW7fWH6DZZP16OOWU0ETr4Yfh2GPr/5w9e8Lvfge/+AX88Y8hMRFJQQmJSCS5uUXFBIWgpEREqlfr/iv1qXlz2HHH8ErX+vXw1VeVJy6JZVOnhs9ffbVlc7KWLTclKBVJSuJ7x46aG6ahbNwYkoG//x3uvrth+3RceGHo2H7ppaETfaY6z0tO0Z1AJKIJCkWkLnJmxLzCwlDb0b59+vts3Biai33xRcjK5s0L7xWf33svJC6JCgqgc+ctE5XE9623zuy15SP3kAyMHw/XXw8XXdSw52/SBMaNgz32gLPPhpdf1pw+sgUlJCIRTVAoInWR1yPmNWkS5pvYYYcwU3cqq1aFL6YiWUl8f+MNKCsLHaETtW1bebLStWvoxK9mYVW74Qa46y64+GK45pp4YujWDW6/PQwDPHp0qDURSaBRtkQimqBQaiLfRmzRPTv7Nfo+cOvXw4IFWyYrie/JnfRbtAgX27kzdOgQXh07bvpcsdymTX4mLvfcE/pvnHFGqKWIs2bCHY44IiSfH34IO+/coKfPt3t2Y6MaEpFIzjS3EJG8kxN94AoLYaedwisV9zDKWHKSMncuzJ8P//53SGjWrNly36ZNQ+1NqmQl8fMOO8BWW9XvdTaUv/wlJCODBsGDD8bfTMosxLH77nDWWfD66/HHJFlDCYlIJK+bW4hIo5YXfeAqJoxs0wb69Uu9jXuYZ2PhwvBasGDT54rlij4tixeH7ZNtt13lNS2Jn9u2zd4/qJ99Fs48Ew45BB57LHsGD+jcOTQfO/PM8H7JJXFHJFlCTbZERGoh36r/dc/Obk2apP7b2qzyuRTz3vr1oSN+VclLxedU87kUFIQalX79wgSABx4I/fuH2pg4/etfMHAgfP/78MorsM028caTzD3U2rz8MkyZArvs0iCnzbd7dmOTJSmziIiI1FZWDjmc7QoL05+Y8ptvNk9WKhKWsrJQ2zJxYtiuqAj23XdTgjJgQOjn0lBKS8P8Ij16hJiyLRmBkCWPHQt9+oS+LW++mT01OBIb/QSIiIg0cuoDV89atgydsCvriL1oUeis/frrMGnSptGsmjeHvfcOyclBB8E++9TfUMYzZoSakTZt4B//CBNWZqsOHeDee8NEjbfdBldeGXdEEjM12RIRqYV8q/7XPTv7NfpRtnLJV1+FJ/+TJoUk5YMPQtu5wsIwLHJFgrLffqHPSl3NnQv77w/r1oXzNvAIVrXiDiedBBMmwOTJobN7Pcq3e3Zjo4RERKQWGuqXm5kNBO4CCoAH3f3mFNscDNwJNAWWuPtBZrYL8NeEzb4HXOvud5rZb4BzgfJo3Qh3n1hVHLpni9TBihXw9tshQZk0KTTzWrcudP7p129TgnLAAaGzfE0sWhT2Ky8P/Uf69q2XS6gX5eWh6VbnzvDuu/Xa/0YJSXbL0uEhRETEzAqAe4EjgN7AKWbWO2mbVsB9wLHu3gc4EcDdZ7p7P3fvB/QHVgNPJ+x6R8X66pIRkaqUlIR5nJo0Ce8lJXFHlIW23TY0p7rxxlCDsWwZvPpqaNq13XYwZgwcf3xoZrXHHmHiwMcfD31VqrJsWThuWRk8/3zjSkYgTGx5//3wn/+E70bylvqQiIhkrwHALHefDWBmjwGDgI8TtjkVeMrd5wG4++IUxzkU+NzdU3R7Fqm9nJj/JA5FRWFI3kMOCcvffRc6pFf0QXn4YbjvvrCuV69NNSgHHrhppILVq+GYY2D69DDM7777xnMtdXX88eGH5YYbwvXstVfcEUkMVEMiIpK9OgFfJCyXRWWJegGtzexfZjbZzE5PcZyTgUeTyoab2VQzG2dmrTMXsuSTquY/kRpo3jz0JxkxAl58MUwA+d57ocP3LrvAE0/AaadB166hGuqMM+Doo+Gtt8IEiIcfHvcV1M0994TakjPOCMmZ5B0lJJKSquBFsoKlKEvu+FdIaJJ1FHA4cI2Z9frfAcyaAccCf0vYZzTQA+gHLABuT3lys6FmVmpmpeXl5ak2kTw3b17NyiVNhYXwgx/AZZeFTt9Ll4Y5O+6+O5S/8EKoSRkzJnQMb+xatw6zuE+bBtdfH3c0EgM12ZItqApeJGuUATslLHcG5qfYZom7rwJWmdkkoC/wabT+COADd19UsUPiZzN7AHgu1cndfSwwFkKn9rpdiuQizX/SQJo0Cf1D+vaFiy4KI1StXl1/QwjH4cgj4eyz4Xe/CxMn7r133BFJA0qrhsTMBprZTDObZWZbDBZtZq3N7Omo+v89M6vfsdukXqkKXiRrvA/0NLPuUU3HycCEpG2eAQ4ws0IzKwL2BmYkrD+FpOZaZpY4E9zxwLSMRy55YdSo0B0ikeY/aQBmuZWMVPj976FTp9B0a82auKORBlRtQpLOKC/ACGCKu38fOJ0wRKU0UqqCF8kO7r4eGA68REgyHnf36WY2zMyGRdvMAF4EpgLvEYYGngYQJSiHAU8lHfoWM/vIzKYChwCXNMgFSc4ZPDhMut21a/gbuWvXsKzadKmV7baDceNg5kz49a/jjkYaUDpNttIZ5aU3cBOAu39iZt3MbIfEZgHSeKgKXiR7REPyTkwqG5O0fCtwa4p9VwNbTGrg7qdlOEzJY4MHKwGRDPrxj+H88+GOO+C448IcK5Lz0mmylc4oLx8CPwUwswFAV0JbZ2mEVAUvIiIisbnlFujeHc46C1atijsaaQDpJCTpjPJyM2HYySnARcB/gPVbHEgjtjQKqoIXERGR2LRsCX/6E8yeDVdcEXc00gDSabJV7Sgv7r4COAvAzAz4b/QiaTuN2NJIqApeREREYnPggXDxxXDnnWHyxEMPjTsiqUfp1JBUO8qLmbWK1gEMASZFSYqIiIiISM3deGOYqf7ss2GF/qzMZdUmJOmM8gLsBkw3s08Io3FdXF8Bi4iIiEge2GorGD8eysrCJJGSs9Kah8TdJ7p7L3fv4e6jorIxFSO9uPs77t7T3Xd195+6+9f1GbSIiIhISQl06xbmDezWLSxLjvnhD+FXvwozub/wQtzRSD1JKyERERERySYlJTB0aBim3j28Dx2qpCQnXX899OkDQ4bA13rmnYuUkGQ5Pf0RERHZ0tVXw+rVm5etXh3KJcc0bx6abi1aFDq6S85JZ5QtiUnF05+KG27F0x/QCFgiIpLf5s2rWbk0cv37hwnRWrcOVWKWalYKaaxUQ5LF9PRHREQktS5dalYuOeCKK8KTWSUjOUcJSRbT0x8REZHURo2CoqLNy4qKQrmINC5KSLKYnv6IiIikNngwjB0LXbuGB+Zdu4ZlNWkWaXyUkGQxPf0RERGp3ODBMGcObNwY3pWMiDROSkiyWL4+/dHIYiIiIiL5Q6NsZbnBg3M/AUmkkcVERERE8otqSCSraGQxERERkfyihESyikYWExEREckvSkgkq2hkMREREZH8ooREsopGFhMRERHJL0pIJKvk68hiIiIiIvlKo2xJ1sm3kcVERERE8plqSEREREREJDZKSEREREREJDZKSEREREREJDZKSEREspiZDTSzmWY2y8yurGSbg81siplNN7PXE8rnmNlH0brShPI2ZvaymX0WvbduiGsRySUlJdCtGzRpEt5LSuKOSKTxUkIiIpKlzKwAuBc4AugNnGJmvZO2aQXcBxzr7n2AE5MOc4i793P34oSyK4FX3L0n8Eq0LCJpKimBoUNh7lxwD+9DhyopEaktJSQiItlrADDL3We7+1rgMWBQ0janAk+5+zwAd1+cxnEHAeOjz+OB4zIUr0heuPpqWL1687LVq0O5iNScEhIRkezVCfgiYbksKkvUC2htZv8ys8lmdnrCOgf+EZUPTSjfwd0XAETv7VOd3MyGmlmpmZWWl5fX+WJEcsW8eTUrF5GqKSEREclelqLMk5YLgf7AUcDhwDVm1itat5+770Vo8nWhmR1Yk5O7+1h3L3b34nbt2tUwdJHc1aVLzcpFpGpKSEREslcZsFPCcmdgfoptXnT3Ve6+BJgE9AVw9/nR+2LgaUITMIBFZtYRIHpPp5mXiERGjYKios3LiopCuYjUnBISEZHs9T7Q08y6m1kz4GRgQtI2zwAHmFmhmRUBewMzzGxrM9sGwMy2Bn4CTIv2mQCcEX0+IzqGiKRp8GAYOxa6dgWz8D52bCgXkZorjDsAERFJzd3Xm9lw4CWgABjn7tPNbFi0foy7zzCzF4GpwEbgQXefZmbfA542Mwj3+kfc/cXo0DcDj5vZOcA8thyZS0SqMXiwEhCRTFFCIiKSxdx9IjAxqWxM0vKtwK1JZbOJmm6lOOZS4NDMRioiIlI7arIlIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxUUIiIiIiIiKxSSshMbOBZjbTzGaZ2ZUp1m9nZs+a2YdmNt3Mzsp8qCIiIiIikmuqTUjMrAC4FzgC6A2cYma9kza7EPjY3fsCBwO3m1mzDMcqIiIiIiI5Jp0akgHALHef7e5rgceAQUnbOLCNmRnQEvgKWJ/RSEVEREREJOekk5B0Ar5IWC6LyhL9AdgNmA98BFzs7hszEqGIiIiIiOSsdBISS1HmScuHA1OAHYF+wB/MbNstDmQ21MxKzay0vLy8xsGKiIiIiEhuSSchKQN2SljuTKgJSXQW8JQHs4D/ArsmH8jdx7p7sbsXt2vXrrYxi4iIiIhIjkgnIXkf6Glm3aOO6icDE5K2mQccCmBmOwC7ALMzGaiIiIiIiOSewuo2cPf1ZjYceAkoAMa5+3QzGxatHwP8FnjIzD4iNPG6wt2X1GPcIiIiIiKSA6pNSADcfSIwMalsTMLn+cBPMhuaiIiIiIjkOs3ULiIiItJIlJRAt27QpEl4LymJOyKRukurhkRERERE4lVSAkOHwurVYXnu3LAMMHhwfHGJ1JVqSEREspiZDTSzmYNU+EUAACAASURBVGY2y8yurGSbg81siplNN7PXo7KdzOw1M5sRlV+csP1vzOzLaJ8pZnZkQ12PiNTe1VdvSkYqrF4dykUaM9WQiIhkKTMrAO4FDiMMwf6+mU1w948TtmkF3AcMdPd5ZtY+WrUeuMzdPzCzbYDJZvZywr53uPttDXc1IlJX8+bVrFyksVANiYhI9hoAzHL32e6+FngMGJS0zamEeaDmAbj74uh9gbt/EH1eCcwAOjVY5CKScV261KxcpLFQQiIikr06AV8kLJexZVLRC2htZv8ys8lmdnryQcysG7An8G5C8XAzm2pm48ysdWbDFpH6MGoUFBVtXlZUFMpFGjMlJCIi2ctSlHnSciHQHzgKOBy4xsx6/e8AZi2BJ4FfuvuKqHg00APoBywAbk95crOhZlZqZqXl5eV1uhARqbvBg2HsWOjaFczC+9ix6tAujZ/6kIiIZK8yYKeE5c7A/BTbLHH3VcAqM5sE9AU+NbOmhGSkxN2fqtjB3RdVfDazB4DnUp3c3ccCYwGKi4uTEyERicHgwUpAJPeohkREJHu9D/Q0s+5m1gw4GZiQtM0zwAFmVmhmRcDewAwzM+CPwAx3/33iDmbWMWHxeGBavV2BiIhINVRDIiKSpdx9vZkNB14CCoBx7j7dzIZF68e4+wwzexGYCmwEHnT3aWa2P3Aa8JGZTYkOOcLdJwK3mFk/QvOvOcB5DXtlIiIimyghEZG8VFISxu6fNy+MUDNqVHY2g4gSiIlJZWOSlm8Fbk0qe5PUfVBw99MyHKaIiEitKSERkbyj2Y5FRESyh/qQiEje0WzHIiIi2UMJSRpKSqBbN2jSJLyXlMQdkYjUhWY7FhERyR5KSKpR0bRj7lxw39S0Q0mJSOOl2Y5FRESyhxKSaqhph0ju0WzHIiIi2UMJSTXUtEMk92i2YxERkeyhUbaq0aVLaKaVqlxEGi/NdiwiIpIdVENSDTXtEBERERGpP0pIqqGmHSIiIiIi9UdNttKgph0iIiIiIvVDNSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIiIiIhIbJSQiIlnMzAaa2Uwzm2VmV1ayzcFmNsXMppvZ69Xta2ZtzOxlM/ssem/dENciIiKSihISEZEsZWYFwL3AEUBv4BQz6520TSvgPuBYd+8DnJjGvlcCr7h7T+CVaFlERCQWSkhERLLXAGCWu89297XAY8CgpG1OBZ5y93kA7r44jX0HAeOjz+OB4+rxGkRERKqkhEREJHt1Ar5IWC6LyhL1Alqb2b/MbLKZnZ7Gvju4+wKA6L19xiMXERFJU2HcAYiISKUsRZknLRcC/YFDga2Ad8zs32nuW/XJzYYCQwG6dOlSk11FRETSllYNSXWdKs3sV1GHyilmNs3MNphZm8yHKyKSV8qAnRKWOwPzU2zzoruvcvclwCSgbzX7LjKzjgDR+2JScPex7l7s7sXt2rWr88WIiIikUm1Ckk6nSne/1d37uXs/4CrgdXf/qj4CFhHJI+8DPc2su5k1A04GJiRt8wxwgJkVmlkRsDcwo5p9JwBnRJ/PiI4hIiISi3SabP2vYySAmVV0jPy4ku1PAR7NTHgieeS77+DRR2HFinjOv9tucNhh8ZxbUnL39WY2HHgJKADGuft0MxsWrR/j7jPM7EVgKrAReNDdpwGk2jc69M3A42Z2DjCPaGQuERGROKSTkKTqGLl3qg2jp3MDgeGVrFd7ZJFU3OGss0JCEqdXX4VDDok3BtmMu08EJiaVjUlavhW4NZ19o/KlhD4nIiIisUsnIalJx8hjgLcqa67l7mOBsQDFxcU16lwpktOuuy4kIzfcAOef3/DnX7sW9t8fzj0Xpk6FoqKGj0FERETyUjoJSTqdKiucjJpridTM+PHw29/COefAiBFgqZ4BNIAHHoAf/Qh+8xu45ZZ4YhAREZG8k84oW+l0qsTMtgMOQp0jRdL32muhVuLHP4bRo+NLRiA01Tr3XLj9digtjS8OERERySvVJiTuvp7QJ+Qlwsgtj1d0qqzoWBk5HviHu6+qn1BFcsyMGfDTn0LPnvDEE9C0adwRhZqRHXYItTXr1sUdjYiIiOSBtOYhcfeJ7t7L3Xu4+6iobExix0p3f8jdT66vQEVyyuLFcNRR0Lw5TJwI220Xd0RBq1Zw332hH8mtW/SRFhEREcm4tBISEcmgNWtg0CBYuBCefRa6do07os0ddxyccAKMHAkzZ8YdjYiIiOQ4JSQiDWnjRjj9dHj3XSgpgR/8IO6IUrvnnjDS1pAhIWYRERGReqKERKQhjRgR+ovcdhscf3zc0VSuQwf4/e/hzTdhzJjqtxcRERGpJSUkIg3lgQfgd7+DCy6ASy6JO5rqnXFGmLn9iivgiy+q315ERESkFpSQiDSEf/wjTHh4xBFw113xDu+bLjO4//7QZOv888Ns8iIiIiIZpoREpL5NmxY6ie++O/z1r1CYznykWaJ7dxg1Cp5/Hh57LO5oREREJAcpIRGpTwsWwJFHwjbbwHPPhffG5qKLYO+94Re/gCVL4o5GREREcowSEpH6smoVHHMMfPVVSEY6d447otopKIAHH4Tly+GXv4w7GhEREckxSkhE6sOGDTB4MPznP6GZ1p57xh1R3ey+O1x1VRiq+IUX4o5GREREcogSEpH6cPnl8MwzcPfdYUb2XDBiBOy2G5x3HqxcGXc0IiIikiOUkIhk2h/+AHfeGZo3XXhh3NFkTvPm8Mc/QllZSE5EREREMkAJiUgmPf88XHwxDBoUJj/MNfvsEzq533svvPVW3NGIiIhIDlBCIpIp//kP/Pznob9ISUnoDJ6LRo2CLl1gyBD49tu4oxEREZFGTgmJSCaUlcHRR0ObNvDss7D11nFHVH9atgwTJn7ySUhOREREROpACYlIXa1cGZKRlStDk62OHeOOqP4dfjicfjrcfDNMnRp3NCIi0gBKSqBbN2jSJLyXlMQdkeQKJSQidbF+fWimNW0aPPEE7LFH3BE1nN//Hlq3hnPOCd+DiIjkrJISGDoU5s4F9/A+dKiSEskMJSQiteUeZi9/4QUYPRp+8pO4I2pYbdvCPfdAaSncdVfc0YiISD26+mpYvXrzstWrQ7lIXRXGHYBUY+nS0E5/+fJ4zt+mDZx6auOf2K8+3HlnSET+3/+Dc8+NO5p4nHRSeDx2zTVw3HHQo0fcEaVv1SoYPhyGDYO99447GhGRrDZvXs3KRWpCCUk2++678Efev/8NHTrEE8PixWH42j33DE1zTj01NNPJd08/DZddBiecADfdFHc08TGD++6DPn1C3f0//xnKst2nn8LPfgbTp0NxcVYnJGY2ELgLKAAedPebk9YfDDwD/DcqesrdR5rZLsBfEzb9HnCtu99pZr8BzgXKo3Uj3H1i/V2FiDR2XbqEZlqpykXqSglJtnKHs8+GN9+ERx+Fk0+OJ46vvw5PwP/4x/A0+fLLwx9y55wDBx0Uerblm/ffh8GDYcAAePjh/PwOEnXuDLfcEmoaxo0LPxvZ7Kmn4MwzoVkzePHFrG5qZ2YFwL3AYUAZ8L6ZTXD3j5M2fcPdj04scPeZQL+E43wJPJ2wyR3unoOT5YhIfRg1Kjx3Smy2VVSkwRYlM/L8L6ksdt118Mgj4X96XMkIhNqQ4cPDHBuTJ4ck6bnn4Ec/gp49Q3xffhlffA1t7lw45phQYzVhAmy1VdwRZYdzz4UDDwy1RgsWxB1NauvXw69+FRLq3XaDDz7I6mQkMgCY5e6z3X0t8BgwqBbHORT43N1TPN8UEane4MEwdix07Roqwrt2DcuDB8cdmeQCJSTZaPx4+O1v4ayz4Kqr4o5mk732CjN0L1gAf/lLuBv9+tehvvaoo8KT57Vr446y/ixfHq7zu+/C8L7t28cdUfZo0gQeeCBMlDh8eNzRbGnhQjj00ND88PzzYdKkxtLOoBPwRcJyWVSWbB8z+9DMXjCzPinWnww8mlQ23Mymmtk4M1M7TBGp1uDBMGcObNwY3pWMSKYoIck2r70Wnjb/6EcwZkx2tsffaqtwF3r1VZg1KyRNU6aEJ8+dO4dmXTNmxB1lZq1bF/qLzJwZEq/ddos7ouzTqxdcf334fp58Mu5oNnnjjdAH6v33QxO7++6D5s3jjipdqW4AnrT8AdDV3fsC9wB/3+wAZs2AY4G/JRSPBnoQmnQtAG5PeXKzoWZWamal5eXlqTYRERGpMyUk2eSTT+CnP4Wddw5/0DVrFndE1evRA264ITRleu452H//MARs796w336hT8E338QdZd24h6fq//xnqAU45JC4I8pel10W/vgfPjz0P4qTO9xxR/j3atkS3n0XTjst3phqrgzYKWG5MzA/cQN3X+Hu30SfJwJNzWz7hE2OAD5w90UJ+yxy9w3uvhF4gNA0bAvuPtbdi929uF27dpm5IhERkSRKSLLF4sVw5JEhCXn+eWjVKu6IaqawcFOzrbIyuPVW+Oqr0MG5Y0cYMgTeeSf8kdjY/O53oVP/r38dOkNL5QoLw3dVXh5qyuKycmUYkvjSS+HYY8NcKY1z0sr3gZ5m1j2q6TgZmJC4gZl1MAtVqWY2gHBfX5qwySkkNdcys44Ji8cD0+ohdhERkbQoIckGa9bAoEGhb8aECdC9e9wR1c0OO4Q/Rj/+GN56C048ER57DPbdF3bfPczw3Viafzz+eGiSduqpMHJk3NE0DnvuGf79x42DV15p+PN//DH84AchOb7lllDbuN12DR9HBrj7emA48BIwA3jc3aeb2TAzGxZtdgIwzcw+BO4GTnYPmb+ZFRFG6Hoq6dC3mNlHZjYVOAS4pAEuR0REJCXzmJ5YFxcXe2lpaSznziobN4ZRtJ54Av72t9APIxetXAl//Wt4ev7vf0PTpuHJ9TnnhJGOCgrijnBL77wTmvv84Afw8svQokXcETUea9ZA376wYQN89FEYG7IhPPpo6IPVsmVIgg8+uN5OZWaT3b243k6QZXTPFpHGLN/u2Y2NakjiNmJESERuuSV3kxGAbbbZ1Gxr2jS46CJ4/fXQTK1btzDT93//W+1hGsznn4eEaaedwiSISkZqZqut4MEHYfZsuPba+j/f2rXwi1+Emqx+/cKQvvWYjIiIiEjmNK6EpDH2P6jKAw+E/gnnnRc6A+eLPn3g9tvD/CV/+1toxjVqFHzve/DjH4en3N9+G198X30V+sNs3AgTJ8L221e/j2zpwAPDz/Ydd4QRrupLWVlIPu65By65JIxUt+OO9Xc+ERERyajG1WRr5EhYsQJuuik0+WnMXn4ZjjgCDjsMnn02dAbOZ198AQ89FPodzJkTJmTce+94ZkH//PNQW/PPf8IBBzT8+XPJ8uVhxLW2bUPH8kyPHPfqq6HJ45o14WfnxBMze/wq5Fv1v5psiUhjlm/37Mam8fwV7A5Ll8Ldd8N774X+CB07Vr9fNpo2Lcxp0bt3uI58T0YgNI265hq4+urwhHvcOPj003hiadUq9D9QMlJ3220Ho0eHQRtuuSWMVJYJGzeG4119NeyyS+i4rrlhREREGqXGVUMCUFICQ4fCttuGEZAa2x+NCxbAD38YJtp7993wh7hIrjv55NAXZ8qUuicOy5bBGWeEEel+/vPQV6Vly8zEWQP59rRNNSQi0pjl2z27sWlcfUggzBD+7rshITnkkDCEbGPpW7JqVegovWRJmERQyYjki7vvDknDkCGhdqO2PvwQiotD35677w79jWJIRkRERCRzGl9CAqET9Pvvh2Ygl10W2o2vWBF3VFXbsCEkU5Mnh+ZAe+0Vd0QiDad9+9C5/e234b77aneM8eND7eKaNWGEtosugjAfoIiIiDRijTMhgVBD8sQTcNtt8Pe/w4ABMH163FFV7le/gmeegTvvhGOOiTsakYZ32mlhzpmrroJ589Lf79tvw2hdZ54J++wD//lPmGRTREREckLjTUggPB297LIw0s7y5SEpefTRuKPa0r33hqfDv/hFeInkIzO4//7QxHLYsPSaWs6ZE/qJjR0LV14J//hHqG0RERGRnNG4E5IKBx4YJkLba68wMdpFF4WJ0rLBxIkhCTnmmNDfRSSfdesGN94IL7wQBqioyosvQv/+8NlnoRb0pps0Ip2IiEgOyo2EBMIQwK++GmpM/vAHOOigMGFanKZMCaMA9e0LjzwCBQXxxiOSDS68MPQF+eUvobx8y/UbN8L118ORR0LnzmH+kkGDGj5OERERaRBpJSRmNtDMZprZLDO7spJtDjazKWY23cxez2yYaWraNPQp+dvfwlwfe+4Jr7wSSyiUlcHRR4c5LZ57TiMBiVQoKAhD9a5YARdfvPm6pUvhqKPgN78JfU7eeQd23jmWMEVERKRhVJuQmFkBcC9wBNAbOMXMeidt0wq4DzjW3fsADTddcionnBCeqrZvHzrR3nhj3YYaramVK0MTreXL4fnnYccdG+7cIo1Bnz5hksRHHw0JO4SR8/baK9R03n8/PPQQFBXFGqaIiIjUv3RqSAYAs9x9truvBR4DkttPnAo85e7zANx9cWbDrIVddgnzlfz852E25+OOg6+/rv/zrl8fJoH76KNQU/P979f/OUUaoyuvDEN4DxsGd90F++8fOr6/9VaY/FRD+oqIiOSFdBKSTsAXCctlUVmiXkBrM/uXmU02s9MzFWCdtGwZOs7ec0/oIFtcHPp11Bf30C5+4sQwstbAgfV3LpHGrlmz0HRr/vzw/+aQQ8I8PcWaSFdERCSfpJOQpHpMmTxeZyHQHzgKOBy4xsx6bXEgs6FmVmpmpeWpOrPWBzMYPjxMpPbdd2Eeg4ceqp9z3XVXSEQuvzzMmyAiVdt7bxg9OvT9ev55aNs27ohERESkgaWTkJQBOyUsdwbmp9jmRXdf5e5LgElA3+QDuftYdy929+J27drVNuba2WefMDTwvvvCWWeFJiHffpu54z/zDFx6Kfz0p/C732XuuCK57rzzwuh4GoVOREQkL6WTkLwP9DSz7mbWDDgZmJC0zTPAAWZWaGZFwN7AjMyGmgHt24eJ1UaMgAceCG3W58yp+3FLS8P8Jz/4Afz5z9Akd0ZTFhERERGpT9X+5ezu64HhwEuEJONxd59uZsPMbFi0zQzgRWAq8B7woLtPq7+w66CgAEaNCjUas2aFUX1eeKH2x5s7N4yo1a4dTJigUYFERERERGogrUf57j7R3Xu5ew93HxWVjXH3MQnb3Oruvd19d3e/s74Czphjjw0daLt0CfMeXHcdbNhQs2MsXx7mGlmzJnRk32GH+olVRERERCRH5Xfboh49wsRrZ5wBI0eGxGTJkvT2XbcOTjwRPvkEnnwSeveufh8REREREdlMfickAFttBePGwdix8Npr0L9/mKCtKu5w4YXw8sthArdDD22YWEVEREREcowSEghDA597bpiQzSx0dr///pB4pHLrraFT/IgRcPbZDRuriIiIiEgOUUKSqLg49Cv50Y/C7NFnngmrV2++zRNPwBVXhNnYf/vbWMIUEREREckVSkiStW0bJmi7/vowhO8++8Bnn4V1//43nHZamMvkT3/S8L4iUu/MbKCZzTSzWWZ2ZYr1B5vZcjObEr2uTVg3x8w+ispLE8rbmNnLZvZZ9N66oa5HREQkmf6iTqVJE7j22jByVllZqDkZPTqMzNWpUxgyuEWLuKMUkRxnZgXAvcARQG/gFDNLNYLGG+7eL3qNTFp3SFRenFB2JfCKu/cEXomWRUREYqGEpCoDB4bZ3Xv1ggsugPXrQ5Ky/fZxRyYi+WEAMMvdZ7v7WuAxYFAGjjsIGB99Hg8cl4FjioiI1IoSkup07Qpvvgk33ggvvRSSExGRhtEJ+CJhuSwqS7aPmX1oZi+YWZ+Ecgf+YWaTzWxoQvkO7r4AIHpvn+rkZjbUzErNrLS8vLxuVyIiIlKJwrgDaBSaN4erroo7ChHJP5aiLHn4vw+Aru7+jZkdCfwd6Bmt28/d55tZe+BlM/vE3Sele3J3HwuMBSguLq5k2EEREZG6UQ2JiEj2KgN2SljuDMxP3MDdV7j7N9HniUBTM9s+Wp4fvS8GniY0AQNYZGYdAaL3xfV5ESIiIlVRQiIikr3eB3qaWXczawacDExI3MDMOpiZRZ8HEO7rS81sazPbJirfGvgJMC3abQJwRvT5DOCZer8SERGRSqjJlohIlnL39WY2HHgJKADGuft0MxsWrR8DnACcb2brgTXAye7uZrYD8HSUqxQCj7j7i9GhbwYeN7NzgHnAiQ16YSIiIgmUkIiIZLGoGdbEpLIxCZ//APwhxX6zgb6VHHMpcGhmIxUREakdNdkSEREREZHYKCEREREREZHYKCEREREREZHYKCEREREREZHYKCEREREREZHYKCEREREREZHYKCEREREREZHYKCEREREREZHYKCEREREREZHYKCEREREREZHYFMYdgIiIiIhItpg8eXL7wsLCB4Hd0cP7TNgITFu/fv2Q/v37L061gRISEREREZFIYWHhgx06dNitXbt2Xzdp0sTjjqex27hxo5WXl/deuHDhg8CxqbZR1iciIiIissnu7dq1W6FkJDOaNGni7dq1W06ocUq9TQPGIyIiIiKS7ZooGcms6PusNO9QQiIiIiIiIrFRQiIiIiIikiUWLlxYsOuuu/bedddde2+//fZ927dv//2K5W+//dbSOcYJJ5zQ7cMPP2xe1TY33XRTu9GjR7fJTNR1o07tIiIiIiK1NGYMbUaOpNPChTTr0IG1117Ll8OG8VVtj9ehQ4cNn3zyyccAl1566Y4tW7bcMHLkyEWJ22zcuBF3p6CgIOUxnnjiiTnVneeqq64qr22MmaYaEhERERGRWhgzhjaXXELXBQto5g4LFtDskkvoOmYMGa95mDZtWvOePXv2OfXUU7v06dOn97x585qecsopXXfffffddt555z6XX355x4pt+/fvv8vbb7+91bp169hmm236XXDBBZ122WWX3v369dv1yy+/LAT4xS9+sePIkSPbV2x/wQUXdNpjjz1269at2+4vv/zy1gArVqxocvjhh/fYZZddeh9zzDHdd999993efvvtrTJ9bUpIRERERERqYeRIOn377eZ/T3/7LU1GjqRTfZzv888/b3HeeectmTFjxsfdu3dfd+edd5ZNmzZtxowZM6a/9tpr206ePLlF8j7ffPNNwcEHH7xy5syZHxcXF39z7733bp/q2O7ORx99NGPUqFFfjBw5ckeAm2++uX379u3XzZw58+MRI0YsnDFjRlF9XJcSEhERERGRWli4kGY1Ka+rnXba6buDDjpodcXyuHHj2vTu3Xu3Pn369J49e3aLqVOnblF70aJFi40nnXTSCoD+/fuvnjNnTsrYTjzxxGUA++677+qysrJmAO+8807LwYMHfwWwzz77rOnRo8ea+rguJSQiIiIiIrXQoQNra1JeV1tttdXGis8fffRR8/vvv3+HSZMmffrpp59+fOCBB65Ys2bNFp3eCwsL/zeEcUFBgW/YsCFlx/gWLVpsTN7GvWFGP1ZCIiIiIiJSC9dey5ctWrAxsaxFCzZeey1f1ve5ly1bVrD11ltvaN269Ya5c+c2nTRp0raZPsc+++zzzaOPPtoa4L333ttq9uzZGe8/AhplS0RERESkVipG08rkKFvp2m+//Vb37Nnz2169evXp0qXLd/379/8m0+e48sorF5944onde/Xq1XuPPfZYvfPOO69p06bNhkyfxxqqKiZZcXGxl5aWxnJuEZG6MrPJ7l7cAOcZCNwFFAAPuvvNSesPBp4B/hsVPeXuI81sJ+BhoAOwERjr7ndF+/wGOBeoGPJxhLtPrCoO3bNFpDGryT37ww8/nNO3b98l9R1TY7Bu3TrWrVtnRUVF/tFHHzUfOHBgrzlz5nzUtGnTGh/rww8/3L5v377dUq1TDYmISJYyswLgXuAwoAx438wmuPvHSZu+4e5HJ5WtBy5z9w/MbBtgspm9nLDvHe5+W71egIiINGrLly8vOOigg3qtX7/e3J177rlnbm2SkeooIRERyV4DgFnuPhvAzB4DBgHJCckW3H0BsCD6vNLMZgCd0tlXREQEYPvtt98wffr0GfV9nrQ6tZvZQDObaWazzOzKFOsPNrPlZjYlel2b+VBFRPJOJ+CLhOWyqCzZPmb2oZm9YGZ9kleaWTdgT+DdhOLhZjbVzMaZWesMxiwiIlIj1SYkCU0GjgB6A6eYWe8Um77h7v2i18gMxykiko9SDc2Y3PHvA6Cru/cF7gH+vtkBzFoCTwK/dPcVUfFooAfQj1CLcnvKk5sNNbNSMystLy9PtYmIiEidpVND8r8mA+6+FqhoMiAiIvWrDNgpYbkzMD9xA3df4e7fRJ8nAk3NbHsAM2tKSEZK3P2phH0WufsGd98IPEC4z2/B3ce6e7G7F7dr1y6T1yUiIvI/6SQkGWkyICIiNfY+0NPMuptZM+BkYELiBmbWwcws+jyAcF9fGpX9EZjh7r9P2qdjwuLxwLR6vAYREZEqpZOQ1LnJwP8OpOp/EZG0uft6YDjwEjADeNzdp5vZMDMbFm12AjDNzD4E7gZO9jCe+37AacCPEvr3HRntc4uZfWRmU4FDgEsa8rpERKRyAwYM2OXJJ5/cbJLDkSNHtv+///u/LpXtU1RUtCfAnDlzmg4cOPB7lR130qRJRVWde+TIke1Xrlz5v/zgoIMO2nnJkiUFNbuCmksnIalTk4Gk7VT9LyJSA+4+0d17uXsPdx8VlY1x9zHR5z+4ex937+vuP3T3t6PyN93d3P37Cf37JkbrTnP3Pf5/e/cfW1V9xnH8/VAstDArtVJZtfwY9zpuSzosWaFt8YCZ/AAAC1hJREFU5pD9IWDQ6DAdpWm2EbO5H2pmFjfJZowm/mHMZthU4thqiqgRMnV2boYJk38arVsjGxYMw4IWW/ld2zrLffbHvdWuK/3B2nvaez6vpLnnfM/93vN825vnnqfne89JblubvCKXiIhMAOvWrTu+ffv23P5tO3bsyN2wYcOwN1ucN2/eJy+//PKhC933448/nt/Z2flpfbBnz5538vLyxvxGiAONpCC54CkDYx2siIiIiEg6q6mpOblr166c7u5uA2hpaclsb2+/qKysrGv58uXRWCy2KBqNxurr6y8Z2LelpSUzEokUAXR2dtr111+/IBqNxtasWbOgp6fn01lP1dXVhcXFxYsWLlxYdOedd34e4P7775/d3t5+0TXXXBMtKyuLAhQUFCxua2ubCnDvvffmRyKRokgkUnTffffN7tvfggULiqqqquYuXLiwqKKiItLZ2TnY7KohDXsfEnfvNbO+KQMZwNa+KQPJ7Y+RmDLwXTPrBbr5bMqAiIiIiMjk9K1vXcm+fUNOcxq14uIutm49cr7Nl19++bmSkpKPduzYkbNhw4ZTdXV1uWvXrj05c+bM+EsvvfRObm5uvK2tbWpZWdkX169ff2rKlMHPLzz00EOzs7Ky4gcOHPhnY2NjVkVFxadXyX344Yffy8/PP9fb20t5eflVjY2NWZs2bWp/9NFH8/fs2XNgzpw5vf1f67XXXst+6qmnLm1qatrv7pSWli5auXLl2by8vHOtra3T6+vrD5WXl7+7evXqBU8++eSs2267bdizOf2N6D4kFzplQERERERERueWW2458cwzz8wC2LlzZ25NTc2JeDxud9xxxxXRaDS2YsWKaHt7e+bRo0fPe3Jh7969M2tqao4DlJWVdUej0a6+bXV1dbmxWGxRLBaLHTx4cHpzc/P0oeLZvXv3zNWrV5+6+OKL4zk5OfE1a9acfPXVVz8HUFBQ8HF5eXk3wJIlS7oOHz48bbTj1Z3aRUREREQGM8SZjPFUXV19atOmTVfu3bs3u6enZ0plZWXXI488cunx48envvXWW/unTZvmBQUFi7u7u4c8uZD8RsV/efvttzM3b96c39TUtP+yyy47d/PNN8/r6ekZ8nWGmviUmZn56caMjAwfLqbBjLqDiIiIiIiMn5ycnPiyZcvObty4cd5NN910AuD06dMZeXl5n0ybNs1ffPHFz73//vuZQ71GZWVlZ319fS7A66+/Pv3AgQPZACdPnszIysqK5+bmnjty5MjU3bt35/T1mTFjxrnTp0//T31w7bXXdjY0NFxy9uzZKWfOnJnS0NAwa8WKFWfHarw6QyIiIiIiMsFUVVWdqK2t/cL27dsPAWzcuPHEqlWrFhYXFy8qKirqmj9/fs9Q/e+66672qqqq+dFoNFZUVNS1ePHijwCWL1/eXVxc3BWJRIoKCws/Li0t7ezrU1tb++GqVasis2fP/qSxsfFAX3tlZWXX+vXrj1999dWLAGpqajoqKiq6W1pahiyKRsqC+u750qVL/Y033ghk3yIi/y8za3L3pUHHkSrK2SIymY0mZzc3Nx8uKSn5cLxjCpvm5ua8kpKSeYNt05QtEREREREJjAoSEREREREJjAoSEREREZHPxOPx+Khv7ifnl/x9xs+3XQWJiIiIiMhn9nV0dOSoKBkb8XjcOjo6coB953uOrrIlIiIiIpLU29u78dixY08cO3asGP3zfizEgX29vb0bz/eESVOQbNsG99wDra1QWAgPPADV1UFHJSIiIiLjLZXHgaWlpe3A2vF5dRnMpChItm2DW2+FruQN7999N7EOKkpERERE0pmOA9PfpDgNdc89n70J+3R1JdpFREREJH3pODD9TYqCpLV1dO0iIiIikh50HJj+JkVBUlg4unYRERERSQ86Dkx/k6IgeeAByM7+77bs7ES7iIiIiKQvHQemv0lRkFRXw5YtMHcumCUet2zRF5lERERE0p2OA9PfpLjKFiTedHrjiYiIiISPjgPT26Q4QyIiIiIiIulJBYmIiIiIiARGBYmIiIiIiARGBYmIyARmZteZWYuZvWNmdw+y/atmdtrM/p78+dlwfc0s18xeMbODycdZqRqPiIjIQCpIREQmKDPLAH4FrAJiwDfMLDbIU19z9y8lf+4bQd+7gV3uHgF2JddFREQCoYJERGTi+jLwjrsfcvd/A08DN4xB3xuAuuRyHXDjGMYsIiIyKipIREQmrgLgSL/1o8m2gZabWbOZ/dHMikbQN9/d2wCSj7PHNmwREZGRmzT3IRERCSEbpM0HrL8JzHX3TjNbDfweiIyw79A7N7sVuBWgsLBwNF1FRERGLLCCpKmp6UMzezeo/V+gPODDoINIsTCOGcI57jCOGS583HPHOpBBHAWu7Ld+BfB+/ye4+5l+yw1m9mszyxum7wdmNsfd28xsDtA+2M7dfQuwBcDMOpSzJ40wjltjDo+JnLPlAgVWkLj7ZUHt+0KZ2RvuvjToOFIpjGOGcI47jGOGCT/u14GImc0H3gOqgPX9n2BmlwMfuLub2ZdJTMU9Dpwaou8LQC3wYPLx+eECUc6ePMI4bo05PMI67nSnKVsiIhOUu/ea2feBPwEZwFZ3/4eZfSe5/THg68B3zawX6Aaq3N2BQfsmX/pB4Fkz+zbQCqxL6cBERET6UUEiIjKBuXsD0DCg7bF+y5uBzSPtm2w/Dqwc20hFREQujK6yNTpbgg4gAGEcM4Rz3GEcM4R33GEQ1r9tGMetMYdHWMed1ixxZl9ERERERCT1dIZEREREREQCo4JkGGZ2pZm9amb7zewfZnZ70DGlkpllmNnfzOwPQceSCmZ2iZk9Z2ZvJ//my4OOKRXM7M7k+3ufmW03s+lBxzTWzGyrmbWb2b5+bblm9oqZHUw+zgoyRhkbYc7bYcvZEM68HYacDcrbYaKCZHi9wI/cfRGwDPiemcUCjimVbgf2Bx1ECv0SeNndvwiUEIKxm1kB8ENgqbsXk7giU1WwUY2L3wHXDWi7G9jl7hFgV3JdJr8w5+2w5WwIWd4OUc4G5e3QUEEyDHdvc/c3k8tnSSS6gmCjSg0zuwJYAzwRdCypYGYXA18BfgPg7v9291PBRpUyU4EsM5sKZDPg5nvpwN3/CpwY0HwDUJdcrgNuTGlQMi7CmrfDlrMh1Hk77XM2KG+HiQqSUTCzecASoDHYSFLmF8CPgXjQgaTIAqAD+G1yysMTZjYj6KDGm7u/BzxE4n4UbcBpd/9zsFGlTL67t0HiIBaYHXA8MsZClrfDlrMhhHk75DkblLfTkgqSETKzmcAO4A53PxN0POPNzK4H2t29KehYUmgqcDXwqLsvAT4iBKeCk/NvbwDmA58HZpjZhmCjEvn/hSlvhzRnQwjztnK2pCMVJCNgZheR+FDb5u47g44nRSqAtWZ2GHgauNbM6oMNadwdBY66e99/Up8j8UGX7r4G/MvdO9z9E2AnUB5wTKnygZnNAUg+tgccj4yREObtMOZsCGfeDnPOBuXttKSCZBhmZiTmpu5394eDjidV3P0n7n6Fu88j8WW5v7h7Wv8Hxt2PAUfM7Kpk00rgnwGGlCqtwDIzy06+31eS5l8K7ecFoDa5XAs8H2AsMkbCmLfDmLMhtHk7zDkblLfT0tSgA5gEKoAa4C0z+3uy7afu3hBgTDJ+fgBsM7NM4BDwzYDjGXfu3mhmzwFvkrg60d9Iwzvhmtl24KtAnpkdBX4OPAg8a2bfJvEhvy64CGUMKW+HS6jydlhyNihvh4nu1C4iIiIiIoHRlC0REREREQmMChIREREREQmMChIREREREQmMChIREREREQmMChIREREREQmMChIREREREQmMChIREREREQmMChIREREREQnMfwAMN0svmxTHNAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = models.Sequential()\n", "model.add(layers.Embedding(10000, 32))\n", "model.add(layers.LSTM(32))\n", "model.add(layers.Dense(1, activation='sigmoid'))\n", "\n", "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n", "history = model.fit(x_train, y_train, epochs=10, batch_size=128, \n", " validation_split=0.2, verbose=0)\n", "\n", "acc = history.history['acc']\n", "val_acc = history.history['val_acc']\n", "loss = history.history['loss']\n", "val_loss = history.history['val_loss']\n", "epochs = range(1, len(acc) + 1)\n", "\n", "plt.figure(figsize=(12, 5))\n", "plt.subplot(1, 2, 1)\n", "plt.plot(epochs, acc, 'bo', label='Training')\n", "plt.plot(epochs, val_acc, 'r', label='Validation')\n", "plt.title('Training and validation accuracy')\n", "\n", "plt.subplot(1, 2, 2)\n", "plt.plot(epochs, loss, 'bo', label='Training')\n", "plt.plot(epochs, val_loss, 'r', label='Validation')\n", "plt.title('Training and validation loss')\n", "plt.legend(bbox_to_anchor=(1.02, 0.2), loc=2, borderaxespad=0.5)\n", "plt.show()\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The validation accuracy is about 65%.\n", "\n", "Why isn't LSTM performing better than densely connect networks? \n", "\n", "One reason is that we made no effort to have more testing samples or tune hyperparameters such as the embeddings dimensionality or the LSTM output dimensionality. Another may be lack of regularization.\n", "\n", "The primary reason is that analyzing the global, long-term structure of the reviews (what LSTM is good at) isn't helpful for a sentiment-analysis problem. Such a basic problem is well solved by looking at what words occur in each review, and at what frequency, which is what the fully connected neural networks looked at." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### GRU" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "GRU layers (which stands for \"gated recurrent unit\") work by leveraging the same principle as LSTM, but they are somewhat streamlined and thus cheaper to run, albeit they may not have quite as much representational power as LSTM. This trade-off between computational expensiveness and representational power is seen everywhere in machine learning." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 160 samples, validate on 40 samples\n", "Epoch 1/10\n", "160/160 [==============================] - 2s 13ms/step - loss: 0.6937 - acc: 0.4625 - val_loss: 0.6927 - val_acc: 0.5750\n", "Epoch 2/10\n", "160/160 [==============================] - 0s 893us/step - loss: 0.6871 - acc: 0.7562 - val_loss: 0.6920 - val_acc: 0.6250\n", "Epoch 3/10\n", "160/160 [==============================] - 0s 937us/step - loss: 0.6815 - acc: 0.8375 - val_loss: 0.6907 - val_acc: 0.5250\n", "Epoch 4/10\n", "160/160 [==============================] - 0s 993us/step - loss: 0.6747 - acc: 0.8812 - val_loss: 0.6897 - val_acc: 0.5750\n", "Epoch 5/10\n", "160/160 [==============================] - 0s 937us/step - loss: 0.6669 - acc: 0.9188 - val_loss: 0.6875 - val_acc: 0.6000\n", "Epoch 6/10\n", "160/160 [==============================] - 0s 918us/step - loss: 0.6573 - acc: 0.9188 - val_loss: 0.6846 - val_acc: 0.5500\n", "Epoch 7/10\n", "160/160 [==============================] - 0s 943us/step - loss: 0.6460 - acc: 0.9125 - val_loss: 0.6827 - val_acc: 0.6500\n", "Epoch 8/10\n", "160/160 [==============================] - 0s 906us/step - loss: 0.6323 - acc: 0.9375 - val_loss: 0.6786 - val_acc: 0.6000\n", "Epoch 9/10\n", "160/160 [==============================] - 0s 924us/step - loss: 0.6158 - acc: 0.9500 - val_loss: 0.6748 - val_acc: 0.5750\n", "Epoch 10/10\n", "160/160 [==============================] - 0s 931us/step - loss: 0.5956 - acc: 0.9375 - val_loss: 0.6712 - val_acc: 0.5500\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyQAAAE/CAYAAAC3hAd1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXxU9fX/8fdJwhZBZUd2FCIEKCpRFFRUtOL+1YpVom3tD3Ftq61tVXAplVaL+4rUutXUal1pa9VWxb0gqMgmiJRNQBZlMyCEfH5/nIkZQpZJSHIzM6/n4zGPYe69c++5k3Az536WYyEEAQAAAEAUMqIOAAAAAED6IiEBAAAAEBkSEgAAAACRISEBAAAAEBkSEgAAAACRISEBAAAAEBkSkgbOzDLNbLOZda3NbaNkZj3NrNbnmzazY81scdzr+WZ2RCLb1uBYD5rZNTV9PwCUxfW+WvtN+uu9md1oZo/U9n6BZJQVdQCpxsw2x73MlvSNpB2x1xeGEAqqs78Qwg5JzWt723QQQti/NvZjZqMknRtCOCpu36NqY98AkhfX+4aD6z2Q3EhIalkI4ds/ELE7MqNCCP+paHszywohFNVHbEBV+H0EEsf1HgBqB1226lmsifZJM3vCzDZJOtfMDjOz/5rZejNbaWZ3mVmj2PZZZhbMrHvs9eOx9f8ys01m9p6Z9ajutrH1J5jZAjPbYGZ3m9k7ZvajCuJOJMYLzWyhmX1lZnfFvTfTzG43s3Vm9pmk4ZV8PmPN7K9llt1rZrfF/j3KzObFzuez2N2siva13MyOiv0728z+HIttjqSB5Rx3UWy/c8zs1Njy/pLukXRErHvE2rjP9oa4918UO/d1Zva8me2TyGdTnc+5JB4z+4+ZfWlmq8zsV3HHuTb2mWw0s+lm1rG87hJm9nbJzzn2eb4ZO86XksaaWS8zez12Lmtjn9tece/vFjvHNbH1d5pZ01jMfeK228fMCs2sdUXnC6Qyrvdc7yu73pdzDv8Xi2e9mb1mZvvHrbvGzFbEru+fxJ3roWb2QWz5F2Y2IdHjAQ1KCIFHHT0kLZZ0bJllN0raJukUeULYTNLBkgbJW6z2lbRA0mWx7bMkBUndY68fl7RWUp6kRpKelPR4DbZtJ2mTpNNi634uabukH1VwLonE+IKkvSR1l/RlyblLukzSHEmdJbWW9Kb/6pV7nH0lbZa0R9y+V0vKi70+JbaNSTpG0hZJ34mtO1bS4rh9LZd0VOzft0iaIqmlpG6S5pbZ9ixJ+8R+JiNjMbSPrRslaUqZOB+XdEPs39+NxXiApKaS7pP0WiKfTTU/570kfSHpZ5KaSNpT0iGxdVdLmimpV+wcDpDUSlLPsp+1pLdLfs6xcyuSdLGkTPnvY46kYZIax35P3pF0S9z5zI59nnvEth8SWzdJ0vi44/xC0nNR/z/kwaM+HuJ6z/W++tf7GyU9Evt3n1gcx8R+RtfEPvdGkvpKWiKpQ2zbHpL2jf37fUnnxP7dQtKgqP8v8OBRkwctJNF4O4Tw9xBCcQhhSwjh/RDC1BBCUQhhkfyL3dBK3v90CGF6CGG7pAL5hbG6254s6aMQwguxdbfL/5iVK8EYfx9C2BBCWCz/Y1ByrLMk3R5CWB5CWCfppkqOs0j+hfe02KLjJK0PIUyPrf97CGFRcK9JelVSuQMZyzhL0o0hhK9CCEvkd8Hij/tUCGFl7GfyF/mXi7wE9itJ+ZIeDCF8FELYKukqSUPNrHPcNhV9Njup4nM+VdKyEMKdIYRvQggbQwjTYutGSbomhPBp7Bw+CiF8mWD8S0MI94cQdsR+HxeEEF4NIWwLIayW/26UxHCYpDaSfh1C+Dq2/TuxdY9KGmlmFnt9nqQ/JxgDkKq43ld8nLS+3pdxtqTJIYTXYj+jm+Q3nQbJbxo1ldTXvNvf/2KfneSJZS8zax1C2BRCmJrgeQANCglJNJbFvzCz3mb2T/MuOBsljZN/6avIqrh/F6rygY0VbdsxPo4QQpDfYSpXgjEmdCz5nZ7K/EXSObF/j5T/YS2J42Qzm2reZWm9/G5VZZ9ViX0qi8HMfmRmM2NN5esl9U5wv5Kf37f7CyFslPSVpE5x2yT0M6vic+4iaWEFMXSR9FmC8ZZV9vexg5k9ZWafx2J4pEwMi4MPqN1JLDEpknS4mfWT1FXSP2sYE5AquN5XLm2v91Xst1j+M+oUQpgvb3EeJ2m1eRfADrFNz5eUK2m+mU0zsxMTPA+gQSEhiUbZKRAfkN8l6hlC2FPSdfIm6rq0Ut6kLkmK3dXuVPHmuxXjSvkX2RJVTVP5pKRjY3ecTpP/wZKZNZP0tKTfy5vX95b0SoJxrKooBjPbV9L98m5LrWP7/SRuv1VNWblC3i2gZH8t5F0FPk8grrIq+5yXSdqvgvdVtO7rWEzZccs6lNmm7PndLJ8tqH8shh+ViaGbmWVWEMdjks6Vt448FUL4poLtgHTB9b5y6Xy9r2y/GfKf2eeSFEJ4PIQwRN5dK1P+uSiEMD+EcLa8W96tkp4xs6a7GQtQ70hIGoYWkjZI+tp8UPCF9XDMf0g6yMxOMbMs+biEtnUU41OSLjezTuYDnH9d2cYhhC/k4xweljQ/hPBpbFUT+biGNZJ2mNnJ8rEOicZwjZntbT5v/2Vx65rL/witkf+tHiW/Y1biC0mdLW5weRlPSPp/ZvYdM2si/0PxVgihwjuQlajsc54sqauZXWZmjc1sTzM7JLbuQUk3mtl+5g4ws1byP8yr5INpM81stOL+6FUSw9eSNphZF0lXxq17T9I6Sb8zHzjazMyGxK3/s6Qz5Xc6H6vB+QOpjut9nDS/3peN+VQzOyp27F/Kx/1MNbM+ZnZ07HhbYo8d8hM4z8zaxFpUNsTOrXg3YwHqHQlJw/ALST+UX3wekN8xqlOxPwLfl3Sb/AvmfpI+lN8Zr+0Y75f3/Z0lH4D3dALv+Yt80OJf4mJeL+kKSc/JBwqeKf9Dm4jr5XfuFkv6l+K+LIcQPpZ0l6RpsW16S4rvh/tvSZ9K+sLM4pviS97/krwp/bnY+7vK+xnXRIWfcwhhg7yP9ffkgyoXqLRf9wRJz8s/543yPt9NY10zLpAPkFwrH+ReVR/j6yUdIv/jNlnSM3ExFMn7o/eRt5Yslf8cStYvlv+ct4UQ3q3muQPpgOv9rtL1eh+/3znyz/x+ebI0XNKpsfEkTST9QX4NXyVvkRkbe+uJkuaZz+J2i6TvhxC27W48QH0z/76CdBfrgrNC0pkhhLeijgfJy8wek7QohHBD1LEA2BXXewANDS0kaczMhpvZXrFm4GvlA5KnVfE2oEKx/tmnSXoo6lgAlOJ6D6AhIyFJb4dLWiRvBh4u6f8YhIyaMrPfy2uh/C6EsDTqeADshOs9gAaLLlsAAAAAIkMLCQAAAIDIkJAAAAAAiExWVAdu06ZN6N69e1SHB4DdMmPGjLUhhMpqOaQUrtkAklm6XbOTTWQJSffu3TV9+vSoDg8Au8XMlkQdQ33img0gmaXbNTvZ0GULAAAAQGRISAAAAABEhoQEAAAAQGRISAAAAABEhoQEAAAAQGRISAAAAABEhoQEAAAAQGRISABEpqBA6t5dysjw54KCqCNCbeFnCwBIVGSFEQGkt4ICafRoqbDQXy9Z4q8lKT8/uriw+/jZAgCqgxYSAJEYM6b0C2uJwkJfjuTGzxYAUB0kJAAisXRp9ZYjefCzBQBUBwkJgEh07Vq95Uge/GwBANXBGBIAkRg/fudxBpKUne3LkdxKfrY/LLxPl+sObVILbc7YU/u1aiHlt5BaVPDYc8/ylzdpIplFfVoAgDpCQgIgEiWDm8eM8a48Xbv6F1kGPSe/kp/hG1d01AdrDlK7ZpvUr+smtSteKv13k7Qp9ti6NbEdZmVVnbSUl9y0bi116SJ17Cg1blx3JwwA2C0WQojkwHl5eWH69OmRHBtAqYKC9EwKdve8zWxGCCGv7iJsWOrkmr19u7R5c2mCEv/YuLH85ZVtW1xc/nHMpPbtpc6dPUHp0qX03yXPHTtKjRrV7vkBaDDS7ZqdbGghAdJYuk7Pmq7n3eA0aiS1bOmP3RWCtGXLzonKmjXSsmXS8uWlz/PnS6++6klMPDOpQ4ddE5X4Z5IWAKgTtJAAaax7d/8yXla3btLixfUdTf2pjfNOt7ttKXfN3rhx50SlvOdNm3Z+T0nSUlHC0qWLtM8+JC1AA5Ru1+xkQwsJkMbSdXrWdD1vxNlzTyk31x8V2bix4oRl3jzplVe8y1m8jAxPWtq1k9q08Ufr1pU/Z2czaB9AWiMhAdJY167ltxSk+vSs6XreqKY995T69vVHeUIov6Vl2TLvLrZunf+irVsnffllxcdp2rTqpKXssubNSWIApAwSEiCNpevUu+l63qhlZtJee/mjoqSlRFGR9NVXnpysXVvx89q10syZ/vzll570lKdx4/KTlnbtpD59pAEDpF69fIYyAGjguFIBaSxdp95N1/NGhLKypLZt/ZGoHTuk9et3TlYqSmTmzCl9XTLbWJMmnigNGCB95zuljzZt6uYcAaCGGNQOADWQbgMkuWYniW3bpE8+8VaWjz/2x8yZ0hdflG7TsWNpclKSrOy/P4PxkdLS7ZqdbGghAQAgVTRuXJpsxPviC2nWrJ0Tldde8wRG8mQkN3fX1pT27ev/HACkHRISNDhRFupL1yKBAFJc+/b+OPbY0mXbt3tdlviWlP/8R3rssZ3fV7Y1pXdv7w4GALWEhAQNSpQF6yiWB6QGbiwkqFEjqV8/f4wcWbp87drSJKXkcc890jff+PqsLE9K4ltTBgzw6Y6Z+QtADTCGBA1KlIX60rVIIGom3fojJ8s1u+yNBclnUJs0iaRktxQVSZ9+WtqSUpKoLFtWuk379tKQIdKRR0pHHOFJSmZmdDEDcdLtmp1sSEjQoGRklD/LpVnpxDGpeGzu6CafdPvjlizXbG4s1LMvv/SxKR9/LL3/vvTWW6UfdIsWnqAccYQ/Dj7Ya64AEUi3a3ayocsWGpQoC9ZFdWy6igG1Z+nS6i3HbmrVSho61B8lli3zxKTkMWaML2/SRDrkkNIEZfBgLz4JIO1lRB0AEG/8eO9eEa++CtZFdewxY3buXiL565K/4QASV9ENhPq4qYGYLl18TMr990uzZ/uYlOefly67zMeh3HyzdMIJUsuW0sCB0uWXS888I61eHXXkACJCQoIGJT/f+3p36+Zdpbp1q7++31Edmzu6QO2J8qYGKtC6tXTaadItt0hTp3qxx3//Wxo71qvcP/CAdOaZPgald2/pggt8pq/FiyuuVA8gpTCGBIgYfd6TU7r1R06mazZjspLMtm3SjBmlXbzeftuTFknq3Lm0i9eRR0p9+viAP6Ca0u2anWxISICIMStQckq3P25cs1Fviou9q9dbb0lvvunPK1f6ulatpMMPL01QDjyQCvNISLpds5NNQoPazWy4pDslZUp6MIRwU5n1LSU9JGk/SVsl/TiEMLuWYwVSUknSwR1dAJC3gJTUN7n0Uu+2tWjRzgnK5Mm+bXa2dNhhPli+Tx/v8tW7t8/wBSBpVNlCYmaZkhZIOk7ScknvSzonhDA3bpsJkjaHEH5jZr0l3RtCGFbZfrnbBiCZpdvdNq7ZaFBWrvSuXSUJypw5XiulROfOpQlKnz6l/27fnuKNaSrdrtnJJpEWkkMkLQwhLJIkM/urpNMkzY3bJlfS7yUphPCJmXU3s/YhhC9qO2AAAJDm9tlHGjHCH5K0fbv02WfSvHmlj08+kR5+WNq8ufR9e+9dfqLSowdFHIEIJZKQdJIUV4pVyyUNKrPNTElnSHrbzA6R1E1SZ0kkJAAAoG41alTaXev000uXhyAtX+7JSXyi8uKLnqyUaNJEysnZNVnJyZGaNav/8wHSTCIJSXltm2X7ed0k6U4z+0jSLEkfSioq+yYzGy1ptCR1ZVJ4AABQl8y8LkqXLtJxx+287quvdk5U5s3z2b6eftoH1pe8v3v38ltVWreu99MBUlUiCclySV3iXneWtCJ+gxDCRknnS5KZmaT/xR4qs90kSZMk749cs5ABAAB2U8uWPiD+sMN2Xr51q7Rgwa6tKq+95utKtG3rycmxx0pnnSXtv3/9xg+kkEQSkvcl9TKzHpI+l3S2pJHxG5jZ3pIKQwjbJI2S9GYsSQEAIG1QAyUFNG1aOstXvB07vGhUSYIyb5708cfS9ddL110nDRjgiclZZ0k9e0YTO5CkqkxIQghFZnaZpJfl0/4+FEKYY2YXxdZPlNRH0mNmtkM+2P3/1WHMAAA0OGVrCi1Z4q8lkpKUkJkp7buvP046qXT5559LzzwjPfmkZ6Njxnh9lJLkZN99o4sZSBIURgSAGki3KSS5Zlete3dPQsrq1k1avLi+o0Ekli3zMShPPSX997++LC/PE5MRI/yXBJFIt2t2ssmIOgAAQM2Y2XAzm29mC83sqgq2OcrMPjKzOWb2RtzyK2LLZpvZE2bWtP4iT01Ll1ZvOVJQly7SFVdI773nWeiECT4w/le/8qmFDz1Uuu02fimAMkhIUK6CAr+Rk5HhzwUFUUcEIF6saO29kk6Q14I6x8xyy2yzt6T7JJ0aQugraURseSdJP5WUF0LoJ++Oe3Y9hp+SKpo8kkkl01S3btKVV0rTpnml+Ztv9uKNv/iFrxs8WLrjDp+WGEhzJCTYRUk/6CVLfAr3kn7QJCVAg/Jt0drYhCIlRWvjjZT0bAhhqSSFEFbHrcuS1MzMsiRlq8zsiai+8eOl7Oydl2Vn+3KkuR49vJVk+nTp00+l3/1O2rLFW1O6dJGOOEK6+25pBf8NkZ5ISLCLMWNKB2WWKCz05QAajPKK1nYqs02OpJZmNsXMZpjZDyQphPC5pFskLZW0UtKGEMIr9RBzSsvPlyZN8pvfZv48aRID2lFGz57S1VdLH34ozZ8v/fa30oYN0k9/KnXuLA0dKt13n/QFtaWRPkhIsAv6QQNJIZGitVmSBko6SdLxkq41sxwzaylvTekhqaOkPczs3F0OYDbazKab2fQ1a9bUbvQpKj/fhw4UF/szyQgqlZMjjR3r0wfPnSvdcIO0bp106aVSx47SMcdIEydK/P9DiiMhwS7oBw0khSqL1sa2eSmE8HUIYa2kNyUNkHSspP+FENaEELZLelbS4LIHCCFMCiHkhRDy2rZtWycnASCmTx+vZzJ7tj/GjvUuXBdfLHXo4JXm//hHae3aqCMFah0JCXZBP2ggKXxbtNbMGssHpU8us80Lko4wsywzy5Y0SNI8eVetQ80s28xM0rDYcgANQd++0m9+U1p88ZprSgd0duggHX+89Kc/SatXV70vIAmQkGAX9IMGGr4QQpGkkqK18yQ9VVK0Nq5w7TxJL0n6WNI0SQ+GEGaHEKZKelrSB5Jmyf8WTIrgNABUxkzq39/Hmcyf7+NOfvUraeFCadQoT04OP1z6wx98PZCkKIwIADWQbkW2uGYDDUgI0syZ0gsvSJMnSx984MtzcqTTTpNOPVU67DCvLg9J6XfNTja0kAAAACQTM+mAA6Trr5dmzPBZZ+691wuH3XGHTyPcoYN0/vnSc89JX38ddcRApUhIAAAAklmXLtIll0gvv+yD3p96yseZPP+8dMYZUuvW0skne//rlSujjhbYBQkJAABAqthzT2nECOnxx33Q+2uv+Uxdc+dKF17o0wkPGuQz1cye7d2/gIiRkAAAAKSiRo2ko4+Wbr9d+uwzadas0ikzx471AfP77ecV419/Xdq+Pdp4kbZISAAAAFKdmdSvn08hPHWq1zh54AEpN1e6/34vwti+vXTuud7la+PGqCNGGiEhAQAgBRQU+JjmjAx/LiiIOiI0aPvs43VN/vEPH3fy7LM+O9dLL0nf/77Upo2PQ7n3XmnZsqijRYojIQEAIMkVFPh3yyVLfEhASQ09khIkpHlz6fTTpUcekb74QnrrLelnP5P+9z/pssukrl2lgw7yYo0ffsi4E9Q6EhIAAJLcmDFSYeHOywoLfTlQLZmZXmxxwgRpwQKvFn/zzVKzZp6QHHSQJygXXyz985/Sli1RR4wUQEICAECSW7q0esuBhPXu7dXh33lHWrVKeughKS9P+vOffSrh1q2lU07x8SjLl0cdLZIUCQkAAEmua9fqLQdqpF270mKL69Z53ZNRo3z64Isu8nooBx4oXXutD5wvLo46YiQJEhIAAJLc+PFSdvbOy7KzS2d4BWpdkybSd78r3XWXtGiRNGeOd+1q0UL63e+kQw/1gfPnny898wyzdqFSJCQAACS5/Hwvwt2tm8/u2q2bv87PjzoypAUznz74V7+S3nxTWrPGZ1QYNsyrxZ95ps/addxx0p13ek0UII6FiGZKyMvLC9OnT4/k2ACwu8xsRgghL+o46gvXbAA1UlQkvfuuTy/8j3/4IHnJx6acfLI/Bg/2Io51KN2u2cmGFhIAAADUjaws6cgjpT/8QZo7V1q40FtJunTx56OO8rEp55zjrSrr1kUdMSJAQtLAUegKAACkjP32k376U+mVVzz5ePZZ6YwzpNdf9yrx7dpJRxzh41Fmz6bmSZogIWnAKHQFAABSVosWXpDxT3+SVqyQpk2Txo71IjpXXSX17y/16OHFGV9+Wdq6NeqIUUdISBowCl0BAIC0kJEhHXywF1+cMcNrmkyaJA0Y4LVPhg/3mieXXBJ1pKgDWVEHgIpR6AoAAKSlTp2kCy7wx5Yt0pQpPii+ffuoI0MdICFpwLp29W5a5S0HAABIC82aSSec4A+kJLpsNWAUugIAAECqIyFpwCh0BQAAgFRHl60GLj+fBAQAAACpixYSAAAAAJEhIQEAAAAQGRISAAAAAJEhIQEAADVWUCB17+517bp399cAUB0MagcAADVSUCCNHi0VFvrrJUv8tcSELAASRwsJAACokTFjSpOREoWFvhwAEkVCAgAAamTp0uotB4DykJAAAIAa6dq1essBoDwkJAAAoEbGj5eys3delp3tywEgUSQkAACgRvLzpUmTpG7dJDN/njSJAe0AqodZtgAAQI3l55OAANg9CbWQmNlwM5tvZgvN7Kpy1u9lZn83s5lmNsfMzq/9UAEAAACkmioTEjPLlHSvpBMk5Uo6x8xyy2x2qaS5IYQBko6SdKuZNa7lWAEAAACkmERaSA6RtDCEsCiEsE3SXyWdVmabIKmFmZmk5pK+lFRUq5ECAAAASDmJJCSdJC2Le708tizePZL6SFohaZakn4UQisvuyMxGm9l0M5u+Zs2aGoYMAAAAIFUkkpBYOctCmdfHS/pIUkdJB0i6x8z23OVNIUwKIeSFEPLatm1b7WABAAAApJZEEpLlkrrEve4sbwmJd76kZ4NbKOl/knrXTogAAAAAUlUiCcn7knqZWY/YQPWzJU0us81SScMkyczaS9pf0qLaDBQAAABA6qmyDkkIocjMLpP0sqRMSQ+FEOaY2UWx9RMl/VbSI2Y2S97F69chhLV1GDcAAACAFJBQYcQQwouSXiyzbGLcv1dI+m7thgYAqIyZDZd0p/xm0YMhhJvK2eYoSXdIaiRpbQhhaGz53pIelNRPPi7wxyGE9+opdAAAvkWldgBIQnE1oo6Tj/V738wmhxDmxm2zt6T7JA0PISw1s3Zxu7hT0kshhDNj3XGz6zF8AAC+lVCldgBAg5NIjaiR8glHlkpSCGG1JMVmQTxS0p9iy7eFENbXW+QAAMQhIQGA5JRIjagcSS3NbIqZzTCzH8SW7ytpjaSHzexDM3vQzPao+5ABANgVCQkAJKdEakRlSRoo6SR5vahrzSwntvwgSfeHEA6U9LWkq3Y5AMVsAQD1gIQEAJJTIjWilsvHiXwdm/nwTUkDYsuXhxCmxrZ7Wp6g7IRitgCA+kBCAgDJKZEaUS9IOsLMsswsW9IgSfNCCKskLTOz/WPbDZM0VwAARIBZtgAgCSVSIyqEMM/MXpL0saRi+dTAs2O7+Imkglgys0jS+fV/FgAAkJAAQNKqqkZU7PUESRPKee9HkvLqNECgjhUUSGPGSEuXSl27SuPHS/n5UUcFoLpISAAAQNIpKJBGj5YKC/31kiX+WiIpAZINY0gAAEDSGTOmNBkpUVjoywEkFxISAACQdJYurd5yAA0XCQkAAEg6XbtWbzmAhouEBAAAJJ3x46Xs7J2XZWf7cgDJhYQEAAAknfx8adIkqVs3ycyfJ01iQDuQjEhIElBQIHXvLmVk+HNBQdQRAQCA/Hxp8WKpuNifSUaA5MS0v1VgWkEAAACg7tBCUgWmFQQAAADqDglJFZhWEAAAAKg7JCRVYFpBAAAAoO6QkFSBaQUBAACAukNCUgWmFQQAAADqDrNsJSA/nwQEAAAAqAu0kAAAAACIDAkJAAAAgMiQkAAAAACIDAkJAAAAgMiQkAAAAACIDAkJAAAAgMiQkAAAAACIDAkJAAAAgMiQkAAAAACIDAkJAAAAgMiQkAAAAACIDAkJAAAAgMiQkAAAAACIDAkJAAAAgMiQkAAAAFRTQYHUvbuUkeHPBQVRRwQkr6yoAwAAAEgmBQXS6NFSYaG/XrLEX0tSfn50cQHJihYSAACAahgzpjQZKVFY6MsBVB8JCQAAQDUsXVq95QAqR0ICAABQDV27Vm85gMollJCY2XAzm29mC83sqnLW/9LMPoo9ZpvZDjNrVfvhAgAARGv8eCk7e+dl2dm+HED1VZmQmFmmpHslnSApV9I5ZpYbv00IYUII4YAQwgGSrpb0Rgjhy7oIGAAAIEr5+dKkSVK3bpKZP0+axIB2oKYSmWXrEEkLQwiLJMnM/irpNElzK9j+HElP1E54AAAADU9+PgkIUFsS6bLVSdKyuNfLY8t2YWbZkoZLemb3QwMAAACQ6hJJSKycZaGCbU+R9E5F3bXMbLSZTTez6WvWrEk0RgAAAAApKpGEZLmkLnGvO0taUcG2Z6uS7lohhEkhhLwQQl7btm0TjxIAAABASkokIXlfUi8z62FmjeVJx+SyG5nZXpKGSnqhdkMEAAAAkKqqHNQeQigys8skvSwpU9JDIYQ5ZnZRbP3E2KanS3olhPB1nUULAAAAIKUkMsuWQggvSnqxzLKJZV4/IumR2iJOfAcAACAASURBVAoMAAAAQOqjUjsAJKmqitbGtjkqVrR2jpm9UWZdppl9aGb/qJ+IAQDYVUItJACAhiWuaO1x8slH3jezySGEuXHb7C3pPknDQwhLzaxdmd38TNI8SXvWU9gAAOyCFhIASE7fFq0NIWyTVFK0Nt5ISc+GEJZKUghhdckKM+ss6SRJD9ZTvAAAlIuEBACSUyJFa3MktTSzKWY2w8x+ELfuDkm/klRct2ECAFA5umwBQHJKpGhtlqSBkoZJaibpPTP7rzxRWR1CmGFmR1V4ALPRkkZLUteuXWsjZgAAdkELCQAkp0SK1i6X9FII4esQwlpJb0oaIGmIpFPNbLG8q9cxZvZ42QNQzBYAUB9ISAAgOSVStPYFSUeYWZaZZUsaJGleCOHqEELnEEL32PteCyGcW5/BAwBQgi5bAJCEEilaG0KYZ2YvSfpYPlbkwRDC7OiiBgBgVyQkAJCkEixaO0HShEr2MUXSlDoIDwCAhNBlCwAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBAAAAEBkSEgAAAAARIaEBGgoiould96RioqijgQAAKDekJAADcVdd0mHHy6ddJK0fn3U0QAAGqCCAql7dykjw58LCqKOCNh9JCRAQ7BihXTddVJurvT669KgQdL8+VFHBQBoQAoKpNGjpSVLpBD8efRokhIkPxISoCG48kpp2zZp8mTptdekr77ypOTll6OODADQQIwZIxUW7ryssNCXA8mMhASI2quvSk88IV19tbTfft5t6/33pW7dpBNPlO64w2+FAQDS2tKl1VsOJAsSEiBK27ZJl13micivf126vFs3H+B+2mnSFVdIo0ZJ33wTXZwAgMh17Vq95UCyICEBonTbbdInn0h33y01bbrzuubNpaeflq69VnroIWnYMGn16mjiBABEbvx4KTt752XZ2b4cSGYkJEBUliyRxo2TTj9dOuGE8rfJyPBtnnxS+uAD6eCDpY8+qt84AQANQn6+NGmSN6Kb+fOkSb4cSGYkJEBULr/c/6LccUfV2551lvT2216rZMgQ6Zln6j4+AECDk58vLV7sfw4WLyYZQWogIQGi8OKL0vPP+1S/iXb+PeggH+z+ne9IZ57pLScMdgcAAEmOhASob1u2SD/5idSnjw9Yr44OHbxOyQ9+IF1/vfT970tff103cQIAANSDrKgDANLOTTdJixZ5vZHGjav//qZNpUcekfr3l371K2nhQumFF6QuXWo9VAAAgLpGCwlQnxYulG6+WRo5Ujr66Jrvx8yLKf7jH9Jnn/lg9/feq704AQAA6gkJSSIKC6VrrpH++9+oI0EyC8FrjjRpIt1yS+3s88QT/feyeXPpqKO85QQAACCJkJBUZdkyr5z9+99L55zjyQlQE88+K738sg9G32ef2ttvnz7StGnSEUdI55/vLSc7dtTe/gEAAOoQCUll3nvPu8IsXCjdcIPPr/e730UdFZLR5s0+ze+AAdKll9b+/lu1kv71L2+BufVW6eSTpQ0bav84AAAAtYyEpCKPPupdYJo39y4x118vnXuuNGGCtGBB1NEh2fz2t9Ly5dL990tZdTSXRKNGXvH9gQek//xHOvRQ6dNP6+ZYAAAAtYSEpKwdO6Rf/lL60Y+8q9bUqVJurq+bMEFq1szvcFP/AYmaO1e67Tbpxz+WDjus7o83erQnJGvWSIMG+b8BAAAaKBKSeBs2SKee6gOOL71UeuklqXXr0vUdOkg33uhf8P72t+jiRPIIQbrkEqlFC5/ut74MHepFFDt1koYPl+66iyQaAAA0SCQkJRYu9LvXr7wiTZwo3XOPd4Ep6+KLpQMP9IJ2mzbVf5xILn/5i/TGG56MtG1bv8fu0UN6913ppJOkn/1MuvBCadu2+o0BAACgCiQkkvTqq9Ihh0irV0v//rd/catIZqZ0333SihXSb35TfzEi+WzYIP3iF/67NWpUNDG0aCE995xPW/3HP0rHHutduQAAABqI9E5IQvCWkOOP964t06b5QPaqHHqodMEF0h13SLNn13mYSFLXXedJ7n33SRkR/lfLyJDGj/fWmvff95njPv44ungAAADipG9Csm2bdNFF0k9+4sXl3n1X2nffxN//+99Le+/t4wPom4+yPvzQk92LL5YGDow6GnfOOdKbb0rbt0uDB0vPPx91RAAAAIklJGY23Mzmm9lCM7uqgm2OMrOPzGyOmb1Ru2HWsrVrpe9+V5o0Sbr6av9i1qJF9fbRurWPC3jrLenPf66bOJGcios9UW3TxidBaEgOPthbSfr2lU4/3eMjoQYAABGqMiExs0xJ90o6QVKupHPMLLfMNntLuk/SqSGEvpJG1EGstWPWLP9S9t//SgUFXuiwpt1pfvxj7771y19KX31Vu3EieT38sP9+TZggtWwZdTS76thRmjJFys+Xrr1WGjlSKiyMOioAAJCmEvkmfoikhSGERSGEbZL+Kum0MtuMlPRsCGGpJIUQVtdumLXkhRe8q8o333jLxsiRu7e/jAwvdLd2rTR2bO3EiOS2bp30619LRxwhnXde1NFUrFkzb9m76SbpySelI4/0wo0AAAD1LJGEpJOkZXGvl8eWxcuR1NLMppjZDDP7QW0FWCtC8DEfp58u9ekjTZ/urSS14YADvGbJ/fdLM2bUzj7T2VdfSbff7l2Khg2TVq6MOqLqufpqaf16H8huFnU0lTPz5OmFF6T586XvfMdb+z77LOrIAABAGkkkISnvW1XZTudZkgZKOknS8ZKuNbOcXXZkNtrMppvZ9DX1NfXoli3eNeWaa3xQ7xtveJeV2vTb30rt2vm4geLi2t13upg+3bvAdeok/fznPqZn6lQpL8/XJYOpU6UHH5Quv1zq1y/qaBJ3yike+7Bhngz27OnFFCdPlnbsiDo6AACQ4hJJSJZL6hL3urOkFeVs81II4esQwlpJb0oaUHZHIYRJIYS8EEJe2/ooEvf5594V5a9/9RaSxx/3riq1ba+9pFtv9WmDH3yw9vefqgoLfbzFwQf746mnvJvThx/6GIx33/XilEccIT3xRNTRVm7HDp9Ra599pOuvjzqa6svNlf72N2npUq+vM2uWdNppXlxx/Hhp1aqoIwQAACkqkYTkfUm9zKyHmTWWdLakyWW2eUHSEWaWZWbZkgZJmle7oVbTtGn+JfeTT3wWrauuqtsuNCNHeg2Tq66i8FxVFizwVpBOnbxVpLBQuvtuTyAfeMC7wUnehaikbsbIkdKYMQ23Ber++z2Ruv326s/Y1pB07Oj1U5YskZ59Vtp/fx8f1aWL9P3vewsjs3IBAIBaVGVCEkIoknSZpJflScZTIYQ5ZnaRmV0U22aepJckfSxpmqQHQwjRVQwsKPCWkaZNpffek049te6PaSbde6+0aZMnJdhZUZF/wT3uOP+Se/fdXpByyhQvLnnZZd7SVFbbttJ//uOVzn/3O+mMM/wzbki++MK/tB93nDSi4U4wVy1ZWT7m6t//9vElP/mJ9MornnT36+c1VjZsiDpKAACQAhKa7zaE8GIIISeEsF8IYXxs2cQQwsS4bSaEEHJDCP1CCHfUVcCVKi72QcXnnuvT8U6bVr99+XNz/c7/Qw95dyNIK1ZI48ZJ3btL3/uef7m98UZp2TLvSjd0aNUtV40be82Yu+6S/vEPnyntf/+rl/AT8stf+lile+5p+APZayInR7rtNm/BeughKTvbE5ROnaQLL5Q++ijqCNNWTWtEmVkXM3vdzObFlv+sfiMHAKBU6lRq37hR+r//82lML7zQ7+a2aVP/cVx7rdS5s48nKCqq/+M3BCFIr70mnXmm1LWrj6no189nc1q0yLtedehQvX2a+Zfgf/3Lp6c95BDvPhS1N97w6XN/+Uv/4p7KsrOl88/3bnTTpklnnSU99ph04IGeJD7+uLR1a9RRpo3drBFVJOkXIYQ+kg6VdGnZ9wIAUF9SIyFZtMi/EL34onebmjjR76pHoXlz6Y47pI8/9ljSyfr10p13+tTKw4ZJr78uXXGF9Omn0ksvede5rKzdO8Zxx/mMUK1bS8ce6y0nUdm+3ad87tbNZ3FLJwcf7K0ln3/urSdr1/qEBJ07+1TCixZFHWE6qHGNqBDCyhDCB7F/b5J3xy07nTsAAPUi+ROSKVP8bvmKFdLLL/vUu1E74wyfNvXaa5OvjkZNzJjhYzw6dvQpb1u2lB591FsyJkzwaWRrU06Oz8J17LHeGvaTn0TTGnXnndKcOT4eJju7/o/fELRq5UnnJ5/4eJOhQ33GuZ49pRNPlP7+d6YOrju1UiPKzLpLOlDS1DqKEwCASiV3QjJxot8xb9fOu5AMGxZ1RM7Mv6Ru2yb94hdRR1M3tmyRHnlEGjTIa4U88YSP3fngA59I4Ac/qJsplkvsvbePJ/n5z33sxvDh0pdf1t3xylq+XLrhBq/hccop9XfchiojwxPEZ57xGbquu87Hlpx6qrTffj4hwRdfRB1lqtntGlFm1lzSM5IuDyFs3OUAUdSOAgCkneRMSEq6ylx8sfTd7/oX4Nq+C7+7evb0ritPPOHjKVLFwoXSlVf6gObzz/exO3fd5V13Jk3y8QT1JTPT78Y//LD01lueHM2rp9mmr7jC7/zfeWf9HC+ZdOrkydqSJdLTT3tCMmaMTx08cqT/rJg6uDbsVo0oM2skT0YKQgjPlneAeq8dBQBIS8mXkKxb59PF3nefDySePLn86WIbgquu8sJyl17qrSXJqqjIa7kcf7zUq5d/CT/2WB8jMneud5nae+/o4vvRjzyWjRt9drUXX6zb473yin/RHjvWf74oX6NGPrPaq696onjJJf6zOfJIrzFz333+M0NN1bhGlJmZpD9JmhdCuK1eowYAoIzkSkjmzvW74O+842MU/vAHv0veUDVr5t2JPvnEC+Ylm/Xrpd/+1r90n366j5cYN86reT/1lNekaCjT3A4e7LM/7buvdPLJ0i231M1d+K1bPcHMyfGWIiSmd2+f7OHzz6UHH/RJJy691FtTLr7YJ4FAtexmjaghks6TdExsSuCPzOzESE4EAJD2LETUdSIvLy9Mnz498TeEIB1+uPTZZ363/tBD6y642nb66X5Xfd48nwY3GXzyiff///RTH6dzySX+RX93Z8mqa19/7S0mTz8t/fCHPs6oadPa2/9vf+vjI155xT8X1EwInkDef7/Xo9m6VRoyxH/Pvvc9qUmTqCOskpnNCCHkRR1Hfan2NRsAGpB0u2Ynm+RpITHzOgfTpydXMiL5neEQfAaqZPDSS/4Zb9jg/f1fecVrvDT0ZESS9thDevJJH8Pw6KPS0UdLq1bVzr7/9z8fnD1iBMnI7jLz2fEefthbTW691Qe95+f7WJOrr5YWL446SgAAUA+SJyGRvOtQ585RR1F93br5XfXnnqv78Q27IwSvKXHSSf5Zv/++t0olm4wML8b49NPeFejgg332r9310596UpaM3e8aslatfLa0+fM9+R0yxLtjlnS/++c/mToYAIAUllwJSTL7+c+9H/1PfuJT5jY033wj/fjHPk3x6adLb7+dPN3LKvK97/l4IzNPrJ56qub7mjzZpxm+4QYf94Dal5HhLU/PPeetI2PHeo2bk0/2Wetuvlli6lkAAFIOCUl9adzYK7cvWuRfrBqSL76QjjnG64rccIN/cd9jj6ijqh0HHOAtPQceKH3/+95SVVxcvX0UFnrrSN++/oy616XLzhModO/us9Z17uzdut55h6mDAaCeFRT45Tgjw58LCqKOCKmChKQ+HXOMdPbZ0k03+eD8huDDD71L04cfSn/7m3d1ykixX4v27b0WzPnn+6D0ESOkzZsTf//48V5T4777fCpb1J9Gjfzn9frrPsvbhRd6S9Xhh0sDBvikBZs2RR0lAKS8ggJp9Gj/cxiCP48eTVKC2pFi3zyTwK23emvJZZdFf4f36adLx4i884505pnRxlOXmjSR/vQnHyPz/PM+TmHJkqrfN3++NGGCV54/8si6jxMVy831IpwrVngRzsxMnzK4UyefQnj27KgjBICUNWaMdxiIV1joy4HdRUJS3zp29K4oL73kfeWjUFzsXbNGjPC7zCVdmlKdmVdY/+c/PRk5+GAfK1OREPyLbna2D7JGw7DHHtIFF/hEBe+952Oe/vQnqX9/TxqfeCK5C5ECQAO0dGn1lgPVQUIShcsu80rVl1/udTPq09dfS2edJf3mN6UVztu3r98YojZ8uDR1qleXP+YY/zJbnqee8irj48en32eUDMx8eupHH/WpgydM8OeRI30MypgxibWCAQCqVNE8N8k+/w0aBhKSKGRl+XiEZct8TEN9WbrUu2g995x3HXvooaQoQFcn9t/fk5KjjpJGjfLksKiodP2mTT4z2kEHSRddFFmYSFDr1tKVV3ohz5I6Ojfd5FMHn3qq9K9/VX8yAwDAt8aP9w4D8bKzfTmwu0hIojJkiA+yvvVWae7cuj/eO+94F6VFi3xQ8M9/7neY01nLll4X5mc/k+680+uvfPWVr7vhBmnlSk8cMzMjDRPVkJEhHX+89MILXsjy6quladOkE0+UevXyrndr10YdJQAknfx8H77XrZt/fejWzV/n50cdGVIBCUmUbr5ZatHCxynU5QD3hx/2iuV77umtAiecUHfHSjZZWdIdd0gPPujd1w491Af733mnj1MYNCjqCFFTXbtKN97oLYN//at34/r1r33q4PPO8/EnUU8sAQBJJD/fy0QVF/szyQhqCwlJlNq2lX73O2nKFB+IW9uKirwl5Mc/loYO9TvFvXvX/nFSwf/7fz5e5KuvfLD/3nv7zwbJr3Fjr0EzZYrPxHXBBd6CMniwT+YweXLUEQIAkNZISKJ2wQXeleoXv5A2bKi9/a5f7xWub7/di/n961/eRQkVO+IIn3HsxBOlP/7RxyUgtfTtK919t08d/MADvqykmx4AAIhEVtQBpL3MTB+ncMghXkX8zjt3f58LFvhA3kWLvIPnBRfs/j7TRbduPi0wUlvz5l7R64IL6LYFAEDEaCFpCPLyfCane+6RPvpo9/b1yis+7mHdOuk//yEZASpj5gPhAQBAZPhL3FCMH+9dhC65pGbTk4bgrSsnnOCDd99/n8riAAAAaPBISBqKli29sNt77/msWNWxbZu3hFx+uXTKKdK770rdu9dJmAAAAEBtIiFpSH7wAy9c+Otfe5erRKxeLQ0b5tXGx4yRnn3W+8cDAAAASYCEpCEx8wHu69dL11xT9fYzZ/oMXdOn+7TBN95If3gAAAAkFWbZamj69/fK4bff7vVDKirM99xzXtxt772lt97ygfEAAADYLTNmzGiXlZX1oKR+4uZ9bSiWNLuoqGjUwIEDV5e3AQlJQ3TDDV5Z+pJLvJhhZmbpuhC8JeS663yq4Oefl/bZJ7JQAQAAUklWVtaDHTp06NO2bduvMjIymBt+NxUXF9uaNWtyV61a9aCkU8vbhqyvIWrRQrrtNumDD6SJE0uXFxZKZ5/tyci550pvvEEyAgAAULv6tW3bdiPJSO3IyMgIbdu23SBvcSp/m3qMB9Vx1lnSscf6QPUvvpCWLfNK4n/7m3TzzdJjj0lNm0YdJQAAQKrJIBmpXbHPs8K8gy5bDZWZF0rs399bQ2bN8haSyZOlk0+OOjoAAACgVtBC0pDtv7/0y196xfU99vAaJSQjAAAAKWvVqlWZvXv3zu3du3dumzZtBrRr1+47Ja+3bt1qiezjzDPP7D5z5swmlW3z+9//vu3999/fqnai3j20kDR0117rlddHjPBK7gAAAGgwJk5Uq3Hj1GnVKjXu0EHbrrtOn190kb6s6f46dOiw45NPPpkrST//+c87Nm/efMe4ceO+iN+muLhYIQRlxk98FOfpp59eXNVxrr766jU1jbG20ULS0DVtKl10EckIAABAAzNxolpdcYW6rVypxiFIK1eq8RVXqNvEiar1lofZs2c36dWrV9+RI0d27du3b+7SpUsbnXPOOd369evXp2fPnn2vvPLKb2c6Gjhw4P7vvvtus+3bt6tFixYHXHLJJZ3233//3AMOOKD3559/niVJP/3pTzuOGzeuXcn2l1xySaf+/fv36d69e79///vfe0jSxo0bM44//vj99t9//9xTTjmlR79+/fq8++67zWr73EhIAAAAgBoYN06dtm7d+fv01q3KGDdOnerieJ999lnTCy+8cO28efPm9ujRY/sdd9yxfPbs2fPmzZs35/XXX99zxowZu8x4tHnz5syjjjpq0/z58+fm5eVtvvfee9uUt+8QgmbNmjVv/Pjxy8aNG9dRkm666aZ27dq12z5//vy511xzzap58+Zl18V5kZAAAAAANbBqlRpXZ/nu6tKlyzdDhw4tLHn90EMPtcrNze3Tt2/f3EWLFjX9+OOPd2m9aNq0afFZZ521UZIGDhxYuHjx4nJjGzFixHpJGjx4cOHy5csbS9J7773XPD8//0tJOuyww7bst99+W+rivEhIAAAAgBro0EHbqrN8dzVr1qy45N+zZs1q8sADD7R/8803FyxYsGDukUceuXHLli27DHrPysr6dgrjzMzMsGPHjnIHxjdt2rS47DYh1M/sxyQkAAAAQA1cd50+b9pUxfHLmjZV8XXX6fO6Pvb69esz99hjjx0tW7bcsWTJkkZvvvnmnrV9jMMOO2zzE0880VKSpk2b1mzRokW1Pn5EYpYtAAAAoEZKZtOqzVm2EjVkyJDCXr16bc3JyenbtWvXbwYOHLi5to9x1VVXrR4xYkSPnJyc3P79+xf27NlzS6tWrXbU9nGsvppiysrLywvTp0+P5NgAsLvMbEYIIS/qOOoL12wAyaw61+yZM2cuHjBgwNq6jikZbN++Xdu3b7fs7Owwa9asJsOHD89ZvHjxrEaNGlV7XzNnzmwzYMCA7uWto4UEAAAAwC42bNiQOXTo0JyioiILIejuu+9eUpNkpCoJJSRmNlzSnZIyJT0YQripzPqjJL0g6X+xRc+GEMbVYpwAAAAA6lGbNm12zJkzZ15dH6fKhMTMMiXdK+k4ScslvW9mk0MIc8ts+lYI4eQ6iBEAAABAikpklq1DJC0MISwKIWyT9FdJp9VtWAAAAADSQSIJSSdJy+JeL48tK+swM5tpZv8ys77l7cjMRpvZdDObvmbNmhqECwAAACCVJJKQlFc8pezUXB9I6hZCGCDpbknPl7ejEMKkEEJeCCGvbdu21YsUAAAAQMpJJCFZLqlL3OvOklbEbxBC2BhC2Bz794uSGplZm1qLEgAAAEgDhxxyyP7PPPPMTkUOx40b1+7cc8/tWtF7srOzD5SkxYsXNxo+fPi+Fe33zTffzK7s2OPGjWu3adOmb/ODoUOH9ly7dm1m9c6g+hJJSN6X1MvMephZY0lnS5ocv4GZdTAzi/37kNh+19V2sAAAAEAqGzFixLonnniiVfyyZ555ptW5555bZbHF7t27b3/ppZcW1fTYDzzwQPvNmzd/mx+88cYbC9u0aVPrhRDLqjIhCSEUSbpM0suS5kl6KoQwx8wuMrOLYpudKWm2mc2UdJeks0NUFRcBAACAJHXeeed99eqrr+61ZcsWk6T58+c3Xr16daNBgwYVHnbYYTm5ubl9cnJych9//PG9y753/vz5jXv16tVXkjZv3mwnn3zyvjk5ObknnXTSvlu3bv12GEZ+fn7Xfv369enZs2ffK664oqMk3Xjjje1Wr17daOjQoTmDBg3KkaROnTr1X7lyZZYk3XDDDe179erVt1evXn3HjRvXruR4++67b9+zzz67W8+ePfsOGTKk1+bNm8sb7lGphOqQxLphvVhm2cS4f98j6Z7qHhwAUHNV1YiKbXOUpDskNZK0NoQwNNH3AkDa+/GPu2j27Eq7OVVbv36FeuihZRWt7tChw44BAwZ8/cwzz+x17rnnrn/00UdbnXrqqV81b968+J///OfCVq1aFa9cuTJr0KBBvUeOHLk+I6P89oVbbrmlXbNmzYoXLFgwd+rUqc2GDBmSW7Lutttu+7x9+/Y7ioqKNHjw4P2nTp3abOzYsavvv//+9m+88caCffbZpyh+X2+99Vb2X/7yl9YzZsyYF0LQwIED+wwbNmxTmzZtdixdurTp448/vmjw4MFLTjzxxH0fe+yxlpdcckmVrTnxEumyBQBoYOJqRJ0gKVfSOWaWW2abvSXdJ+nUEEJfSSMSfS8AIDpnnXXWl08++WRLSXr22WdbnXfeeV8WFxfb5Zdf3jknJyf36KOPzlm9enXj5cuXV9i48Pbbbzc/77zz1knSoEGDtuTk5BSWrHv00Udb5ebm9snNzc399NNPm86cObNpZfFMmTKl+Yknnrh+zz33LN5rr72KTzrppK9ef/31FpLUqVOnbwYPHrxFkg488MDCxYsXN6nu+SbUQgIAaHC+rRElSWZWUiMqvmjtSEnPhhCWSlIIYXU13gsAqKQloy7l5+evHzt2bJe33347e+vWrRmHH3544V133dV63bp1WbNmzZrXpEmT0KlTp/5btmyptHEhNsR7J5988knje+65p/2MGTPmtW3bdsf3vve97lu3bq10P5WNxGjcuPG3KzMzM0NVMZWHFhIASE6J1IjKkdTSzKaY2Qwz+0E13kvtKACIyF577VV86KGHbho1alT3M84440tJ2rBhQ2abNm22N2nSJPz9739vsWLFisaV7ePwww/f/Pjjj7eSpPfff7/pggULsiXpq6++ymzWrFlxq1atdixbtixrypQpe5W8Z4899tixYcOGXfKDY445ZvOLL76496ZNmzI2btyY8eKLL7Y8+uijN9XW+dJCAgDJKZEaUVmSBkoaJqmZpPfM7L8JvlchhEmSJklSXl4eE5UAQD06++yzv/zhD3+43xNPPLFIkkaNGvXlCSec0LNffC5zbgAABptJREFUv359+vbtW9ijR4+tlb3/yiuvXH322Wf3yMnJye3bt29h//79v5akww47bEu/fv0Ke/Xq1bdr167fDBw4cHPJe374wx+uPeGEE3q1a9du+9SpUxeULD/88MMLR44cue6ggw7qI0nnnXfemiFDhmyZP39+pUlRoiyqybDy8vLC9OnTIzk2AOwuM5sRQsiL8PiHSbohhHB87PXVkhRC+H3cNldJahpCuCH2+k+SXpK3iFT63rK4ZgNIZtW5Zs+cOXPxgAED1tZ1TOlm5syZbQYMGNC9vHV02QKA5FRljShJL0g6wsyyzCxb0iD59O2JvBcAgHpBly0ASEIhhCIzK6kRlSnpoZIaUbH1E0MI88zsJUkfSyqWT+87W5LKe28kJwIASHskJACQpKqqERV7PUHShETeCwCQJBUXFxdbRkYGY+dqSXFxsclvjJWLLlsAAABAqdlr1qzZK/YlGrupuLjY1qxZs5ek2RVtQwsJAAAAEFNUVDRq1apVD65ataqfuHlfG4olzS4qKhpV0QZJk5AUFEhjxkhLl0pdu0rjx0v5+VFHBQAAgLpWn98DBw4cuFrSqXWzd5QnKRKSggJp9GipMFbwfskSfy2RlAAAAKQyvgemvqRohhozpvSXsERhoS8HAABA6uJ7YOpLioRk6dLqLQcAAEBq4Htg6kuKhKRr1+otBwAAQGrge2DqS4qEZPx4KTt752XZ2b4cAAAAqYvvgakvKRKS/Hxp0iSpWzfJzJ8nTWIgEwAAQKrje2DqS4pZtiT/peMXDwAAIP3wPTC1JUULCQAAAIDUREICAAAAIDIkJAAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICAAAAIDIkJAAAAAAiQ0ICAAAAIDIWQojmwGZrJC2J5OA110bS2qiDqGfpeM5Sep53Op6zVPPz7hZCaFvbwTRUXLOTSjqeN+ecPrhmp6DIEpJkZGbTQwh5UcdRn9LxnKX0PO90PGcpfc87HaTrzzYdz5tzTh/pet6pji5bAAAAACJDQgIAAAAgMiQk1TMp6gAikI7nLKXneafjOUvpe97pIF1/tul43pxz+kjX805pjCEBAAAAEBlaSAAAAABEhoSkCmbWxcxeN7N5ZjbHzH4WdUz/v717B5WjjqM4/j3kKjER0SCKJkUUxAcBiVhEAyLGQlCMjWChBLEUHyEgamNrIaKVTXwEDBG5BhQRUWJhl8JEMBhBUIlXr96A+MBGxWOxI4Q0N+Lu/Mj+zqfZ2X91lh3O7G92dmdMktZIOirp3eosY5B0oaRFSV8M7/lN1ZnGIGn3sH8fk3RA0trqTNMm6RVJK5KOnbK2QdKHkr4cHi+qzBjT0bm3u3U29OztDp0N6e1OMpCs7i9gj+1rgW3Aw5KuK840pseA49UhRvQi8L7ta4DrafDaJW0EHgVutL0FWAPcV5tqJl4D7jht7UngkO2rgEPD8zj7de7tbp0NzXq7UWdDeruNDCSrsL1s+8iw/RuTottYm2ockjYBdwJ7q7OMQdIFwC3AywC2/7D9c22q0SwA50laANYB3xfnmTrbHwM/nba8E9g3bO8D7hk1VMxE197u1tnQurfnvrMhvd1JBpL/QNJmYCtwuDbJaF4AngD+rg4ykiuBk8CrwyUPeyWtrw41a7a/A54DTgDLwC+2P6hNNZpLbS/D5EMscElxnpiyZr3drbOhYW8372xIb8+lDCRnSNL5wFvA47Z/rc4za5LuAlZsf1KdZUQLwA3AS7a3Ar/T4Kvg4frbncAVwOXAekn316aK+P869XbTzoaGvZ3OjnmUgeQMSDqHyUFtv+2D1XlGsh24W9I3wBvAbZJer400c0vAku1/z6QuMjnQzbvbga9tn7T9J3AQuLk401h+lHQZwPC4UpwnpqRhb3fsbOjZ2507G9LbcykDySokicm1qcdtP1+dZyy2n7K9yfZmJj+W+8j2XJ+Bsf0D8K2kq4elHcDnhZHGcgLYJmndsL/vYM5/FHqKd4Bdw/Yu4O3CLDElHXu7Y2dD297u3NmQ3p5LC9UBzgLbgQeAzyR9Oqw9bfu9wkwxO48A+yWdC3wFPFicZ+ZsH5a0CBxh8u9ER5nDO+FKOgDcClwsaQl4BngWeFPSQ0wO8vfWJYwpSm/30qq3u3Q2pLc7yZ3aIyIiIiKiTC7ZioiIiIiIMhlIIiIiIiKiTAaSiIiIiIgok4EkIiIiIiLKZCCJiIiIiIgyGUgiIiIiIqJMBpKIiIiIiCiTgSQiIiIiIsr8A/4s/hUW5fNWAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model = models.Sequential()\n", "model.add(layers.Embedding(10000, 32))\n", "model.add(layers.GRU(32))\n", "model.add(layers.Dense(1, activation='sigmoid'))\n", "\n", "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n", "history = model.fit(x_train, y_train, epochs=10, batch_size=128, \n", " validation_split=0.2, verbose=1)\n", "\n", "acc = history.history['acc']\n", "val_acc = history.history['val_acc']\n", "loss = history.history['loss']\n", "val_loss = history.history['val_loss']\n", "epochs = range(1, len(acc) + 1)\n", "\n", "plt.figure(figsize=(12, 5))\n", "plt.subplot(1, 2, 1)\n", "plt.plot(epochs, acc, 'bo', label='Training')\n", "plt.plot(epochs, val_acc, 'r', label='Validation')\n", "plt.title('Training and validation accuracy')\n", "\n", "plt.subplot(1, 2, 2)\n", "plt.plot(epochs, loss, 'bo', label='Training')\n", "plt.plot(epochs, val_loss, 'r', label='Validation')\n", "plt.title('Training and validation loss')\n", "plt.legend(bbox_to_anchor=(1.02, 0.2), loc=2, borderaxespad=0.5)\n", "plt.show()\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The validation accuracy is about 60%." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Regularization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We consider a weather timeseries dataset recorded at the Weather Station at the Max-Planck-Institute for Biogeochemistry in Jena, Germany. In this dataset, 14 different quantities (such air temperature, atmospheric pressure, humidity, wind direction, etc.) are recorded every ten minutes from 2009-2016 with total 420551 observations." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fname = os.path.join(base_dir, 'jena_climate_2009_2016.csv')\n", "\n", "f = open(fname, encoding=\"utf8\")\n", "data = f.read()\n", "f.close()\n", "\n", "lines = data.split('\\n')\n", "header = lines[0].split(',')\n", "lines = lines[1:]\n", "\n", "float_data = np.zeros((len(lines), len(header) - 1))\n", "for i, line in enumerate(lines):\n", " values = [float(x) for x in line.split(',')[1:]]\n", " float_data[i, :] = values\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The plot of temperature (in degrees Celsius) over time" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAExCAYAAABh3lQ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydd3wU1RbHf5NCQui9Q+i9SZciCChFxfoE9dnrs3eevYO9+xQLdrGLgqCA9N57D6GThBACAdLn/bE7yezu9Lkzc2f3fD8fJTs7O3N29pZzzz1FEEURBEEQBEEQBEEEiPNaAIIgCIIgCILgCVKQCYIgCIIgCEIGKcgEQRAEQRAEIYMUZIIgCIIgCIKQQQoyQRAEQRAEQcggBZkgCIIgCIIgZDBTkAVBiBcEYa0gCNOCr2sKgjBLEISdwX9rsLoXQRAEQRAEQTgFSwvyvQC2yl6PBzBHFMXWAOYEXxMEQRAEQRAE1wgsCoUIgtAYwBcAXgTwgCiKFwiCsB3AYFEUDwuC0ADAPFEU22pdp3bt2mJqaqpteQiCIAiCIAhCi9WrVx8VRbGO0nsJjO7xFoBHAFSRHasniuJhAAgqyXWVPigIwq0AbgWApk2bYtWqVYxEIgiCIAiCIAhlBEHYq/aebRcLQRAuAJApiuJqK58XRXGSKIo9RVHsWaeOohJPEARBEARBEK7BwoLcH8BFgiCMApAMoKogCF8DyBAEoYHMxSKTwb0IgiAIgiAIwlFsW5BFUfyvKIqNRVFMBTAWwD+iKF4D4HcA1wVPuw7AVLv3IgiCIAiCIAincTIP8kQAwwVB2AlgePA1QRAEQRAEQXANqyA9AIAoivMAzAv+nQ1gKMvrEwRBEARBEITTUCU9giAIgiAIgpBBCjJBEARBEARByCAFmSAIgiAIgiBkkIJMEARBEARBEDJIQSYIgiAIwnfkninC2n05XotBRCmkIBMEQRAE4Tuun7wCl3ywBCWloteiEFEIKcgEQRAEQfiOtfuOAwAW7MzyWBIiGiEFmSAIgiAI35KWdcprEYgohBRkgiAIgiB8i6BwbNPBXNzz3VpyvyAsQwoyQRAEQRBRxe1fr8bv6w/h0PEzXotC+BRSkAmCIBjz9bK9OJpX4LUYBBETaNmIRTIgExYhBZkgCIIhGw/k4onfNuHGz1d6LQpBxCyCkt8FQZiAFGSCIAiGvD1nBwBgw4FcjyXhk9Tx0/HmrB1ei0EQBKEJKcgEQRAMoZggfd6es9NrEYgYQdR0wCAIdUhBJqKOqesOYuq6g16LQRAEQXiEoJjbgiCMk+C1AATBmnunrAMAjOnWyGNJCIIgCILwI2RBJgiCYEis263W7z+OI7n5uuet3ZfjgjRErLBo51EsS8sue73v2GkPpeGL0lIRS3YdNf25dfuPx3QeaVKQCYIgCGaMeX8xBr0yV/e8Sz5Y4oI0RKxwzafLMXbSsojjlOYN+GzxHlz1yXLM3pJh+DPr9x/Hxe8vxtuzYzeglhRkImo5kpuP3Vl5XotBEDFHYUmp1yIQMURhMbU3LfYcDZTiPnxCf2dHIiN47uQl6U6I5AtIQSailr4T5mDo6/O9FoOIMchgRRjhrm/X4B2dbB5nCksgkglUl5dnbjN03u/rD+GFaVscloY/CiwsIKRWdzK/mK0wPoIUZCKqKI1hfymCIPzDtA2H8YZGPujME/lo/9RMfLJwj4tSRR/yGeGe79bik0Wx9zx/Wn0AAM2PZiEFmYgq5m7P9FqEqGVX5kkU0da5LrEepEew4eDxMwCAaRsPeyyJv3n97+1YmX7MazG4IJYD7qxACnIU8PPqA5i8OPZWxUpY2Uoi9DmcewbD3liA5/6Ive1JLUpLRZwujN0tSMI58osCYxm5WJhH7pM8bcNhXPHhUg+l4Qcqv20OUpCjgAd/XI9nSXEhHCTnVBEAkCUmjJf+3IoOT/2F/KISr0Uhoow3Zm0HQCXLrXCmkPqjEqQfm4MUZCKqIGML4Sbfr9oPAJi9tTx9EllpCBbkninyWgRfcvaEObrnHAq6rxCEFqQgEwRBWESK8L7/+3Vlx2iRRhDecSg3H6JOLpm8gth0ixJo9W4KUpAJgiBsIsg2L2kOIgjC7xzMISs7KcgEQRAG+GP9IWw6SP6gBOE26/Yfx1sxXNHNC56LwXzR4SR4LQBBsERva40grHL3d2sBAOkTR2ueRy4W6lBGBsIKF7+/GABw37A2TK4Xq82QdrfMQRZkImpYuDMLd3271msxiBiHJiF1/tp8JOT1L2sOeCQJn2w8kItbv1yFYso3botYVYD1sPNcxv+8AT1fmM1OGB9ACjIRNTw9dbPXIhAEocGR3PyQ1w/8sN4jSfjk3ilr8feWDOw9dtprUQgihCkr9+NoXgFOxVCAIynIBEHEHLlnigwV+Nh/7DRSx0/H8rRs7RNlVmMqVqWMKIp4hvK1a0JNhw2xmqVib/YpTTcmaXdr/7HTlp/R4dzYCd6zrSALgpAsCMIKQRDWC4KwWRCEZ4PHawqCMEsQhJ3Bf2vYF5cg1Ek7esprEaKWaKsW1/XZvzHg5bm65y0NKsY/rjbuCvDPNip3rsTJGFValIhVBc4umSfz9U8C8FkMVpbddDAX57w6D58u0v/uA1+Ziys/WopeL87GA7IUlUaIJfcVFhbkAgDniqLYFUA3ACMEQegLYDyAOaIotgYwJ/iaYMw3y/d6LQIRA1z72QqvRWDOsVOFzK5Fbsf6xNLEqkeOTtubtz3LJUn8xbp9x70WgVv2B91yVqXnqJ4j74ObD51A1skC/LL2oNOi+RbbCrIYIC/4MjH4nwhgDIAvgse/AHCx3XsRkTz+66ayv9OPnkLq+OlYsYfKARNsOR2rpVsNKnUFxdpBVUUlpSjUOYcgJJ6ftgU7MvL0TySIILT+ZA8TH2RBEOIFQVgHIBPALFEUlwOoJ4riYQAI/ltX5bO3CoKwShCEVVlZtGq2w+LdRwEAv8bAijD3TBEyThjbbiMIrzn39Xlo88QMr8VwHOqT9lDzH805VYgvl6ZTmjyD0GMiWMBEQRZFsUQUxW4AGgPoLQhCJxOfnSSKYk9RFHvWqVOHhTgxSywNCv0n/oM+L80xdG5BcYxaPxHYVZjw51aaWFUoLRXx6l/b1BU7Rr4T+4/FRmDLAUbVt3JPFzG5jt9Q66UP/LAOT03djM2HTrgqD+EfjAxVaikoiyitoCJMs1iIongcwDwAIwBkCILQAACC/1LkCsGM8CAXM0qwKIpIHT8dr/+9nbVY3HHLl6vw0YI0WwGM2XkFDCXii7X7c/D+3N144AeVQBVaV7jO0t3Z6Prc35izNcNrUVxHbR17LLhgiHVFhrqjOkaejVr7OnEmNhekerDIYlFHEITqwb8rAhgGYBuA3wFcFzztOgBT7d6LIMI5mR/o2A+ayKcqDRLv/rPLCZG4ophBzrEeUZwcvrgk8HzIP5gV9tvbuv2BQKwV6RRLUQbtAAEAsvMKLefhjbZMPGo4Xagovyh2xkoWFuQGAOYKgrABwEoEfJCnAZgIYLggCDsBDA++Jlwglip5nSoIWI6nbTiseo4QtvlUVBo7HZwwRngbscJTUzfpn0QQYWTK3HtEspFq8tivG3Hhu4ssfXby4nS2wsQob83e4bUIrsEii8UGURS7i6LYRRTFTqIoPhc8ni2K4lBRFFsH/yVzgEtEu7FBvs1oZUL5bvk+luL4ktOFxbj9q9Wmk74XlpTinTk7kV8UXX7dIkTM2HgYJSoWdyPq85dLKeUiYR75Lk+0j90ssOouJu/btBAxhzyGJZbymVMlPcJ3fGYgEboWp2IwZdn6/cdRKpsgZm46gpmbj+CVmeb8sNOyTuGNWTvQ7smZGPr6PMZSesfK9Bzc8c0afLEk3WtRiBhATRHWU5CFWNoedJBoTFtpZHF1IMdaCfNnY7QCJinIPmbTwVzF49E+hh6ngALTPPDDeny0IK3stTSY/rr2ICbO2Gbpmruz/F+5MHxO+WRhmuJ5J/OtWU2mrNgXddZ2q+xRsfztyz4dU+Vr7fDpoj14d85Or8XwPR/M3e21CJ7w8UJl45LegiFWC5IleC0AYY43Z+3A5kMnUCMl0VT522jFyKo52hcMRtmRcbLsb/lj+3D+bowf2S7k3AkztqJ7kxoY0am+S9LxwaFc5XRvB44bs7yEpycb/8tGbDtyUuXs2GL2FuWsFINeDZT8Tp84GgBwIj+2FsDy8UkvHeMf6w8BAO4e2tpJkTwlv6gEJ84UoW7VZM3ztKyhP6zaH3FMPg0UR2Ecip15buArc8v6nxKx6vpDFmSOuee7tRg3aVnIsbfn7MTsrRmKyvHx0+xK50YTx04V4u3ZO0NcDGIRrfEz62RoKreP5qfh9q9XOytQFLL1SGSe2myGJa39zFfLjFmh/jcvYN3Lzouu53bnt2uQOn66phKsNkTF0sh17acr0NtAjvuME+rpJ/UsotFoMzGqxGpZg3dl5umWQY8lSEH2EL1qcL+vP4SladmGr/fa37ERXWp2NTv+5w14c/YOLNtj/FlGJbJZ4dDx0C3tu75d47IwfKDXliiYxxybDqoXssg16RoVban3pgcz7ezKzFO19h1RmQ82HFB2p4tGjKf3o75phR9Wqe88D3tjPoa/uSDiuDyQdG+2/13rjEIKsoeYqQYHBHz1iFCMDJFSUJ4UxRxLVeXk31WeyuyNWaGLqRMqPrZnojCYxUmi0TJllC2HTuDp3zd7LQb3xM7o4ywvmwwwJoxxNK8Aq/fmqL6vZbmPNkhB9pDwanBa5BeVlPnqESYJzkiSgjhnW2wUdSwtFZFuc1H12WJ7GUP8jtRmYmhNZRmt3TCinPC2xCIHdyyyYo+5zLGvz4qNHVYWMTeX/W+J/YtEAaQg+wBRFNHuyZmGz6fhVpu1+457LYIrfB6WsszKwClVmotVyMXCO6L5yRcU28vlThCsifUYHSUoi4UPMGNpjjWMuEvE6gTEInVWtGcAYd02KE8toYcIEZ9RVTfCI9bvVzYQrd2v7lYRq5CC7AOmrIhMWRPLWFVqYl130fIrI9gQS/7t4aj1y5u/WEmL/DBO0fPwlGjvpacLi/HKzO14ZERbpFQwpuaVRFdMLBPIxcID5m7PxPWTV2ieY2ei3ZFxElsOqUeT+518k4FjMayzhKBWqAEAth4+oWhZMLqmuOPr1XjGhwFaOae0Myss2HkUgPHFlVm/SIkzhSW+8eEVRRGzt2SoluUOZ/bWTCxLs/ZcohX5mEQ+yO4TjU9cvkD9dOEefL4kHR8vMB5DsmhnlhNi+RpSkD3gps9XYt527cb4/LStZX+btZiuTM/BqHcWWpLND3yxtDyPoygCRQaXvlsPnyjLsUpEsmR3ZBq8VQatzjM2HYnwefYDE2Zs1XxfSs0l55YvV6merxQIlGlA8R338TJTGW28ZMamI7j5y1X4dFFo1UGlheiCHTTphuPEgr2kVIzp3QsiFCktW4mJNvHOP7ucEse3kIsFp3wtS6qfV0CptrR402B08gvTtZWhaENpbBz+xvyQACE5xQoLjfkGFJztPq4UZ9QKKmfWlgwcyc1H/Wralb4k9h/TzySyTsUv0GvSsvKQUiEh5LtKCv/BHG0f9wU7snDtZ9o7ZVpEo5VPCbt+8H+sP4S7v1uL289pGVERk4htNhzgc1zxC2RB9gAjw6F80Hxnzk7nhIkC9utM1GRXKWdnZh72qShseYXW/CLPf6s8sbwRZdCPhC82zFhm1MpX+4FzX5+PvhOsWbatuIzI20+s9Fu7Lhav/LUNQKBkPEHI25PeTjWhDSnInEK7ZcZYtfcYTlPASxl/bz6C1/7yLoF+5kl/JZEvptRGzGAxZg18JTZyvbNsdUoKdtbJAkxdd5DhXaKLnZl5XotA+ABSkDkllqft0lIRXy5NR36RvmvJ7+sOxUzhDyPc+tVqvDc34Eu23GTAGItgod99Niln+Uyh9xNH8wpNna/k4hONhC8k7LpYKAWQ3vTFStw7ZR2y8/zTvkVRxPPTtigeZ80BnV1Hv0NZUthACjKnxIr/nRIzNx/BU1M3G7KEGhk6YzG92fK0bGw8mGvqMyzS4MkDKKOZtKw8nMjXzoChhJ8CqZQCFLUI/2Yvz9xm6vOP/brR1PnRghNN4nDQrceKj71XnCkqwaeLIrMuKD0fu0Utzmpa3dbneacwRhabTkNBeh5gZECM5a1fKWfq8TP6CsgOHweIOcmVk5aZ/syxvELkF5Xgz43mFKNY5KYvVqFQJdhRizHvL8a9Q1tjaPt6DkjFlhemR1rz5LAeoX5YdSD0+j5aTJhF/t2kv0ZbzDwkX9dmnMgPca2Ihieo9B1mbj5i65qVkxNtfZ6IDciCzBG5Z4poy9ckdgOgonkSNsv3q/bjxelb8cAP670WxVOOn9Z3DbCiHAPAhgO5uP/7darvr97LT75geddo/fifqhW4JE5asKjb4cP5u/GXTUXJC9RcKjZbzF0vr974n2/W4KU/t/lyHlFz8SpVGKNPm8yFH3mvQPD73O3R5543Z1sGvnR4J6+guATnvjYv6tM4koLMEf0mzEGvF2czu966/ceROn46th6O3qIhdvlx9QH9kzgi51Qhnv1js+Hcz2Y54pNiFXbQ88vs9tysiGwcVpUXs+gVLvGKohIRkxeHbn+HqzOP/7qJ6T31ynZPnLENt321muk93WD0O4tCXrNcpPvJnSy/qAT/+WY1Dh7X9gdWUpDteoOJAN6YtQM3TF5p80reUlBcgjX7crD9yMmyBW1+kfPuFfuPnUHa0VO+LA5lBlKQOcLuqjicGZsCW+V+TfXihnF3ZTCQ7UxhCXJP86mcyHl++hZMXpyu6AbBQmme79O2YgYj3kv7c5xLV+fXPQstuU8XFuOMgaDaWEWrb36xJB0/2Vio+zVeZdaWDPy58Qhe+lM7P31xiYKC7NcvzZh7vluLSz9YEpJq0x2Cv0mU/w6kIDtAaamIb5fvM7wN+8K0LY5GG9uNko5mpCdz3lvz0fW5vz2VxQhS0I3S4mHKin22rx8LwR08T64899QjYe5Mcll/WLnfXWF8RrgVVP7q44V78NCPNtyaNNrzqvQc9HlptuvuL6bQafT/KGQpOnbKXIaUcDgeAkzx1+YMT+4rxoZ+TAqyE/y+/hAe+3VjWbotPT5ZtAdPTWW/VbEzI5Dr0a9utuGKzD3frcXmQ+YyMxhl/7Hybb6CYn9awljvQBDOcDK/GFd8uARnfPZ7hWdFcdrP0S+cKijm1uf39VnbkXGiABsOODNuskRt4VpcGrlot1sV1adToiZeVM3Tc4PyO6QgO4CU/inHxCrXiXQ80sr7qI9yYcqRK/br9h/H7+sPRfjvsebn1QfQ9omZSD96ytH7EN5hZEgvKC6N8Lllycr0HKxSCMjjeboRRWDJrqOKfvu/rTvkwP38ocaMfmch09gRM6zYc8zUPONHFPRjQoGPFzo3XoVzMkbyLFOaN8YcP12IzBMBhXTejkwcyc1H/WrJup+Lc2Gp8ufGwzhxpghjezd1/maMUSuPbJfwOXjGpkBU/I6Mk0itXcmRezqFP9QJf/DW7J26WRvsoqT/8f4bXvXJcsXj6xx4Vkq+pzySnu1defV/fbTUs3uzRm09pBSkR3jLpR8sAcD3gp4FZEFmTM8XZpe5Vuw/dgYXvWfM4infqli866gjsv3nmzUY/4s/k/E7ZU0K98+evTXg07XdJ/mVl+7ORur46djiUpYFP3CmsMRQFUYtnFaOAf6V4XDcjmWwm+s2mskwmW1m8a6jSB0/3TEXNRZsCcu29OL0LcgvKnHERdAvuxMsCM/IQxiHFGTGhBf4yDxZgNlbMjB20lLNThknU5CvVrHSWGXy4nSm12NBQXEJPlmYpuhaIpUBdWMw/2WNcmnk12ftcPzeLBj3caAgyCiLRQbswuPg2/6pmZpb3tHuN8eKcIU4hnQKx0nLsufCNc9g/l6ppc/aElj4L0vjJ892eDd8M2zM/XjhHkxZsc9xC/J3DIKbeeaGz51LZRftQwIpyC5w+9ersSztmGZ1vFibsj+YuxsvTN+KH1aFRr8XFpfinTk7AQDbZFZcmpxDmbQgzWsRAAA7M/m0tJ/Mjw0fOTfxU9li3nErN730i0nKKM+WUyU3uuJSESUOy/xfn+6qGoVcVKxDCrILGCkbHRdjGrKkwJwKc/ZXilgGKFWdRE4wV/OWwydQzEFKtqN5/gsQirGuZpmCsDSVSuMYzwqX28iV3l/XHsAhnQIYRChqipxb67Ivl6bj3NfmuXOzKCHa+z8pyJwQF2PbvqTwWkNe2pMHV4FHftrgtQi+RWlysVrC2gmMWOGX7s52QRJ/MPLtgJtTflEJ7v9+PcZ9vAzL0rKxLI39M8oxWNRIcuWQl3HelZmHaRvYZx2xi2qQnksa8lNTNyMtCrMXOZmRKdrz5pOC7CLZGtY2HpQdLzD6vaN8oWqJ2Gwx0YNSk9arKsYb+T7NGe4kkiX08PF8jJ20DGMnLWM+fk2csc3U+eUuFsCwN+bjrm/XshWIAWoWZBYVQv1McUkpJvy51XI6PyfXF/L6AdEIKcgu0nfCHNVGHmv6sdkJ42GyVEYQ/gipmpkxeO5rB2lb3jQzN/GZ7SLarWusWLMvB0dy81V93O0WBfE7s7Zk4KMFaXhu2havRYk5bCvIgiA0EQRhriAIWwVB2CwIwr3B4zUFQZglCMLO4L817Ivrf/bnKEf9O+2D7EbaKjNknjSXpogChPSJxu1Bwjv80uce+9X9IKvnp23BmPcXu35fu/Di2namsAQP/xgwehzOzceQ1+a55mvsN4qCD4YWXO7DwoJcDOBBURTbA+gL4E5BEDoAGA9gjiiKrQHMCb6OeS56T3lQddoHmbfB/M+NAauP3Xy1BOFbSCFgwjEPKsl9umiPqtFB4ND5ScoIxIur2o+r9+OMbOw/U1TiasAXL8+B4BvbCrIoiodFUVwT/PskgK0AGgEYA+CL4GlfALjY7r2imc1U6IEwyRdL0r0WgSCIMA6o7BLyAM96odPp3OTsyOAzPaUSWScDlXm5/vGiFKY+yIIgpALoDmA5gHqiKB4GAko0gLoqn7lVEIRVgiCsysrKUjolJjiZbywqmSAkyCfN39z3/Tqkjp+O3Vl5XotimRNnih3J0uAXNhyItCLf9MWqiGNO6H5u5VJ2i+MGM3OwIFNSOhVQsmSfKSzBi9O34HSh+/nVnw+O806Uc2dFxol8LHGoArCXMFOQBUGoDOBnAPeJomi454qiOEkUxZ6iKPasU6cOK3F8R6yleeMJtTRCmw7yW5aVsA4v26u5ZwIKgTx1n9+47/t1GDtpmddieMa7/+yKOOaWEiWllTMDL23fKJ8u2uP6PZV25j5fko6PF+7xtEDT4Vx+A3gveHcRrmJcAZgHmCjIgiAkIqAcfyOK4i/BwxmCIDQIvt8AgLHamDFKrAZYzd/uvXJw5IRywOAF7y7C35v5jJAnjFHggzRktDT2B0Ulpbj6k2VYvTen7JhUwlmOkhLKS3AcL3IY5XCuuWBuFkzfeDjimFSUycuUczwHMWZpWOT9DIssFgKATwFsFUXxDdlbvwO4Lvj3dQCm2r0XEX2sSD+Go3n8dq49MbpwiQbmbc9E2ydmYu2+nJDjHM8zhEG8qOC1N/sUFu/KxiM/rdc8L+d0ZNAgj+OIl1XQck75y6VQnkeaiB1YWJD7A/g3gHMFQVgX/G8UgIkAhguCsBPA8OBrgojg8PF8RV8+grDDgh0Bnzi5xY+IDvYd4zcQTsnSd//369wXRAG5grcy3bt+oVYUhFekglb+kpqwS4LdC4iiuAjqu4RD7V6fiH4ufG8RACB94miPJSGilSkr9mH6xsP46qY+XotCMIDn7WaekcdbeJli028hN6/+tR0AUFBEuYhjCdsKMkFEMzQP+5fPFpcH+Iz/pbyYxPwdfIVDxGqZeaPsP3YaBcUlaFW3iteiANAeE4o5L+bAS7EJHnNFS2zRSLlK2aZiCyo1TXBFvgcr9J9WH1B9b9qGQ6rvlZSKyCtwP+0PYY5wpWDDAcpO4icGvjIXw95Y4LUYZZzMV+/z6dn8+RrLkXs2zN4aGWBIAKcKQy3rK9OPlf39o8Zc4TZv/L3daxGiHlKQCW7YkXESZz0/y/X7vjFrh+p7mw6eUFWCH/5pPTo9/ZdTYkUFQ16bh6+Wpnsqw29rD4a89pn7I6HAKQ8XploR+7y3rWKZi8WUlftdvffafTl4f25kWjwe6D/xH9X3xnGaxvAdhRSDBFtIQSa4gdeB6KbPVyoe/2XNQcXjTuFl1LlZJi/eg0PHz2DP0VN4cupmT2XZkeHfQhxEOXL3hdfIemaJD+fvLn/h8nByyQdLynx5eePgcX5zDBPeQQoywQ3FnEbeyLfYvGTxLn9ULMs4kY9n/9iC6yev8FoUwufI06PlF5cryCfOkC+onyG3++hkY5S5r1GQHsENfrGQfrIwDfM8KHDilwCRl2duAxDqq3kyvwhVkhO9EikE3toZ+bGrM+S1eWV/y3WqNfu8SAupr9WptSzaxQiF9OPoJD37FDo3rua1GMwgBZngBrmFSKKguAQV4vna6Hhh+taQ16IoUiYCGUquJ0t3Z+O8jvU9kCYSvtRj90oT+52FO7MwolMDDyXgreUQbhD+qx/IOY3GNVI8kWXcpGXo26IWVu3lY1cTCASrS0RbDyEFmXCdR3/aoHi8UEFBbvvETDw/pqPTImnCi+eHn3XwW79aTXmuVeAk8xb3HM7Nx0t/bkW1inzsRCjB2eaE5yxPy0bXJtWRnBgfctxPY1lJ2ARwutC7/NFL07KxNI0vV7tjpyIrR0YLfJnmiJjg+1Xmoqf/2HDYIUnY4NakmB3FA5Gb8KbE+K2qmFcIACYtSOM20MtviA7b+/Zmn8KVk5bh8V83ReSHnuuii1r3ptWZXs9Huj1hE1KQiRDyi0pwJDffazF8hVvqzeO/bnLpTtHLyLcX4qtle70WI4RwCxUP5J7mz99d6ykdOn4GcxzP6xtdqpHT67ITZwKuQ5sP5aLV4zNC3nOz/HvnRtHjE8s7oiiisLiU+4I5RiEFmQjhus9WoO+EOV6LwS2UDsg4PKoTW2QngggAACAASURBVA+rV8nyCq8syCvTj2HTQeWo89dn8WelnbJCfefpwncX4aYvVjksgf7v5LRVliVW3BxaPfYnPl20R/9EGUUeK0txfvLn8CHhQc9tnpiByz5c6pE0bCEFmQhh+R5+nP/L4GjOUUoo72RWhDdm7cBjv27UP5FDOPrZuMarsrtXfLgUF7y7SPE9Ht0+tmecVH2PF/ej2Vv8U53Oyk9cXCri+WlbTH3G6wBmKwpyqcauzgvTt3K50PYKpUe1fr8XWWbYQwoywT0rOMlDrIaTqsQ7c3bi2+X7HLwD4TU8Gri8Utr9zmt/q1fl5A2z45aW0qiF10FcVpIgaX3T+TuycNXHfBa18oK4KB4qSEEmCJtwaGzjggKFrCSEP+BRafee2H4o0q5CvEGNSGpDXivIVizI0q6gmqXY6DOIBZIS4vVP8imkIBME4QhKafuISOxMtSWloiPP2a/TP29FYKIJyYDsN93QiouH1IpGvr1Q8X3ya1Zm7rZMr0VgCinIBGETPwXmEPxhZ669+pNlaPPEDP0TY4QDOU4G0cZ2P5csyF77FJvFyvi8Mv2YZnYZLQV5ya6jOH6aD594N5A/39/WHfJQEvaQgkw4zon8Ite32RpUS3btXm4YrSYvNhc5zgNkQTbGRpVMEkZYluaMf77flCCJjBPsU1TO2pKB9k/ORF6BdwUinEAURezKPIkVBgOzJQXZbxbkeAtt+YO5u/HHenVlT83F4nRhMa76ZDlu/Hyl6Xt6xZhuDW19Ppo3bUhBJhyn5wuzcdbzs1y9Z4cGVV29n9M8+4e5yHEeKHQxvdN9U9biB5MFaKxwafdGzK9pRMndfuSkI8pftOFETulbvlyFM0Ul2H7Em8wF9w1r7di1h72xAP/6aCkOHj+D/CLtBYBY5mLhLw25qoXKi4UlpXjwx/Wq7yul+1y99xg6PPUXAGBnRp7pe3pF1WR+K1N6DSnIhCIHck4zu5YXlsSBrWszvyaPBR2IAL+tO4RHVEqYs6RPi5qO30OJ899agD4vuZef3Gc6UBklDpqzvLCUDWxdG/cNa+P4ffpP/Ae3frVa85ySMguyvxpHggWTd2K8YHq8/22tP90L/juqna3PR/OsSAoyociDP6ivnv2AE1vEe7NPKR6P5i0mlpwuLPZaBNs0rpHitQim2H/stKWqVn5N81Zq4queyC/CqQLjbdKLbv7VTX0cu3b491mwQ7v8sxh8tjy7WPRODV3AvndVd9SpkmT6OvFx5lWjrJMFpj/DAykVEmx9ft8xdWPa7qw8DH51rm+fDSnIMUxRSSkW7lQeFLksGOIxahMkBekZ49Gf/VnwRE69qu75ttsl82Q+Br4yFy/+udX0Z500Ej5/cSfHrm2mwEmXZ/5Gx6f/ijh+prAEv2v4n0YLZn/iMh9kjjXkr27uHfJ6YKs6uKireR9bM1bn3i/OxpQV+zBz8xHT94kGrtbICf3aX9uRnn0avV6c7aJE7CAFmSHpR5UtjLzy1uwd+PenK7AsLdtrUZjjRLonSiFlj12Z/vHL06Jd/Spei2CInFNFAIBFO4+a/uxBB7NB1EhJxGtXdMWzF3Vkfm09F4sdGSex5ZC2L/Fz0zbjnu/WYhXnBYrsorfLdrqwGCfzi8pe+8HFInzno1pKoqXdRKMKsiiKyDxZgPG/OLP4X+6DuflUobrvOo8VOc1ACjJD8kxs13lJdl5gu+P9ubsBeJ/I3S+o9XWfjwGuEb7AmLnpMH50IbCONduOqJc85glpZ8OKPlPssL/95T0ao0099gsNvWpv5725AKPeCc1tezg3dDFw6HggGPJk2Hgea/28z4tz0PmZv8tel/pAQWZl3E5MMKYaSc0t/JGoNZVV6ceQOn56SJtbt/84isLcoHZn5WFHxklcOcnfFftcjNN2BFKQY5AHf1yPbbKIbKlvz9h42BuBHMAJH2S1uZcWGNa4/es1eNiFwDq2+EdLkhQ6K/7ETu6iS3L1dSDgUU2xX7zrKPq8pLzNG95//fML20NvR0xtgXA0j19/Ulbjfo0UY5kd1BYNalK8N3cXAOD1YEny7UdO4uL3F+PlGdtCzhv6+nyc9+YCExLb4/ZzWho6L6WCuap5OQr5oP/afMQ3LkykIDOE44V1CHn5xYppaO74Zk3Iaz+7FDjiYqEydR5SSPmjxtWfLDMcAPn1sr2Gr+sH/GJ5jRZEFeuWEZwcyzo1qha8hwOLWBUFecKMrcg4oazYqQ0VPhnOLWN2k4BVFh+WAeC3DWoR8prVb2Y09ZlqbmgVQTYH3Xt+Wn0AQHlg3+p9OeaFZEjFRGXF9+5zW4W8vm2QMUVaYvXeyO9121ercc93a01dxytIQbbAoeNnLPn18UL4vKQ2T/lYP3Zk8lV7HmpJ45VYvCsbP685YOjcJ37bZPi6BBGOncpnTmWx+O6Wvmheu5Ij1waARbvsj8tqi+vwbfBYQ/797fjGGh3/rMBq2Dea+aK0LLNH6I1P5ke6W749e2dENoe0owFD1dp9xy1I6TzVLOSQjiZIQbbAeW8uwDWfLo847pfUSCvTQ1d1ap3Tx/oxkgz6kJlBLeCAhTK+dHc2MqkQBABg/f7j6Prs3+S6wggr7hJ+2Q0L508VNzErY3N4v376982WZGLBb3f29+zeEgnx5WNqNqd9UxAEvHxZZ9vXMVpoyoxf9puzd0Qcyz1dpHCm+7StX9nQeX4PujMLKcgWUAvGM7PVzhMfLUhTtI742cXCiQle7XGw8Ncc9/EyXPTeYvsXigL+N283cs8UGbJS5Z4uwqi3y4OunG6zfuoS5RbkyPf0nhPLQKxdL45kdi091NwAtL6OH37Tbk2qey1CyDY8L5nelH46FoYqo03CbvltXmpPjejUAGN7NdE9z8746sdCW6QgM+TtOTu9FsEwRprqxLDAgWgg7aVRlj53Mr9IdSJtWL2iDYmA1PHTAQBHYsSC3G/CHExevIfJtWZvzcCWw96UAOYdW6WBGSpAckus0znD1eZgK18nLSsPmw/l2pKHJZed1dhrEcpwwoXNCorubQxEM6oHltroY8vTsrmyyLY1kL5SqyiIHt+t2Bd5vezTeGrqJm6VZ1KQCVU+WcRGieEJq0nur/5kuerkbuSKoijiGQ+3aHnicG4+nv1jC5NreTFPX9WnqeLxFnXY+NbO2ZqBF6dvwat/bcP7wah3K5RZkBXe05uXjytEn/uBYjOl9IKo9etn/9iC0e8ssisSM17/V1e8enkXz+4vtx6uMFBIaneW83nPbxrQ3FIpaT2MLOREUcSR3IBRw8o4dOWkZa66MfZsVsP0Z8IXQunZ1hVkpR32u6esxZdL92LDAT59sElBZggni2pD+Nl9wgs2HMhVtU6tUojUDScrrwCfL0mPOK6Xt9UNFj06xGsRQjBjZQy33DjdrEUAlZOUS7PKywJPvr6X5Xvc9MUqfLxwD96fuxuv/rXd8nXKsDAwLd7FrkCB/O5OVyK0oB/7wsVCj7uGtNI/ySbyx7Q3W78o1kfz05wTJkhKhfjIoHMWFzbQJt6cvRPnvxVIxWYmUFuOU+O/0iJez02nZqUKutedaMO/W2kYkg7x2gVJQbZJYXEp3pq9A2c0qsnwiHxS8JNibxQnAibVFhWGIp5VRgCnCzIYIT5OwMRL7Qe2eEF429WrpMYCtXbQSOZqU6dKEirEsxleZ2/J0D1nV+ZJfLs8dAtTklLRgmxfLEu0rGMsGMgqqr9/WEORj9fe90DjKI01zWtXwkPnt8U3N/dR+IR9lKzFvBQLUfq5Wbh/GGkT78hcKq2O4065WFRJSsDlPUJdcq7tl6r5GaWS3OFPsl19Y8GLSij/VtJ7fPZCJiO4IAifCYKQKQjCJtmxmoIgzBIEYWfwX/P2fR/w5G+b8NbsnXh/7i7mflkvXtIp5PWy/w5ldu2p6w6W/R0nCI5ZUeZuz3Tmwh6gNgYaWXmrfdbKlrATtKzrrOLiFOF9zohlyw6iCPRMZV/gQot7p+jnDB359kI89mtouVtRI0jPTdy8v2qQXtjrKyctLfs7fHLmYa6eemd/fHJtz4jjg9vWxWfXhx6X5O/fqrYjsvzro6XB+5QfM2Ixdet3DzeGsLit2TaglNbN0H0sfcrYdcN/oirJyjtfEl74lUsLLR76nBKsLMifAxgRdmw8gDmiKLYGMCf4OqqYuz0T3wdL5eYXsbcgX92nGT4ODpIPndcG9aslI33iaCbXnrs9q+zvklIRV33sTEnLX9cc1D/JCRzo62kqPnVGVr/hbgObDgaCf3iwICtZ28f1Lt+iU7JabXjmPEv3mr6BbbXGcMmHveF89almtVJ0zxFFMGuDpwzsThWVRLYjrSA9Xi02TpEetnDacICf4DslujapjmEd6im+d2475eNuohTLsflQLpbuDrjmtH78z7KiGCxR9Dc20M+qBN2ihrWva+g+TgeTSjhlQVbq3zUMGHIa2Qw410Lpm0o/Z1QH6YmiuABA+D7MGABfBP/+AsDFLO7FEzdMXun4PYZ3qIf0iaNx17mtHb2PET9aK4Snj5MUQ6cZ2q4uGlZj6++oVhbZSN8OH69OB5WeYgXFxmlSFRS88DlmwqWdcXG3horvqR0zwp3frtE/yQRKlZrcQj6Z/KtnZIaBJEYuFlaR2qXXm+FmLVNO5DA/rpFvls+pmT/kgVRKiurodxZhXNDQorRgY0F4U1JM86bQ3Do0DLgG3Ni/uaH7uLV+dOs+RufC6mEltlkalbXcYTjVjx31Qa4niuJhAAj+q7h0EwThVkEQVgmCsCorK0vpFF8gCO5PRKwVQCcIt9QcyGGXKzq8Lnzv5uVb37UqJ2EJQ5cULYxYAdTOKPagOtcrl3cNeS1CVBwIX/9XN2x57nxFCyQvaZ6UAh/dIjmxfPgMf6YA8NMdZ7spDoBQy5H0Nw/+or1Sa+B/V59l6NzBbevYutd2kyXN/W5Ml4u/8BHnAm7lWQjiPWhTzWtXMuTXrxgMJh0zKLZbTcIpA4kosonDsXuFurLdRyWrtrTOcstibxbPg/REUZwkimJPURR71qljb2CMNabfM9Dxe/y0+gD22Ujt4iRXhiU2V8ocMOv+QY7LYcSNODxa+XRhwGetyIOlc4QVRqOEdkqFBMUk+N6rXN4gQkRicJKukaK9Zdm2fhVHtyyVkG9Vlv2pVCjEHXHK+PH2szGycwPNc/4MjmeXdG9k617zTMc98Dk5G0Xef5vU1Hf/McIWBfcI+VPSWiB/4dCi9cpeTfDsmE4Rx42MRdefHbAct62nn+sXAG79cpUZ0SzjmIsFvI89AELbjNY35XWR6qSCnCEIQgMACP4bPdFaCnjh02bEp8guD/24Hpd8YL3Cm5u+jpUU0m+1Njgg2sHKIPfSn1sBBLKguMnA1rUjEsKrZ8sNoDQZujH48jDAK9GyTmU8N6YjPjBoEXUKpb4lz+Ig6v6yfNGhYVVse34ERnRqgN42AiHtthsqcQ6MemdhxDH5Il8rRs/Jkty1FOa8yDRvkcKN6FQf6RNHo1ZlAxmHABS4NC7LXVU+mr8bqeOnq1bqNct9w9qgXXCsDx8pLjW4CLW7UygfopTGq2VpAc9cngqmyHFSQf4dwHXBv68DMNXBeznG3O2ZSB0/HUfzCjTPW77nGHZlskuMHu4+4CXZNiaMQ7mh1eG8UHqu6RsIOBvUxpkdCiN9O3wAkCbhjS75ZEuc1bQGqiYnhgR7piTGo2NDc+l7WKfRW56WjWydPuYl4Zana/ulorbGZCstQpxs70rtLmQ3QzNIzxmZJJ4Y3d7S55KD5Yx/uL0fS3E0CX8WVJlRGfljSoh3fyBXCnwVRRGt6wb6mjRn8rKw1tMZgNAg7a+W7QUAHMuzv0ATRaB+tWR8HUz9JwUWD2wdyHTSt2Utxc+xf3Zyly/1s6LaB1kQhO8ALAXQVhCEA4Ig3ARgIoDhgiDsBDA8+Np3fBasJmckIpfVyg8AHteYYK7t1ww3DQhsGc1/eLDlycjvNFDwwVbyy37h4s5InzgaX97Y2xE5DPkgi8qvf18XmuXjRL56MJFT1KhUAcmJ8dj14kjF9zV9+hhw93drceWkZejxwuyQ43YWZlZ5dEQ7zfeN5g+vEAw0Yz3hHM4t9wNVanVyC3JZkJ4HCoPRfKlt6gXSC9au7NxuWPpR7dR/8ue4M8Oc/zIPuOW/KW/7i3YddeWecs5WUeruHx4IYO8RrBTndhpGNY6EGYeUcDp7Q+3KSfj0up748JoeAGSFeizc1mxe92kbDoVakKHeF4tc3kk1CqssFuNEUWwgimKiKIqNRVH8VBTFbFEUh4qi2Dr4r35tSo5xOy1SokZjfG5MJzx5QQcAQLNalYwVqohCbhrQIuLY3IcHY9vz4RkHncVKkJ70Orwq24XvOlPa1kjgU4JKm3M6yOuP9YfK/s4vKsGpgkCGj8d/3aT2Ece4Y3DLsm1JObxYpUIKXCi5WMgmXElxUrQgO6xUGX1eL1wcKFCTWotNmW4gcndj8GvzNM+/4sOlSB0/HQAw/E3n0wSyxo2pqaC4JGSc23+MXbC1UbqqVIJLiAuMW5J4japXxIrHhro+DyixZl9OyKI2nB0KCzK7Y83wDvVwx+CWZa+Htq+H6grxEl2bVMeozvVD7x2eUzr48l89G5tO7fnU1M1l2ZqAwDy5P0c5nim/mM9Ca54H6fEOL9H6RCRKyeqTEuLLtmnV6GGhJr0WGSf0t9LClWhJuWkRVllsr0MBkV0aByYXK825Sc3IQDOnukW7J2d6Yp2ygpGxgbUrSrWK5WmYlPSiUoUgPSU3HqXqaLxxtUK5XCMUcDrZKiG56XRpXM1jSbTJL+TDwhfem0SUu+VUrVhubKhbNVl3HnCDSz9YgnNemaf6/nIH+uHH1/Y0ZDSbemd/fHB1D0PXrJAQZ/p5iqKIM0XyBb36uV6kOzUCKcicYiYXaMeG7g2uWqthv/DFjb3x9/2DyrZ37XL716uxcKd2isLwLBY5wbysvTjZDtTi/avOwntXdffk3sPfmO+7Mu5yWC8kQqLCFeaUkCC94N+5ZyLddrYdLrdc5WrkCLZK89r2LcKNaljLAOJB5kTLSHln5UFTk2+IzMbjOQIfmQaUFqW9UmvgidHtMeGSLoau4USebS0KTTbIn9cccEiS8gWG0R2ksvMZ/PZau/A8FMxSghRkg7j588XHCbigS2RddDVa1a2s6D+qWHXIBlPXHUS/Cf+UVUuyAmvDo5WvWDkpAW0YZ7fQy2IiBWCEozRQDXj5H9z8hTtphoxQPaVCRHtkbRlVY2dmHka8rbzt/eta5yYSiQ+v6YG/ZakCvVYS5DsRSm2nNMTFwhhXf8quiuZrV3TFFzf2RkOD6e0kfUdJVqu5dnnNqaqE/BtOvqEXVjw2FEPaGqv2JuFGmxQEPjMNVEyMhyAIuHlgC1QLK3Khxuv/isxZ7gRmH5fU3N+avZO9MGH3MPy+jRW+kluh2jMpMZIr1QNIQdZhwQ73i5fseGGkoVr3cpT8R9c/fV5Z1CoL7p2yDgCw7Yj3Ud7T7h4AgI+yq0aQ+9nKmaZQevlAzhnM3prB9P6NqgeCMxpW01ZcrunblLvtXjW3k7cdnEgkRnSqjzb1qmjGBMhpW68KRnQs9+t76Ly2TOWRJpjPFu1B2ydmRrwvd6cwGjex6SC7/nx5j8Y4x0S2mFhzYFMLNBMBDGlbF3Wrsi/+ZNZVRandCOArW/TA1rWRPnG04X4px4zxyU1YrD+M6rNq9wr/eL8WgR3O0V20c5grEV69Uuv78brrE5k4lvAcs8qxGpWSEvDoiHZYuJNt4BcPhoROjQJK3EuXdsLsF9kqk06QX6Q8Avy92R3Z/9WzCepWSQ4J1vv42p4RmUCkoCk9eHDNj2O8Q6LFB1efha+X70X7BqE7D7UqVcB9w8rLwP8VVpjmwq4NMWlBGrN0fpIV7+WZ2xTfn7c9C0PbBxaNmxkqvk4h5aXtphCAJbWxxjUqMq3A6SXf3tK3LCAQCOSqnbc9E2N7WfO3VqJhteSQ9JpK2X60UNrtFgSBCwsyB8MO17yqUNFTjtmdv1Z1q4SkBLWDWsVWINQ1jCfIgmyQvzYd8VoEUzxzYQfHrs1TU06Ktx6I4WaflAcryHFLxxMEAUPa1Q3x4RveoV7ZQsPYNWR/sxTOImlZ2um7WNKkZgr+O7J9hA/k6ieH49/9UjU/y3IxISkvaj57cleeL2V/h1sFeXFDaF67EmbeNxDjR6qn1zP7/KTy46IoKmYJ4Ik+LWri1Su6oqKDee/NuncqKcICgC+XKLuJeQGn+lQI8j5WrGMibVS9oqtGB9XH56AQ4e0wJKCYfJD9w7ztmRFJvqes3O+RNOZJnzga1/cP5Elm1d43HDjO5kJECE5OjKyR3DMqJsYz2+UgzCEpukbyp/ZKLc/W8vLM7YbvoRTUZ4V6VY2ln2xXv6rmVrlZq5e0tTt5cTrO4zht24x7B2oWm7FK+CIuPk4oc0kzgpql+MgJ/by+fkMqIuU1KRXiXUmd1y/o4qOUytIs5l13Ql//tbnc6EhBej6htFTE9ZNXYuwkdoErXsIqmCpbVt3Hak5ovVW0JYJfr0qyeW+hkZ3N+1Wx5o5zWuqfxAmPjQoUpOnboqbr6Q9v+4qfoEUvOWqgypZUvlzua/ndin0h52h1YVbFMhY+ci6TXLRWm9pin6QLtMsrl2tnb0iMF0ztFCm1jd1Z2lViM11SnlvVDWQeGtONjR/x82M6YVzvJqY/Z8Q+IH9mernk1XYYWXNx90ZY9cQwdG+qner0+rNT8ec9AzXPeeHiTqbuvXxPNrbICq7Jcx9TkJ5PkFbPuzLzdCsw+QFWegyLij+OrhItXPq+oa3LCq54BQ+5Oo2SnOjdcPGXS77avLPFQEXPIoWFKCursBofXnMWXrsi1P/RSu5UlvDut2zHTeCysxoDCCjH/VtpB2KbvY+SBfmi9xZrfubgcXeedcPqFbFnwihc0dO8UquEIAi4c0gr058zMpW9O2eX7D7a57Lqn0aMV1q7FpKYF3ZtiA4NtathmjWSpGWdwoQZ6rETPEIKchjyhq9XgckuH15zlqPXZ8lpgwm/9bBreWxYLRkjO5VnCbBzubg4IaTwghe4EfjSgkFO2liC9U/C0tauFMwWjiS+Vtti3epGdGqAy3s0ZnpNO7tfoihiO+f+x0k2FpxS8Z5aldRLdEtV0sz+1lbsGG7ukLPYvRrbqwkePj+QYaZxjRTb11NC/kj0ZGY1RtjNP+7mxqDcqr7ERupYJ6EsFmHoBa/szWZnVXaqY8ph1eDlddiNBvgsT4ts9E9NtVc+eMl/h9r6fDisxoMCi7Xk3Qg2eelSY5kpCIdgOOsUG9iKlKxIWkqLVYlmP3AOhr0x3+KnraEl6yuXdcEjP2+IOP61St5xnmhZx3qhoruGtEKHBlVxbjtzOZONYGXR/vGCNOZyOMnEy4wVFWGF3m40K5e1nswKTzk/MfkhjoUsyGHojQ0r03Ns3+O2QS0w96HBqK+QfqdTI+1tDbOw8kGW91+j4+eVCn7ch3Od8VWz2p3VxqVvb+6DL2/sbfg6+RZ9yHhInWQUr1O7Zefpl/Q2w9MqmV7OaWs8j68RRsl2POxSZKAkq3RGXn6x7jlm33NzTuvdPDDZa+VVPkulbHy0u+QkxMfhvI71FRWr8ENmhxjRwlo/naHhKFqQ+2XrlXb3emyVYFk5Tw+1/PY8QQpyGHplIecwKOBQPaUCmteuhNqVk7D52fNRM7hN9tVNvfHNzX0tX/fja3vi7bHdQo450fG0rrk7Kw+bDynnfP117UH2sgT/tRo4qMbZrWqjrsEofEA/CEMNJ8ah285pgStlPnptGVcN9AqlBZcd1H6xR0eopxyzwq2DWmDTs+czuZaRQFepKzz260bVc46eDF1sHD+tH/znNl2bVMfOF0daWrAsipEAPSUiFGSTo8weUnaZcKqw3GiiNz1YNbCwxs3g61f/Mp5ZxytIQQ7jzVk7NN+fwTgfcqWkhLKJun2DqrZ8Yod3qIcx3RqFHHOiub81e6dqhx76+nyMfke5MMkmRsUS5EgdWqmSoLHPs5HD4u2ZK/YA0LFhNbwsi2yvoeGnyIr+rWo5HsS3K1M7kt4sapMB660/QRBQOYmNN5shC7KBNvXJoj0hr7s9N8vQ/RsoVGKs6WD7SoyPc62suVvMefAcfHNzH8fvIz03s0PMxe9rB+QR5snXccFTKyTlNg8Mb4MqSQloyyANnFmyTrLdIWQBKcgyMk/kY/LidEeuXSGh/FEPUIk65nm3XT5FnS4ssVQv/pvl+/RP0uCfB8+JOFY5KQH3DWuNH2/vZ+maSpOvpMyYmZjjLWraTgS3OKF0A0BqrUAAiLTlXSFsVfDeOP8EnQKB8ql2g1rcpsiAD3J4DnejSO1Gq/ko5e3uoeLmwAwf6sd9W6j7grasU1k384QdpHErKTjnhPdTJ9h2hO+ASK85cIx/dwIA6N+qNjY+ez6qJKsb6p4f0xEfXtOD+b1X77XvvsoaUpBlPPRTZLAHK54cHcgh27NZDXRubDwnpV2qpzhj3TlVoO7f6BQtVIJa7hvWBm0suhHI9dp/920GIOAjbharZY95XhSF06JOZax5cjiuOzsVALDt+RH46qaAn7YoAr2aswoQ0YeFP3LtykmY+9Bg+8K4SJGBYNBvl1sramR1seZD/VURlsWQHjqvLbNrWeXWc1rgziEtce3ZzQx/JucUf6420cD8HXymMbPCv/ulYgTDuIpy+JsMSUGWUVjsoB9QUBNzuwnUqcKmUlP4djQvpWpZUj0lsGqWFAUzRmF5QFRBcYnhoihOBOlVquBccpqalSqUtYW4OKHM91oU3Q00MeJqwBvvjOtu+xpG8pFbbVNKn0tK8H6K0GpWLNucXq5fOzw6oh1+/c/Zjl1fieSEeDx8fjskJRjPRd3/5X8Mn3v8X6C9RQAAIABJREFUdCFmb4nuYEhWRLuFvUODqpq7Jn7F+9GPI/TmHkcqwcF5xWLb8yPw/lX2tr+jxUqkRVnAnwXlP1tmeWn7xEy0enyGoc85oeYNbR9I/XRWU/2cuXaRP7NYaCN2uKir/epfRqy8LBXk9g30s+rcdo75HRdW+KXN3TG4pW71MlZI80n4r3l9cOdHi9OFxo1El/1vCW7+chUW7WQbEPnHXQOYLCadYO2Tw70WIYJKCm5PbtKoekX8ee9AXMhgfOMNUpDl6MwrpwqMDx7vXdXdRMCdZF12xiqWnBiP0V0a6JaONEO0BM7ILeNSOdaODQP/mvmGVgPUnPAXlr7Tt7f0xRqHB3TJtSQhLs5yJo9YpEJ8nKXJdsKMrVi9VztllJqCPHPTEfy9WT3IWOljn9/QC1NuVc6sM653E9SpkoQezZy1HGmNo26XPFeja5PqZWnpAG/Tdkk+4ZWSQhWnZy7qyPQ+u7MC2S4+Xsg2B3LnxtWYLCbNYLRolxsBz2bp36q2q+kX5Sx8ZAj+vDegV1iNw+EZKhQiQ09BjTOhA13QpSF2ZuTh7Tn6wWzf3NwHP6zajzoaJSBZ0KFhVdSsVAHHLPiZqaUOKi0V8feWIzivQ33Lfrheck7rQMDZJ9f2xLAO9bDwkSFoUtP5Ai4STvogJyfGO17qt1dqTdw8oDluHtjCVQXZ72Nx9ZRES5PtgZwzuOx/SzXPUdvouv3r1QCA9ImjVT4nInX89JDiE9VTKqBvi1qK50+4tAsmGJDZLm5ZXu0wrlcTbDl8QjffrRu8dEln3DKwBepWicyz7wQVOHDDscvZDgZNOk29qsnY+AybNJJmkc+VIzs3wI+rD1gOtuMxHsf/LZshej+QWQXA6Olt61fBkxd0cMUawvoOU1bux+1fr8GUldYCg/RwevuoWkoi0ieOxrAO9QCEdngz6avOKGxN/m/ebmSe1C6M4qdCIUrExwl44oIOqF8t2fdKq5s8wjjXspwSA5kulCgO+m/8sy2TpTiOwkuTG9u7aYhf/KmCEozr3dQTWZIT4w25xrDCDQU5wYfGF7dISohDpaQEVGKUStIq1Som4uc73PWzdxqyIMvQU1bMqjJyN4Ro697SdzsSrBakpwj6kVomLPq/rTuEt8aG+s29PHMbFu7Mwre3qBd/8bl+HIKfFOS7z22l+f4n1/ZEioOLs8t7NHbs2lbRc90gtJHHqKRUiPdVf7DD9A2HHb2+2o4HQTgNKcgy9HQVs/6i15+dijdnaxcecRtWg7bkYsFLBSBekaop3TqoBSYtiPTV87sFWY6ffJAfVEjD9fVN5cUbpB0F1ix/bKjibgMAtKpbmUkxlAU7rAVNrd2nnubs5zvORrWKNF1o4eSCinAW/4xckbDKVOU1PM6ENOLJ0NNVzOYJrZZivSqec1gbCtR0H0np23QwF18tTbcmUhQj7QxWTXa2q1VOSkCeB7mp5bipIM/YyN5qNaC1836I9aqq+4W2rVeFebVAM2ilkHO8GIhFeFqTtayrnKedsEac4EwhpWjjpgHNvRYhaiEfZBl6FuIrP9IOjtFCKt3oVJUzo7CaUMKzWMzemoknp25mc3EZIoBvb+6Da/sZT3bPE/I8wUqwsiC/eEknJtexQ3ycgHddSs+0PSP68oo2q8UmOLRJzchy0EYwqozMf3gwZt0/yNI9WMNTNp1wn2N+JPMP/VuVB4TOfWgwPr62pyv3ZV1e3k0SXKiUGKvQk5Wx/kCu5vtGk313blReKe/S7o1wx+CWnqVhCUev3vn9w9pgxr2R6eDC436cLhTyqCyI6exWtfHcGO8VQD3mKgQ3xQsCjp0qxOp9ypG9LAKi2tWv4ljFRLNc0KWBo9e3WkY5lrhpgHZeYrVFutHgvma1KqG1xcqVrOHJgpwYpqhI5XqtpoCMReTW0Ga1KmG4Q65O4aQ4WFyJ8C/Ucx1AbhV848puIcoe7ztG9aomKUZAm7V0rttvr2xrVR/6Oy7ZHen7KQjA2ElLMW97ZKnRE/lFWJZmPjDq6j6hlqorejbhZgHmdCYWyX83ily3maPXFtRcKT5euMcBaZyDl4j52irBvFIVwtsGtXRTHCKKmPvQYK9FcA0ex3RSkB1A+YfmRIPRQdJvHh/VPuS4WV+wVemxFxFfUBxpgYsTBOzIUPYr7fLM38zuzdNWM2GN5rUr2fr8a1d0BaC/CC/hcSayAC9+0e9dFelWxGNBCYINRioSssLumEDYgxRkl/HL3NSxYagV2azvNKvv6ZfnBYSWm5YwU1zGKgKA9g342PJ2Cz+1C6Nc3L2Rrc9LbUDv2eQXWsuTzCM8uFgoidCyTnnAHu9NlRd/cr9Q1XCFXMLvkILsAEoDojSQ8z5YVlTxxVKzIDdVqToXTenLtOgnqzJWrFDCzInMDuHPVoS5nM0Ef3RpXM12qdbyXQTtvvfL2gO27kOEova0y35OzsdCXvzJY4knL+jgtQiEAUhBdgAlaysHhg5DXNA5GGQVJvDnS5T9E1urpDaym54nOSEeZ7eshQ+uPsvehRzmmr7l2TUKFVws3KiOyDssq3pJ/rMsg0Rn3DsQb4/txux6ZpDad3hhCStbq9JuhZ4+9sG83aavzSvUv6ILztcSANjM5U1qWMs0E3EdixlreMTpwH8rkILsAJrWU45HgA4NqiJOJcJnZXpoFoa92acBqG9xmrUgPzemY8jruDjg21v6Yki7uqau4zby769U6nPTQe3MKCzwOnWgHn/c1Z/ZtT5dFFiosfzK7RtUxZhu9twbrDKyU308OqId3hnXPUTZU9uZ0UKyIOstTvUy2fDIJ7J0XwsfGYKfbu8HoFxZqethsQTOu18IVTwuR+wnBqrkRddak43sVN/Qtf2cVk4Pq2tWHvuR4wqyIAgjBEHYLgjCLkEQxjt9P16RJj8O20AZIQ1bR1CpKIVSYJoVru2XGvKax85ihWMKfsl2kZ5N5eBk14bzLVKWeTq/WrYXANC4BpucwV4jCALuGNwSdauoFxAxfq3AvzxaYuxSt2q5AtykZgp6ptYMed/LKo5qz5vHwNlrfJpP3gvG9mqKVU8Mizh+/HSR6mfeMrgTxaq98jhPTry0s9ciMMNRBVkQhHgA7wMYCaADgHGCIES9841So5W2TG8eqJ2j1Gm0yqGaKZUqfUe1js67VdMu1RQCNUpFEbkagycrSkUR6RNHY9Oz52PuQ4MxqE0dx+/JGx0asnPb4BErPvzSZ5alZbMWx3PUlE0uFgU6t+ZxJLyxv371tf8Mbokpt/Z1QZpyeJo2SkRRMYVf2tFTqp9JSjA2h6rt1GrRqVFVxYwpvHFlr6ZRU3bdaQtybwC7RFFME0WxEMAUAGMcvqfnKPXxahUTkT5xNC7q2tB1eeR8dVNvxeP3Dm2Nd8fJ/H11+q+0RVSsUlxAIV7NFBUT+e1gn9/Qq2wBIH9MGScK0PU5dqnbjBCraYC0FmB6k/+/ejZmLQ4XZOcFdiu+XrbPY0nYY8XlxC3UWmLreoH4jFYclqCuVbkC2jeoqmnte2REO9d2p9RiWdzkkRFty1IlAkCpiq9S1WRtNxUjyqEVD4tpdw/EBV0aYv3T52Ha3QMA8LWgkGNlgc/jV3FaQW4EYL/s9YHgsTIEQbhVEIRVgiCsysqKLKbgR3jO4NCjWU30aV4z4vj9w9ugfjX1bd7KYb5rUv8uLrFXlUuN8zsa8+XygsT4ONwxuBUAIEX2XFbvVa6WxxqOm5cqepOKnAsNLCK1HkFTncCV8Ipn0UI0x6tVS2GTWsvNhfeozg0w7e4BnhtF1Jhx70CMDSuPHY5bTapZrcBCv0Yl71Ko/WdwK1zeo3zxrDaP19LJcZ1soI3ZcbGoVjFRcQeTJ6wE6fO46+z0TKHUCkKegiiKk0RR7CmKYs86daJkq5i/39k2F3cPHeRLRREn8ouwfI9yQRC7j8DKFpRb9G1RC3cMbon0iaNRwQNly4/Na9rdkeXL1Xh3XHdc2LVhxKJMjtZgKvn7V09JxA39Uw3f189UiI/j0ueVJc9c2AHf3RK65S99Z6Nzq3xYeT4sMNgM8s9q3btTo2qUacMA40e2xeQbeqFHs0jjjVcMUAnSa6lj7TYSMBrtTYJHZdcKTs/uBwA0kb1uDOCQw/c0zcYDuUgdP53Z9XhvGlZWr+ErwuOnizSj4dXK2cpZ8+Rw03K4Te3KkdYCeQSyFwOdH8eeprXMbZG/O647Nj17vur7Rp/B0xd2RFvOgxiVMLsLdfvgltyUG3eK6/s3R7+WtUKOmc0vLx/7OjeublmWmpXKlSA/BUXeOrAFRnSsH5KeUgunxzfJdSYxPg5D2vKVsUgtcHa0lApVBbmbhhp2c57zjjULMns57OK0grwSQGtBEJoLglABwFgAvzt8T9Nc+N4iy5+9TiEqmPfVk5HqbuHWqHB/rOTEeE17lZ6CnJwYh5o+KMe64JEhir+xhBfDnDyiP1bJMREMObhtlOxMqfD+VWfhgeFtYtJSafobyz6QnKg9EGpZAv2kFMupUakCPvx3D8Nb9G7tSnA+ZYag9uzGBd1VOjWqpnsNuzukZQtDTh+cfIE/1GCqVh5dUx1VkEVRLAZwF4C/AGwF8IMoipudvKfbKE1K/P3MoVjJ+Tpl5f6Q11sOn9CckD9akGb6HjySUiEhZEAc1TnUN9oLV5Drz051/Z688divG3XPkcZbuV8hwH//BMwpJgePB3KSR7sFWQvjLhaC4t9KvHmlesouUQTa1TdW2tvXONym/FJhVk542srGwaIfd5zT0tDnOzWqaruv8r4YlveJ+DhB1287/DO84LgDpSiKf4qi2EYUxZaiKL7o9P3cRmkFx+NKSE6nhvor3HCUkqbb6aKcP6IQKiSodxMvhim+h0Z+kPpmeBYBP7S9Vy7vYvjc8zoEFm3ySXPe9kzbMnjhX2+asq9s7EeV6xV6O1ha/SwpIQ51ghZmHzQnXRqoBGiz0MNeuUy9LfthLHtidHsAQL8WtTTPM/KsejargS9u6G1KwW1UvaLqtf3Q9kpFsayvaMHjd/HBCOg/eJ+AzQ56tStXKKucJ+dQ7hnLMihlqfjtTnYV11gi79xe/LbRXHXJDXi3tijRsLp+Cdm7hrRC+sTRSA2m+pM3k3nboyMjkB5SIOcoHb9QCbnVWCnHrRFuHdQCw9rXK3vN6za3GV5ysLhDM1n8QfWwbCQPnd8W8XGCqoLOA1IqTT2XHCMMaVcXtSonoWE14yWi5z88GDteGBlyzE8jWnGpiFsM1H/gsR+RgsyA8Pm3e9Ma3ghiELm8j41qp+hjKz/naF4h9h2LVJAzT5QH6SXGm+uySoEM3ZpYD5pxkst7NMGAVgELuhd92Msyun6Gv+HWHHpla8PHHflCgPddLFakVEjA2ieH4+kLjWWkYLHWfGxUe8TFCb6ojmqUIW3r4p6hrSOO23lcF3VtiNsGtUAvWdXD8NiUC7o0xO6XRhlKjeYVTqyvtVKqhpMQHxeRmrJ+1WT8q2djfCwrwc4rpSLQoo5+vn4e+xEpyBaR+4GG959rfVTO89ZBLfHsmE62r6MVk9e9aaji+/yYjppuC7wRHyfgofPbAois4OZGpw5vX2r3jBMCVkW/0rFhVcx/eLDt6/BoibBC58barlAr05VTLALAl0v32r6/XwLRalSqgPg4AV/f1Ef3XKOKSf9WtVBP4Vy5EuknK54REhVWD3Z2X1rXrYz/BhcTEvKuyXsuX6N4MdzExQl45fKuhgICvUZpPK5fVaEfcjjc+EdL4Qz5QBs+iPA+PxsJADI7LGpZrEaEu1P4cMu7W5Pq+OOuAbgzTAFV7OiMMTpJpU0YXabI+5Hp9wwsKxhgh7I8pSpNsmpF40VLvESeCmpc7yYR7y9LC1WQE2LcFWdA69r46qbeeGB4m7LAqXAmX69cSTScb27ui5Z1KmP2A+eEHL+gi4IrR1g7q+TTMrtKw0z4IaV2qIZSAHOpKJb5ts96YJAZ8TyHxbRuZvHeRKfgkV8QRQUdSeFp8rggJwXZImcF3SgGtakTsW1nZDvBSwwFE6TqJ2yXX0er3yeFWYu1bv/8xZ24HRg6N64W4Q/cpGYKhjicRiz89+J9ATbvocGu31P+G/QOVooc2TnSReGpCzrg/mFtXJPLDnJ/2QmX6gftdQzb3ZDgoYyvWwxsXQf3DG2t2kfqVEnCy5d1xllNjblztapbGYPb1imLwq8j81suz8BQfrMtz52P1T7I765Ea4V84fKxZ/INvQy1Qy0S4+Mw58Fz8Nn1PVXzDPOG2WI0LNjwzHn458HB7t3QQapVTIyY8zNORNZQsJI72WlIQbZI50bVsO35ERjavl6IRTZ94mjLwR9uYcTOFB8nYMn4czH5+l6q5xgdMK7qE+pyIh9037qyG6bKgvP+3bcZFj5yrrELc0J9EwEXAPCwSStveDqqlCS+LVSptSvh38FCBI+Paq94jlEFxSiTbyi3DCYlxGPVE8Pw4iWRgUc3DmjOtb+jnF4KJeG1UNtpeGdc94hjNQyUbuZ9IWaVK3s1xS//MR4Q/PkNvbHqiWHY/sII1JBlvrj73FaokpxQZiwBAj7Rfmlf4fTVydKgZOCTBytGnK7QgComxqNJzRSc2079c7xRL7hL2NlFd4aEOCHC79iv1KuaHGHkaajgvsTjeBMdv4BHlA2EUbqz2bB6RQzRSPJ93/frDF0n3N9YvqC4uHsjdOU0OM8oV/dpaur8i7o21D9JhnxwWfbfoaiazL/vniSzmq+505klaldO8v0E04XRhKz0qK8/uznSJ45mcv1YQBAEJCWEKr49mtXExmfOR/UU/gseGUJBQZGP1eEBdg2rJeOdcdq5oiWkVG88bqOHs+bJ4Vj1xLCy1x0aVsW0uwfg/uHKO0/PjemIJjUrlinSANBFJX7ASr5uP9K+QehuVrhbZ3guaYDPtuHvGcQjnrqgAyrK/Mz85vrndd/z+v6sMRsoUSnJnA+slLf6tnNamIp+5gE1nzupCWjlSAXgyzLRrBCEQA7UV1VyIkuFKnSvo7CCTzCQdYa/6co60+4egEWPDvFaDK5RUlDkY3VJWF8uFQMWcyNI1Sx53EYPp2alChG7wJ0aRbrXSQxtXw8LHzk3xBhwSXflYlxGv75fdyEkRsky8IgQI+Z8pUfJY9vwR7QKZ9w4oHnIa7fKcbIjIG+L2t74SvvtabHG7PeXBl4/KYtxOimwpAEztXYlTL9nAGqoWOHqVk3C9oyThu759/2DdEuc+wlBELB4vLq7UUWDwWBKC1ItC9Xk63vhhs9XGrq2X/BDtL/XVE+pgJGd6mPGpiOK74f3Lb1FlvxsaceoNIr6JwD0aKac0lWtf+m5aVzcrSF2ZOTZlstrbhnUAp8s2oPcM0WKVnPF58OhjwVZkBngNwuy5H+oVB3PDfy+fWQXta9fJVl5vSqd7qe5RSoIoGctF0URHRtWUy2MoWS1KSwuDXndqVFgO69NvSoRW3vRjNH5xGxv69PCnO8zj8TRzGaJ/13TI+S1fKwqDg5Aj41qB0DfZ1nePqWx7d8+SoGqxeonhuGZCzvgLZVy5PLnJi8worRQe/+qs8r+fmtsd/x570B2gnpEcmI87hsWSImotIvoFxWALMgMSE6Mx6nCEjw6op3XohiiVuUkLHp0SIjPlKv4pHM4hZpiU61iIk7mF0cc79CwKn5Ze1A1dRWP3DG4JWpWqoDLzmoc8d6w9vVw4kwRAP0tx9V7cyKOPTdtc9nf654abmg78s0ru6JlnejK5rBu/3FD5yn5e2v5+5VZ/zm06BjFf7t6/CO1h+oVA7s9ek9Ynk4xOTE+qnzea1VOwvX9m6u+P7xDPTw1NTBOdWtSPSIlo5zRXRrgzm+Zi8gVkS4W/uiftM5mwPe39cX9w9rgjsEtvRbFMI1rpHgWxBRN2+BWUMsZrTZo3DSgOabdPUA/ypwjkhLicW2/VEUL8KjO9ctmVz0dTCnYRT7ZVE+pYEhBvqR7Y3Rp7O9gUKsoNav5GqWopfP9HDxrdv69smd5fl8jGT5iBflCQ/I3lp6t2jNuV78KXri4U1kmm1ikgSyzUaxOd1LzEBG5YFWa63h8TKQgM6BV3Sq4d1hkmc5oYWwv48nhjbA8LZvp9aIFtSAQQRCiwofyh9v6oUnNiriwa0PD9r36VSOt5v6wPfCD0vNavkfdohUvCPjtzv74/AZjRTV4RP6da1XSzzJx6VnlQVXXyaqkxjpyPaZ25cBzvKhbQ1zdpynGjwykcHz6wg4hn5l53yBc07eZYqaCmIRHzc8Fykqxi5GLKZ8YkElBJvQ5ryPbnJUDWztbWIN31FKc+cmFwgq9m9fEwkfORWJ8nGKRBSWeuaiD5vuEPmZT6okIbAv7uRSw9J2fuqADpt0zQPW8hY8Mwa//OTtkcUruGeXIn4T0TJMS4vHiJZ1RM7jw8Mt2uVfIx7hYelTyMd6IiwWPHl3kg0zoksA44kUtGC0WmHnfQKQoZB8Y17sJLu/RGAt3HvVAKvcxWp2qikLO51iaZLTobbCQiNnHFQ0Kj/QNBrWpE7LdHU6TmiloUjMFa/aV+7pf3z/VWeF8hHxxpVbKvKC4xC1xfImai8XTF3ZAi2BcxPiR7XwX7K9HmYuFqOBioaBS1DCw0+M2ZEE2wKfX9fRaBE9h7atsdGKPRtrVr4rkxHhsf2FEyPEJl3bxRQEQVkiLJLVJV4toUOCsEL6wKigyppiYfVxqrj6+ouwrGDNLxcsekp8t5yz44bZ++PLGgHuNvCmolTKv6POcvU5xYdeGuP7sVNVg1xv6N8c5bQK7qbef0xK3DvJPDJMhZKk+I1wsgh30nXHdy/qbV2lntSAFWYFxvct9btMnjsZQjXKasUCFhCiYMDkjvCoX4Hx1OZ54+bIueHREu5heLJll6fihIa9zThcZ+pySwnuVyeqPfiNOMLZDIREViwJG9G5eE4OCipt8TFIdn2THR3Ssr3xODPLuuO545qKOMR+kp4SUP7ppzRSuXQtJQVagksHqQLGC3IL84iWdbF+PfPyUUZqk5Tk0o4kalSrgjsEtDS8KGqnkSY4lqoVlV6hssCKj0rO7vEdk+r1oQh5BbwRSkNlQr2qS/kkxRkjBFM+kcJ8yH2Qx8ntf268Z5j00GN1kmXLIB9kniABu6J+KnFOFXovCBXIFuaGGP59Rwid6IoB8jh7Upg4W7MhCIlU8wIx7B6J+1WR0f36W16I4zp/3DMSKPcayvLSuZyyvs3wRUjExHo+MaIvuPk7hZgT55GwEUpCtIz25lArx+O+o9p7KwiN+ziduh/I4k8ggvWOnCtEzNbB7aDRg2wto9lXh6Qs74q2x3b0WgwsS5SVFaR5xDLlv7YVdGngoCV+0b1A1JICjsKRU42x/06FhVc0CBHKKDezdNqgWWgyoekoibujfPOrdee4b1gaA8cwwserXzpIx3RoZykkea6iVo452mgd9igO+66H9q26V8nGJ5x1lUpAViNEFnypyCzK/Tdn/xClYsagpRpKWdcprEbigpES/dYS3qPAcv7efE2WBQUFGdW6A9ImjdUudS5AF2Tq1KwfcKhpW86gyK+dcEKPGjn4ta2HmfQNxTd9mERZkeZVFCR71LnKxUIBHU7+XhCjIZGlxDPkcTc+Z0KNmZf20SOEjWbhCTFXjAljJpkIEOL9jPXx4zVkYFuPB7OoYCHSMUtrVV858YmT3iwfIgkzoYseC3K5+FbbCRDHyVFPlOST9MZAQ7vPEaH1/z8O5+S5I4n+Udm8IYwiCgBGdGlDlPEKV8FiaEpmCLO1qNa2Z4qZIhqAWTegi90E2uwCWfPv+3bcZS5GiErl1gdRiQo8UB7LttK0Xmwva+Biz7BHuQU0LaForVPmVW5Av79EY6RNHU6EQv0BGu1ASQizI1nr7kxdQyWA9QsvdBqCmSJjhRoMBfmrUjdE0XeRWRzgF6ccB/nf1WWV/G4mf4AFSkAld5P55ZlfDUjeokBDdTe31K7ravoZ8l1fakaLFGmEGslZZg/oZ4RSCgutcLDKyc3mwYlGpPzIRRbfWQjBBybIp5/2rzlI4Glv0aWG/IlxcyHOO5aGUYIXZKlWxqijG6NcmCE8ooSA9IloIifBW0NuqJKv7QsZKkBmL6GR5LtakoMX9rnNb2b4uET1UMBkIJaXgIrSpmcKf/yMRHZCpI5JicrHwL7Gi1BkldIuIursS8jXEwNa1bV8jPk5A+sTRuHMIKchEOY+MaKv5vtSELu3eCID5yPBY9cWlgpXqDG5bB/1a1PJaDN9Cbk+RaBnVeMIfUrqM0hTRvWl1rN133HVZeCQ+TvDNFolbyBcOT1/YEcPemG/6GnGUxYLQQdrNubJnE8X3pSbUrkEVfNG9N3qarOIVq7YBKumuzuc39PZahKgh1pXls1vWQn5RCZpwmNJNCRoVDPLtzX29FoELBAH49T9nRxyTqOqTlSFr5M/A7CDYvkEgmTqVuyX0kHZz9IJeRRE4p00dw5Xk5J+LRSgPMuEUtOtazre39MUv/+nvtRiGIQXZIBUrUI15ILCF26i6euDPz3ecjfg4oczSFSsTrqDytxEklx4qd0voobeGsusLH6suFgThFGT38C+2FGRBEK4QBGGzIAilgiD0DHvvv4Ig7BIEYbsgCOfbE9NdYkWps4LeBCwIwK4XR2LaPQMAxM6EG+KnbXJELA02OLl+TG0wEiplW45T/YraHUEQRAC7++GbAFwK4CP5QUEQOgAYC6AjgIYAZguC0EYUxRKb9yM8RhAiFUAhzH4qCELZsViZcOWPxKwh+LFR7YPXkH8wRh6cCWpWSvRaBN9gtPWEr+Wo1RGEc5C7hb+wZUEWRXGrKIrbFd4aA2CKKIoFoijuAbALgG88/WPF6mkFAdouBNKEG2vbSmFLBMVzKoW56bSsUwlVkxMwuG1dByWLHshHW5+yCowWhrBG1SvikfMexE05AAAMq0lEQVS1s2QQBGEOGrb8i1MRVY0ALJO9PhA8FoEgCLcCuBUAmjZt6pA42ozr3QTfrdjvyb39RsCCrPF+2OtYWWqEulgon7Pi8WEhr+c8ONhBiaIPFrmm/Y7uE1A5Yeqd/ZF5skDzo4vHn2tJpmihXf0qtFglmENWY/+iqyALgjAbQH2Ftx4XRXGq2scUjinqSqIoTgIwCQB69uzpiT414dIuIQpyrLgFWCPypw3N4CConBXdGHGrMJNRgNpgJKQfl6PWPm4b1BL7sk/jqj6hxoauTaq7IJW/mXnfIK9FIKIQIXR7kfARujO2KIrD9M5R4AAAeaLOxgAOWbiOJwwhK4IqghC5ItZS5mKl6AprK0FsPDVzUJIPlM22au2jZqUK+N81PdyThyAIIkpxKs3b7wDGCoKQJAhCcwCtAaxw6F7MGdaBouXVEMr+p/E+ylfNMaPo2ciDTBiDfJCd45aBzb0WgSCiEhq2/IvdNG+XCIJwAEA/AP9v7+5j7KrLBI5/n07bGaCv1JaWvlCIFCzU9aWw4LpIoGkrGiBmjY0xVjFxEzXBGF9omhhxwV012RhjNoaoCcYXfCWwbkS7667+sWpFQXmtFHFjA1kwvr8i7eMf9zczp7d3Zu7tnZkzZ+73k5zcc3/n3Ht/c+e59z7nnN/Lf0TE1wAy8wHg88CDwF3Amx3BYn6IiA4937OyfWxt1uo0F7S/J/2OaTwgJ957YoI8aJ8qaX7xK6xZ+uqkl5m3A7dPsO1m4OZ+nn+u+fe3vNgRLphiFIv2rQPydk138nbF+TbzaedEKuOm+wDKAzJp5vk5a5bBnBf4JG3bsLzuKtSuY+/Lyoe+fZi3Qfk+OK4fRkTfba+dufFEy09xHOTx47BB+WRJzeYoFs3lVNPqSaeJQpITLx1N9JWwbvnIjNSrbtW/f1A6Js62PRdvnHqnec4fW6lZnP+puTyDrJ7EBD/RQefPfjVZ/N8brmDJiCGnk7N4yOP5UR6DSc1wfH7sB7dJzFbUk05NbatJ8HgTixN3PHPFKTNVrdqNLBxvEuGEFjPD9xX+ZmOrmZcTWkjNYErcXCbI6luriUVAZsfmF4NggR3IZpxvMVxw5nIe/qfdjCyyjbrUBGetOnVs3Ss/zeI1S/Wk41TTWRn/uO3WLwRNF4d5azE5lppjeOEQK05tdTD257BZTJDVk+jQCrnTOMjjo1j4laDpYX48/ez0J8288RNG/h42iQmyetK5DfL42b3RH9xB/uH949POiSNJarH/RDOZIKsnE37O/fyP+dNfjvLKFzok2XQb5IOumXLe2qWAY7xLs8Hzx81igqyenTDVdIeJQjptGxSZ8L5XbOO+9+ysuyrzyvBCv66m22VbVvPNd1zONc9bX3dVpHnLPjnN5C+OetLpLF7SoZPeaBvkAfxCWHnaIoYWBEtHnPltOjlSyMw4a9VpdVdBmtfeufs8wNlAm8Zh3jS9zGFsbyZJGvOqizbxqos21V0N9cgzyOrJRBOFjI1eUTLkRWXWs2UDfsS899Kz6q6CJEnqkQmyeraobcrfVhOLMnpFSZTXLh/hxqsv4BOv2z7LtZtbNpXL10uHu7tY84+XnTOT1ZEkSV0wQVZPAhhaEBy6aTc7t54BtNoZdzqzvPdFm1m3fP5OL92NpSOtxHjnBWu72n/fVc/hp//yspmskiRJmoJtkNWTY6XT3fDCIW669kJWLRnmivPXnNBJTy1D5cjh2CD2VpQkqaE8g6yePPbz34+tr1k2wj+/YhuLFy4Y65hmB7XjhyMbWmCCLElS05ggqydT5b+mx/CsJcNj61vPXAbA5eetrqs6kiSpRzaxUE8mOhHaPv7xIHrNJZu4457HjyvbcsZSHnzvLk5d7EdNkqSm8Axym9c6LNekpmoqMMjTAd907Tbuu3HXCeUmx5IkNYsJcpszVwz2qAtTWTHRuMaDmxdLkqR5xgRZPTk2RRMLSZKkpjNBbuNgA5N7+ujRjuWjo1ckvoGSJKnZTJDbbF51at1VaKTRznkTnWGWJElqChPkNi/dtq7uKsxJl56zCoChBZ1DxiYWkiRpvjBBVlcu2rwSgDOWDU+6X9pGRZIkNZwJsrpy/Y4t3PXWv+f8tcs6bn/1324C4LRhhzSTJEnNZjajrgwtiAmTY4C37zyP66/cwuKFHnNJkqRmM0HWtIgIFi+0JbIkSWo+T/dJkiRJFSbIkiRJUoUJsqRGuvW6i+uugiRpnuorQY6ID0bEwxHxo4i4PSJWVLbti4jDEXEoInb1X1VJGveSLavrroIkaZ7q9wzyAeDCzHwu8GNgH0BEbAX2ABcAu4F/i4ihPl9LkiRJmnF9JciZ+fXMfKbc/Q6woaxfA9yWmX/OzMeAw4DXQyVJkjTnTWcb5OuAr5b19cDPKtuOlLITRMQbI+LuiLj7qaeemsbqSJIkSb2bchzkiPhPYG2HTfsz846yz37gGeDTow/rsH/HOYgz8xbgFoDt27c7T7EkSZJqNWWCnJk7JtseEXuBlwNXZuZognsE2FjZbQPw+MlWUpIkSZot/Y5isRt4F3B1Zv6hsulOYE9EDEfE2cC5wMF+XkuSdjxnTd1VkCQNgH7bIH8EWAociIh7I+KjAJn5APB54EHgLuDNmXm0z9eSNOA+tvciAM5ds6TmmkiS5rMpm1hMJjOfPcm2m4Gb+3l+SWr36Puu6tjJQZKk6dJXgixJs21ogemxJGlmOdW0JEmSVGGCLEmSJFWYIEuSJEkVJsiSJElShQmyJEmSVGGCXCwdcUAPSZIkOczbmO/t38GxsZmyJUmSNKhMkIuRRUN1V0GSJElzgE0sJEmSpAoTZEmSJKnCBFmSJEmqMEGWJEmSKkyQJUmSpAoTZEmSJKnCBFmSJEmqMEGWJEmSKkyQJUmSpAoTZEmSJKkiMrPuOoyJiKeA/6vp5Z8F/Lym19b8YAypH8aP+mUMqV+DFkNnZebqThvmVIJcp4i4OzO3110PNZcxpH4YP+qXMaR+GUPjbGIhSZIkVZggS5IkSRUmyONuqbsCajxjSP0wftQvY0j9MoYK2yBLkiRJFZ5BliRJkipMkCVJkqSKgU+QI2J3RByKiMMRcUPd9dHsi4hPRMSTEXF/pez0iDgQEY+U25WVbftKvByKiF2V8hdGxH1l24cjIkr5cER8rpR/NyI2Vx6zt7zGIxGxd3b+Yk2niNgYEf8dEQ9FxAMRcX0pN4bUlYgYiYiDEfHDEkM3lnJjSF2LiKGIuCcivlLuGz/9yMyBXYAh4FHgHGAx8ENga931cpn1OLgMeAFwf6XsA8ANZf0G4P1lfWuJk2Hg7BI/Q2XbQeBSIICvAi8t5W8CPlrW9wCfK+unAz8ptyvL+sq63w+XnuNnHfCCsr4U+HGJE2PIpdsYCmBJWV8EfBe4xBhy6TGO3gZ8BvhKuW/89LEM+hnki4HDmfmTzHwauA24puY6aZZl5reAX7QVXwPcWtZvBa6tlN+WmX/OzMeAw8DFEbEOWJaZ387Wt8Yn2x4z+lxfBK4sR+W7gAOZ+YvM/CVwANg9/X+hZlJmPpGZPyjrvwUeAtZjDKlL2fK7cndRWRJjSF2KiA3Ay4CPVYqNnz4MeoK8HvhZ5f6RUiadkZlPQCsBAtaU8oliZn1Zby8/7jGZ+Qzwa2DVJM+lhiqXHZ9P6wygMaSulcvj9wJP0ko4jCH14kPAO4FjlTLjpw+DniBHhzLHvdNkJoqZyWLpZB6jhomIJcCXgLdm5m8m27VDmTE04DLzaGY+D9hA62zehZPsbgxpTES8HHgyM7/f7UM6lBk/bQY9QT4CbKzc3wA8XlNdNLf8f7ncRLl9spRPFDNHynp7+XGPiYiFwHJaTTqMv3kiIhbRSo4/nZlfLsXGkHqWmb8C/ofWZWpjSN34O+DqiPgpraaiV0TEpzB++jLoCfL3gHMj4uyIWEyr4fmdNddJc8OdwGhv3L3AHZXyPaVH79nAucDBcvnqtxFxSWmX9dq2x4w+1z8A3yjtu74G7IyIlaV38c5SpgYp/++PAw9l5r9WNhlD6kpErI6IFWX9FGAH8DDGkLqQmfsyc0NmbqaVx3wjM1+D8dOfunsJ1r0AV9Hqdf4osL/u+rjUEgOfBZ4A/kLraPgNtNpW/RfwSLk9vbL//hIvhyg9fEv5duD+su0jjM9UOQJ8gVZHiIPAOZXHXFfKDwOvr/u9cDmp+HkxrUuKPwLuLctVxpBLDzH0XOCeEkP3A+8u5caQS6+xdDnjo1gYP30sTjUtSZIkVQx6EwtJkiTpOCbIkiRJUoUJsiRJklRhgixJkiRVmCBLkiRJFSbIkiRJUoUJsiRJklTxV/wOsaHG2vh6AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "temp = float_data[:, 1] # temperature (in degrees Celsius)\n", "plt.figure(figsize=(12, 5))\n", "plt.plot(range(len(temp)), temp)\n", "plt.show() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The plot of the first ten days of temperature data (since the data is recorded every ten minutes, we get 144 data points per day):" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dd5gb5bX/v6/6Sitt7971ru21jXu3MZhqQg0tIYEklEAugfTk/pJAfHMvNSGBJDchN4UEQi41XFoIHWzANGNsbOPe196119ubpFV/f3/MjHakVdeM2p7P8/ixNKPRnN2Vzpw57znfwzjnIAiCIAoTTbYNIAiCINSDnDxBEEQBQ06eIAiigCEnTxAEUcCQkycIgihgdNk2QE5lZSVvbm7OthkEQRB5xebNm3s551WR9uWUk29ubsamTZuybQZBEERewRg7Em2f6ukaxth5jLG9jLEDjLFb1D4fQRAEMYaqTp4xpgXwPwDOBzALwFWMsVlqnpMgCIIYQ+1IfhmAA5zzQ5xzD4AnAVyi8jkJgiAIEbWdfAOAdtnzDnFbEMbYjYyxTYyxTT09PSqbQxAEMbFQ28mzCNtCxHI45w9wzpdwzpdUVUVcHCYIgiBSRG0n3wGgUfZ8EoDjKp+TIAiCEFHbyX8MoJUx1sIYMwC4EsALKp+TIAiCEFG1Tp5z7mOMfQvAawC0AB7inO9U85xEfrHhUB8sBh3mTirJtikEUZCo3gzFOX8ZwMtqn4fIPQ712HHWr97BiinlePLGk8ft3981gisf2AAAaLvnwkybRxATAtKuIfDBwV586/FPEAgoN0DG6fHhi6ID33CoH/LhNGt3d+Gv7x7COb9ZH9zm8QUUOzdBEGPklKwBkR2+9JePAAB3XzYXJUV6Rd7zx89sR8+IO/h8aNSLUrMBAw4Pbvj7eOmKXZ3DWNBYqsi5CYIYgyJ5IojT41Pkfdw+P9bu7kJdiQl3XDIbAHCo14GnNrXj3tf3hrz211+YDw0D1u3uUuTcBEGEQpE8EcThVsbJbzzcD6fHj/uvWgi9VogjfvLsduw5MRJ8zUXz6nDT6VMxp6EEz205hue2HsP3z5kOxiK1VhAEkSoUyRNBHG6/Iu+zbk83jDoNVk6tRG2JCQCw58QIamxGAMClC+rx+y8twpwGoaLmsoUNaO8fxaYjA4qcnyCIMSiSn+B83NYffJxMJP/u/h6Y9FosbS4PbuOc44Vtx/G/Hx7Baa2VKDJo0VpdDL2WwevnuOfyeVg5rQI6TWhscc6sGgDAFX/6EI99bTlOmVYJQKi+qbaZ8M3HPoHVpMOKKRX4wpJGFBm0AIAdx4YwpcoCs4E+xgQRDfp2THCu+NOHwcf2BJ28y+vH1Q9uBBBa+vjclmP4wVPbAADXnNwMAGCM4cFrl2JX5zBOm14FrWZ8OsZq0mNhUym2HB3El//6EW46fSqsJh3ufS00f//KjhP4rxd24pEblmHA6cV3ntiCM2ZU4eGvLkvqZyaIiQQ5+QmMyxuannF6EkvXvL23e9w2f4Dj9n/tAgA8cPVinDmzOrjvtOlVOG16bF2iJ/5tBdbt6cadL+7Cn945GNy+eHIZWquL4fVz2Ip0+Nv7bcELjGBLDz45OoBFTWXgnCPAEfFCQhATFXLyE5hBpzfkeaKR/IcH+4KPXV4/dBqGP68/hKFRL1afVIPPzK5N2haTXosL5tbhrJnVuOj+93Cg245Hb1iOU1srQ1530+lT8cr2Try2sws/ueAkXP/3j3Htgxvxb6dNwT8+boc/wPHCt09BtdWUtA0EUYiQk5/ADI56Qp4nkpPnnOPVnSeCz0dcPry680QwtfL7Ly1MyyaTXos3f3B61P01NhOuO6UF153SAgD493Om45Znt+PXb+wLvuatPd344tKmtOwgiEKBnPwEJjySdySQrnlzdze6ht1orjCjrc8Ju9uHnceGAABXr5gMk16riq3RuHJZE6ZWF6PcYkBLhQXz73gdn3YM4YtLM2oGQeQsVEKZI/j8AWw+MoABhyf+ixVinJNPIJLfLJY5/vDcmQCAM+97G09+3I7lLeW489I5yhuZAEubyzG1qhgaDcO8SSXYLl50CIIgJ58T7Dg2hGlrXsHn/vgBvvXEJxk775b20Lr0vbJmpWgc7XdgSqUFlcWGkO2nTquMckRmmdtQit2dw3D7lKn5J4hU2dcV//uUCcjJ5wAX3f9e8PHeE/aMnffP7xwKef7egV7444iUHe13orHcDKspVOPmovn1ituXCvMnlcDr58E7DoLIBs9t6cBnfrMeb0WoRMs05ORzDH8gEKLYmCmays0AgO4RV9TXBAIch3scaKm0oL40tHqlucKsqn2JcsaMapSa9Xhm87Fsm0JMYDYeFoKM9n5nli0hJ591wiPOAac3Y1GoST/25796xWQAsaUN2voccHj8mFVnQ6nZgBe/fWpwX65ozhQZtJhVZ8Oh3szdERFEOFI5clGGCxEiQU4+y/xr29jI2yqrERoGrN/fm5Fzyz+ANaLGTLTFV3+A44dPfwoAQc2ZOQ0l+K/PzsJ1K5vVNTRJmsrNORFBERMX6XuUaIOhmlAJZZbZ1jGI+Y2luGppI5a2lOO7T27BhoN94Ku56tGx/AMoLaRGcvJefwDXP/wxNh8ZQFO5GSfVWYP7virWq+cSjeVm9No9GHF5x60dEEQmCIgp1z0nhrNsCUXyWaV72IUtRwcxt8Em1HtXFeOsGdXY2NaPC3/3Hrx+9aYlBQIcbl8AF86tw12XzkGxUbjeR6qV//BgH97d34slk8vwzM0rcyY1E43Z9TYAwP3rDmTZEmKiIk0629eV/bQhOfks8uhHRwEAF84dq0z51lmtuP6UFuzqHMbdL+1W7dyjom7N3Ekl+MqKyTDqhNRNpDF8Umnln65ejCqrUTWblEJSsXxg/SEqpSSyglv8HuVC2pCcfJbwBzh+t3Y/AGBOgy243aDT4CcXzMTKqRV4+IM2fHBAnfy85OTNomyvQSd8FDz+8U5x85EBNJYXobI49x08AOi1GvzPlxYBAP73gyNZtoaYiEjif90j7oQ1odSCnHyW2Hl8rCszPG+s02rwl2uWwGLQ4sH3DoNzDp/CqZtRMS0jyRAYRSfv9oaeh3OOTUcGsHRyOfKJU6ZVAADueXVPyKxZgsgEcoXXP8tUVbMBOfks8Z4YoW/6j9UR91uMOnx3dSvW7unGwjvfwLQ1r+CdfT2KnT96JB/q5B96vw29djcWNuXXkO1SswHPfWMl/AGOpXe/ic6h0WybREwg3L4ALpgrqLFGSoFmEnLyWWJP5wgaSmOnQG44dQpWTCkPasz85NntCMTpSE0UKZIvCovkwz+QnxwVavbPOqlGkfNmkoVNZbj9YmGQ+KMbKG1DZA6XN4CSIgNKivTj5jZkGnLyWWJ353BIKWIktBqGx7+2AhvXnI27Lp2DY4Oj+MPbylSMSJG85OSlSN4d5uRtJj0qi41oKC1S5LyZ5tqVzVjaXIaXt5/I+peNyDzDLq+qVWrRcPv8MOo0MOk1cHkpkp9wuLx+HOp1YGatLe5rNRqGaqsJX17ehOk1xXh7rzIpm2AkL6VrtFJOPtQRDo96UVKU3+0U165sxuFeBz57/3sYcXnjH0AUDPNuex1f+etHGT+v2xuASa+FSa+FK8sVXuTks8CBbjv8AY6T6uI7eQnGGE6eUoHdncOKpGyCkbzo5BljsJl0GBoVnGB7vxPNt7yEl7Z35n1D0UXz6nHbZ2dhf7cd6/dlppuYyB0+Otwf/0UKMuzywuMPwKTXwKTTBgOqbEFOPsM89N5h/L//E4Zdz4yTrglndn0JHB4/2vocadsRnpMHgEqrEb12D9w+P65+cCz6kRx/PvOl5ZOh07CQqiaisFFq/SpZVv/qHQCAUacV0jW08Dpx6BwaxR0v7sKeEyOY21CClgpLUsfPEjs5dxxPv1XaGRbJA0BlsRG9djde3XECbX1jTRx3Z2kYiJIYdBq0VFqwvzv7HYgTBa8/gD579spXw9eXMkW3WLKr1zIYdBp4KF2Tn9jdvoQ+wA63L6hf8dEh4bbx2pMn48HrlkCjSU4eYHqNFaVmPV6XzVhNFVekSL7YgF67G2t3d6PCYsCr31uFnbefi5U5MhAkXaZWFeONXV20AJsh1jy3HYvvejNrJYTZ+DvL5zEEOIdRp6USynzlc3/4AIvvejP4/EifAz99fse48X03P/YJzvvvdzHg8ODxj46iqdyM//rsbFRbTeFvGReDToPlLeWKjLdzRnDyVcVGHOxx4JUdnThjRjVm1tpgMeb3oqucxZPLAAB/eIs0bTLBc1sETf+2Pgc2tfXHHUijNNlY8JR3txq0GiGSz0J1jxxy8imyVxztdf5v38Wg04PT730bj2w4gjPuexvXPrQRAw4PAgGO9WID08I738DGtn58eXlT0hG8nMpiI470ObF2d1da9o96/TBoNdBpxz4CjeLgEK+f4+IFuTHpSUmuP7UFxUYdXt+V3u+OSAxJD+kfH7fj83/6EH97/3BGz7+9I/PrL+/LZEgsRh0MWg1F8vnI5iNjq/W7O4ex4I43gs+HRr14Z18Pnt1yDBsO9Y079rpTmtM691XLmgAA9762N633cXn9IUNDAGCybI1AmhRVSGg1DN8/Zzr2nBjB4d70F6+J2JQUCVVZD74nOPcjfZkT6+Kc459bx2Y1HOzJzFrMNx4TZjSb9BpcNK9ezMmTk8871u0ZP7ex2KjDtv/8DCosgi77gW57cMLTB7echVvPn4mdt58bjG5SZU5DCe64ZDb2nBhJSxPD6fGFLLoCoSP8yi2G8EMKggvm1kLDgGc2d2TblILHVhRaeltsylzqb92ebry0vTP4/Gyx4iVT/PWapSgyaMnJ5yubjwxg3qQSvPa904Lb/nz1YpSY9Xj7h2fgpDobnth4FL96Yx8AoNZmwtdPn6pYfvuyhQ0AgH9sak/5PUa9AZgNofY0yqJ3Wwa/kJmkrqQIy1sqcmLAcqEz6glVX9RrM+du+sPWxjKNUbxLLuicPGPsNsbYMcbYVvHfBWqdKxO4vH4EAhzPbenAhkP9mNtQghm1Vjxz80p8/fQpWN4iqDRaTXp8VTYOr6XSklYOPhJWkx7fOGMqjvY5U27ZHvX4gwqUEia9Fp9fPAn3XTE/5weDpMOylnLs6hxG93D0oeVE+oSPvrO7Mie5a4wwWzUT1TZSg+OiJmGR36DVZK2UU0LtcO03nPP7VD6H6rh9fpx539voHBpzClLN+uLJZcGqDYnVs2qAZ4TH910xTxWbplUXwxfgONLnxLTq4qSPH/X6ggqUcu67Yr4S5uU0586uxW/X7sfbe3vwhaWN2TanYHF6/CHpiq5hF7z+QEYieklwT06kwEZpyi16LJlcBq0Y2BkpXZMf/OAf20IcPABcPD969Um5xRAU9GqtSa6rNVHqxffvSjEaHfX4c2KSfDaYUWtFkV6LPeLEK0J5OOdwenyolK3tvLS9Ezc9sjkj5490H5qJtInLGwimaoCxdA3n2em+BdSP5L/FGLsGwCYA/845Hwh/AWPsRgA3AkBTU5PK5iRPx4AzuIBz+cIGNFWYcf2pLXH1XJ68cQV0WgabSrov0gJvX4q5R6fHj3JLfkx6UhqthmFGrRWbj477OBIK4fYFEOBARbERx2UB0toIRQtq4PWPd6rhA3HUwO3zB6uKACFdwzngC3DotdlJgaYVyTPG3mSM7Yjw7xIAfwQwFcACAJ0AfhXpPTjnD3DOl3DOl1RVVaVjjqLs7hzGbS/sxKm/eAsA8Nn59bj3ivn43urpCTnuxnIz6krUk+eVql8e/+hISk0mLq8/YrpmonDmjGpsax/EMKlSqoKkjVRRnJ0qrUhrVZmY9+v2BkJSRYYocxoySVqRPOc88lijMBhjfwHwYjrnyiR/fucgfv7KnuBzaTEylyg1C1+eDYf68fAHbbjh1Jakjh/1Ttx0DQAsECdd7Tg2hJVTC0O2IZeQtJHUDHRiIXeqWg2DP8AzsgDq9gVC8v5yJ5+tG2c1q2vqZE8vA7BDrXMpjdzBA7m5GKmVVey0pdDY4/T4x9XJTyTmNpQAAD7NQlfkRMAptvevmFKO/7jwJPztuqUAgAWNmRkjKc+/PySeOxORvMvrjxzJZ7GMUs2c/C8ZYwsAcABtAL6u4rkU45DYGVdhMeBf3z4VA87s1tvG4sqljXjy43YEUljUcXkntpMvtxgws9aK57ccw9dPm1LQJaPZQCqftBh0uGSV0NexvKUcmVp+lKdrjFGmnqmB2xeWrtHmebomFpzzq9V6bzXw+QMYGvXiqU1CJ+SPz5+J+tKiYBVLLnLP5+Zhy9FBHEhSPtfrD8Dr5xM6XQMAn1s0CXe/vBt9Dk/MWbtE8khO3mwc+4yZ9FoMZihoyp6TDy3TlBogw3sGMklhtjWmwC3PbsfTYqv7pLIiXLqgIcsWJcbJUyvw+MajCAR4wk1X4fNdJyrzJgkpm3W7u6leXmFGvUK6Rt5VbdJnrjFIipxrbaaglIja1TWcc6GEUhbJF4tOPptjJ6lOXuRpmZbJ369fFsyl5TqtNcXw+AI4kUS9fL9diKbKClSfJlGWtZTDatTh02OD2Tal4AhG8rKUoFGnzZjGu0csoXzvx2fKhtSre25JBK3KNiYjbjVJTj5z3b7h5IcnU5nwRoWpVcl3kGaLZlE5cuU963D/2v0JHdMjDjupsk7sFAVjQr38XmqKUpxI8wpMeg1cGahVB4CeERe0GgadVpOxdM2JIeF7NUPWACn104y4yclnlWHxKvuds1tx+Of5JbEzWaYc+as39uF/3joQ99awVxxPVkV5aCxuLsMnRwczliueKIxGieQzUeFid/vwxMb2YP+I1IGqtpN3iIJsFtk6xFgkT+marCJNc2quMOddlUV4HfK9r+3F395vi3mMFMlXWid2ugYATp9eBX+AYxuVUirC2t1duP7hj4MTkuTKq5mK5MOns0k5ebUrXJySk5etQ1C6Jkfod+ZvjlobYbE10rASOb0jbmgYUDFBZQ3knFQrCM3to5SNItzw901Yt6cbxwZHwVioUJhJL0Tyauu4DI2GRs3GDOXkHe7xdy9Fei20GpZRBc5wyMlj7Mpfbs4/Jx+JeBN4euxulFsMES8QE40yiwFVVmNwnCOhDEf6HDDrtSF3xkadBgEeWVdGSYajOXmV7yKkSN4su3thjKG0SJ+yxpQSkJMHcKhH6BitK01+uHYusKpVaMu/8bQp+PLyJhwfGo1ZxdAzQnXhcmbUWLGPnLyivH+gb5wEtlQ/rvaAbbdYI//bKxcAEBytQad++aYUyYeXJjeUFaFjIHOjD8MhJw8hvTGl0oJqa346+T9fvRjv/uhM/OSCk7CwqQycY5w0spwBpwdlBXLXogRzGkrwaccQtrVTKWU6BMKE8uQzg4HMRdRe0ZnLLzJGrUb1dI3T4wumZ+Q0lpvRMTCq6rljMeGdPOccW9oHxw3+yCfMBl1wdF+pKHMafssqZ9DpQZlFHQnkfORzi4TGt/vXHciyJflNeIQePidYmtakdq28lA4yyIaTGDPQiOXw+EMqayQay8zoGHCmpBarBBPWyUuLPx0Do+h3eDA/Q8JJaiMNT44loTvo9AZVLAlhsMvS5jJ0j9A4wHQYDWvdP3VaqLqnlK5R29lKkgbyCVRGnVb96hq3b9zcZABoLC+C189THvCTLhPSyXPOcc5v1uPkn6/FFvEWff6kwnDy0sCCQWdkJ885x+CoF2VmiuTlNFdYsvYlLBRGwyL01bNqQp5L6Rq1I3lJ8VGvkzt59SP5AWfk71VjmXCX3d6fnbz8hHTyXcNuHOi2o3PIhf/b1A6DToMZteqM6cs00uLxscHIOcBhlw/+AEdpEUXycqqsRvTaPVkd05bvxHPeUmmh2mJdY5H8WG7coNPArfLFpc/hRkWEggYplZqtxf0J6eSPyq6o7+7vxYopFXmjVRMPm0mPcosBR/oia8wPiRF+KUXyIZQU6eEPcDiyqBaY74x6BOd65owqfHjrWeP2SzIaPWLHtVpIC6+hOXmt6pF8n90THMspZ3K5GTU2IzYc7lf1/NEoDM+WJMfDotw59bYsWaIOTeXmqLXykj4+VdeEUpLAgnUhwjnH3hMjiiwKSumaG06dEnEiVI1YvaZ2WkxaeA3NyatbXcM5F5x8hEheo2ForrCgm3LymWOrmIf/mjgyr6ncHOvleUdzRQJOnqprQpAWovvsE0vD5kdPf4pz/3s9Xvz0eNrvFZSwjjKMptSsh0GnUd3JeyIuvGpUXXgdcfvg8QdQGWWmbW2JKSmlWCUpKCd/+7924sH3Dsd8zY5jQ3j4gzYAwI/Om4l7Lp+Lzy2elAHrMkdjuRnHh0YjDjMeDKZrKJKXM61aqOk+0DNxmqJcXj/+T5TYPtiT/AjJcEbFjs9ocwoYY6ixGdV38r7xOXm1F14l0b9og8trbSZ0DbuzsuZTUE7+b++34c4Xd8V8jXyKkkGnwZXLmkKu+IVAXUkROI+c+5TUFqV6ekJgcoUFBq0GeyaQhs0/tx5T9P0kES5JlCsSNVb1I1qvPwC9loVJKqibk5dkC6LpQdXYTPD4AhiIUvWmJgXj3RLNt0mdoI/esFxNc7JKXYmQ+3xCnBglR/qQlZCTD0Gv1WBOgw0fHcrO4lg26B4eCwJ+t3Y/HGlqnkvKk8XGGE6+xBRyXjUQnHyoa1M7Jx+voKFW/E6eiNGJrhYF4eTtbh/e2tMTfB7rlqh9wIlSsx6ntlZGfU2+I32g7l93AK/vOhGyb8Dpgc2kg67A7l6UYFVrFbZ1DMZsJCsknvy4PeT5xrb0LnBSJF8cJ5LPxMJruJO3GHUYHlVPCVLq9o2WqqqxZWbRORIF8U0/0G3HTY9uDj6Pplk94vLi8Y+OoqXSEnF/oSBF8gCw+chAyL5euxuVE3wiVDQWTRZ0f3ZMEG15qZfisa8Jd7XeNNMZdrcPJr0mZvqztsQIh8ev6hANT4RIvr60CEOj3rTvVqIhdfuaojh5KfAiJ58itrDIYcQd+QMkLcouay5X3aZsIk/FtIVV2fTaSYEyGvMahMHeuzqHs2xJ5rhwbl2wumwwzfLREZcPxcbYacCxiFa9lI3XF4BBGyoS1lAmlHRGaxJMF6kRLJqTr7YaUWzUYVtH5kXwCsPJh+WXo01hkRZdv7d6uuo2ZRPGGK5eMRnA+LrvXrubxv5FocxiQLFRl1XFwEzRLy4Uzm6wjekdpenk7W5fzEVXYMzJH+1Pv5onGl5/IETSAAAmiU4+3qyFVJGyB9HKR/VaDaZUWWKqw6pFQTj58A9WtCks+7vsOHtmddQ/RCFx56VzsPqkmuD8WgDos7txqMcRtcyLEFJdnUOF7+SvfOBDAMKoOqtRB8aAhz9oS+tnH3F5Yy66AsCCxlKY9Bqs39eb8nniESknP7PWCq2GYf2+nihHpYfUI2CK0TlvMejgdGe+o7ognLw0w1HCHiHv5vMHcKjXjtaawtCoSYRSsx79DuG22OMLYPFdbwIQbh2JyNSXFuH4YGELlQUCHPu6hLvac2fXQqNh4FxQZL3lme0pv6/dFT+SN+m1WNZSgdd3noj5unSIlJM3G3TwBzge2XAEbb3K30U4PX4YdJqYBQ0WozY47DuTFISTB4CHrluC/7xoFgDgthd2jlN8O9LvhNfP0Ro2raaQmVRWhK5hNx7dcASDo2OdnF8RUznEeGpt2etMzBRSL8AvPjc3uCAo4Qukvvhqd/viRvIAsKy5DMeHXKqpUXr943PywNhalRp5eafHFzLbNRJmg051cbZIFIyTP2tmDc4RpU33d9vxzcc/CdkvjfibOoGc/BeXNgIAdh4fDuZbf3fVQup2jUGl1YB+h2dcf0EhIakhLp48VoDwzM0rAUSvTEuEEZcvZvmkRH2pkB8P15BSikh18sBYFVG/CvNWHW4/LBG05OVYjNqIWQa1KRgnD4yp3AHApx1D+O2b+4PPXxNvDyOpxBUqdSVFOKnOhp4RV3CCPTVBxaay2Ah/gAc1fgqRXruQwpN/XxZPLsP5c2rTWnwdcXlhTSCSbyhVt9LF6xufkwfG0pTpVhFFIuFInpx8epj0WnxxSWNwIs1v3twHQBAke1rU6LAk8CEsJKqtRnQNu2EXF3yKI4wnI8aQykt7C1iorGfEDYNWM670uKRIHwwGksXl9WPY5UuoPDdYzqhSFZMnQnUNAFhNQoCjRo2+w+OHOY5vsRh1cHqFHoFUf8+pUFBOHgB+8fl5eOSGZVgizmxt73eG/FEjzWAsZKqtRnSPuGTiURPrIpcskpPqs6vbep9NeuxuVFmNIdougFCKnGq3r9TkE57jj0SNzQQNUzGSj5KTN+k10GlY1BLrdHC6fbDEieQtBi04B868723Mv/11xW2IRsE5eUCoE19z4UkAgN2dwyElleGVOIVOjc2EnpGxSD7eLeVEp8oqpPN6CtjJCw1x49OWJUV6uLyBlDReDvYI1TpTquJ3k+u1GpSaDaqlxDy+yDl5xhisJp16kXycnLwU6Ut3ib4IKrFqUJBOHhhbYP3z+kOq5ODyhWqbEQE+Nl+SnHxsJkq6JlJaRUrfXPDbd5N+T6nJp6E0sdkMgrNVJz8dbeEVEAT6Ht1wVHGhMKfHFzdLUB5W8JCplE3BOnmbmH/bfGQAh3rscV5duFSL03h+u1ZYhJ4IjWDpUFKUmcEW2aLX7sbuzuGQEZgSRWIkerDHkbTuueSwbUWJpQPVdPLRInk5a/d0KXY+zjnsrvgLr+FNiJmSHS5YJy/nL+8KmjW/+eL8LFuSeaptoRFbvFvKiQ5jDI1lReP6LAoFSdpjVWvVuH1G2WJlstrrIy4vtBoWVYUxnJIifXC2gZIMjXpxYtgVt6tbyQrZziEX+hwetFbHbrQMT5Gp8fNHoqCd/Ds/PCPk+WULC2sCVCJIWiESWs34BSkilKZyc8RItxDoFgfJXLmscdw++TD7ZNUa7S6hESp8MTcaDaVFqmgE9drdCHBgdpS5zT88dwaA5H++WEjT1upLYy86l4cNFBnMh0ieMXYFY2wnYyzAGFsStu9WxtgBxthexti56ZmZGpMrLMFa4Gh/9EKHxAY/vhEAACAASURBVMiSp6ncjKN9zqyMalMbaZi0NFRbjtzJJ9qZ6Q9wDLu8GElA0kBOXUkRukfcEUdUpoM9znSqm0+fCgCKdttKUgXxyrPDp7Flqhcj3Uh+B4DLAayXb2SMzQJwJYDZAM4D8AfGWFaSwXNF+dhvnjktG6fPOoYYgklEZBrLzRhx+zIWaWWS7hE3jDpNxNy5PF2TqMbKIx+2Yd5tr2P7saFgHXoiSDMNBhTuPpU6SqN1n2o0DAatJq3O3mjnjJcK1WgYnv3GSqz/4ZkA8mThlXO+m3O+N8KuSwA8yTl3c84PAzgAYFk650qVm06fiqZy84SN5AHg9e+flm0T8gpJX70QUzaPf3QUjCFiWiXEySeolrh2TzcAQUpEl0QqsFLsPFe6iik4gjDGXYVJr1E0kpeUJRPpwVnUVBZsBnvmE2Vn7EZDrVW4BgAbZM87xG3jYIzdCOBGAGhqalLckGUt5Vj/ozMVf998YnqNFT86bwamFPhELKVoqhCcfFufA/MbS7NsjXLY3b6Y2ilajTxdk1gkL68o0STh5CukpjOHsv0IUromllCaSa9VdN6rJDNsTrDRUFoX2905jMO9DtUn1cW1ijH2JoDaCLvWcM7/Ge2wCNsiJjg55w8AeAAAlixZUnhJ0BzhG2dMzHRVKkytKoZBp8H2jiFcsiBibJKXSBK7q6LMN26tLkZ9iQnHh1wJR/IDTi8WNpViUVMZLluY+O9Kqn7pUyuSj+HkiwxaRdUgpbsCoz75xMi+rpHsO3nO+eoU3rcDgHz5fhKA4ym8D0FkHL1Wgzn1tqyMalMTqSz0lvNnRtxvMerwxI0rcPq9byccyffa3ZhZa8VPRZnvRKm0SE1nCkfy7viLoKVFekVr1IOj/1Lops9Eqa5aq3IvALiSMWZkjLUAaAWwUaVzEYTiLGwqw7aOoWBdeSEglSxOKovelSotHjoSjHR7o3TPxsNWJEyjevyjo/ArWLRud/ug17KQ9YVwyi2G4DAdJZB6CpKJ5KV1so6BUTyzuQMrfrZWNXnrdEsoL2OMdQA4GcBLjLHXAIBzvhPAUwB2AXgVwDc555lXyyeIFLnh1BYYdRr8ft3++C/OEzoGnLCZdDHlpqXFw0+ODOCGhz+OKdTm9iWuPBkOY8I0qkO9DryyozPp46PhcMev1y8zGxStnHJ5/WAMMS8s4UyvsWJmrRXt/U78+/9tw4lhF+wqTY1Kt7rmOc75JM65kXNewzk/V7bvbs75VM75DM75K+mbShCZo760CCdPqcCnHUPZNkUxeu2eEA35SEgph+e2HMPaPd04+9fvRH2tNHwjFScPAJPFBe53FZz3anf54tarW8Rh7UoND3F5/TDqNAk3gknUlxaFDPZWen1CgoqoCSIKcxpKcKjXoWh3ZDYZcfvi1rKHV8jEinh7RyQnn9ognpe+swrLW8rx4aG+lI6PxEgCIwili8DJP1+ryDld3gBMCco5yKmxGYMdyABw/cMfK2JPOOTkCSIKkmyuEvXy/9x6DIvvfEO1uaaJYHd5k+pKBYBZddH7S6RF08oUB8MXG3WY31iKE8MuxbqLHQk4eWlwTrL6PNFwef0pLbpaTXrY3WMX0Rk1sbVvUoWcPEFEYXK54OSP9KXv5L/75Fb0OTzYKw7RzgaJDtq+5/K5wceaGB5C0tyvtKQunVFVbITHF1AsR253x58zm2xaJR4uXwCmFMonzQZtSOftHZfMVtKsIOTkCSIKUlPU0X6HYu954yObFHuvZJFExOJx+aJJ+OXn5uH8ObVR68k55/jR058CEIafp4rUbPbR4f6U30NOIj9jmcKD7F1ef0rpGrmdV6+YjGpb/KlaqUBOniCiUFKkR7FRh+ODymnLdw1nb+LUiDv+oiQg6B19YWkjrCZdsGU/nMO9Yxe+dOSr500qAWNQ7A4nkbuVK5YIarS1CjlVl9cPYwpOXv57S6YyJ1nIyRNEDOpKTOgcUmcWaSbhnMPuTk4p0mzQRW2KWr+vBwBw6YL6tOwy6bUoLdKjx67MhTQRJ6/XanDVskb4FVoHcHsDMKXgpCUNGwBQOIMUAjl5gohBXViZW77i9PjBeex2/3CKDNqgLouE1x+AxxdA57ALei3Dr7+wIG3bKoqN6BlJ/w7HH+BwevwJ3a2Y9Fq4FJI2cPlSS9csmDSmi1Sk4jAfcvIEEYM6m0kRJy/Xi1GrszEWiagzhmMxaOH1c/TZ3cFKmrN+9Tbm3vYauoZcqLaakhIli4bT7cNrO7vwl/WH0nofSR45kbsVs0GLEbdPEblftze1hdcSsx4fr1mNr582BV9b1ZK2HdEgJ08QMagrNaHX7oYnzXK7gCw14FJQATFRRhJQZwxHii4X3/Umltz1JgCgvX8Ubl8AXcNu1JYok9O+/ZI5AID/fnNfWqWUkgJlIpG8NKZw/u2vpz3PN9VIHgCqrEbcesFJwZnUakBOniBiUFdiAudI2xH4/GPOS0kFxESRIvlkcvLhDlf+vGvYpdjC5TmzanDr+TPh8PjT+t04ElCglJA75XWiJn6qpFonnynIyRNEDOpKhMWxdFM2chGu57ccy/howee3CAMqio2JR4ynhkkSD7vGFmEP9TrGzQ9OBymNFEvvPh4jSaSkimQ6+G/v7U7r7+FKMV2TKXLXMoLIAaThzOlW2PhkTv6ul3bj+a2ZmQoEAJva+vHwB20AhFb6RJlZG9rtOhyWv64tUW5+sBR9j7hSd/KJDAyRaCgdq2x5bWcXXtt5IqVzenwBDLu8KadrMgE5eYKIQa0YyadbK+8LBCBfo1SiizZR2mTnku5MEsUii3iHXaFOXslIXspJj7hSXwgdFC9CiTj5U6ZV4vurpwef70mxTn/viRFwDrSqJEmgBOTkCSIGxUYdKiwGHO5NT1fe5+ewySR+9drMffXk2unJDnZ/8Tur8PnFQvPQgCPUAU8qS+6CEQsl0jUbRKGzckv8jla9VoPvrm4NPj+RYjpOsrdeoUVoNSAnTxBxmFVvw87jw2m9hz/AQxY9RzO4+NovOufrVjYnfWxLpSV4XMeAcEcgVabMqI0uXpYs0u8mnXSNxxeAzaRL6Q7j2GBq6bhRr2CvPMefa5CTJ4g4zKq3YX+XPa0ySl+Ao0qmu+5QaUBEJFxeP0qK9Ljt4tQEsKQhI1JK4+7L5mD7bZ9JqhwzHtJ72dNw8i6vP2lt+y8tbwIAvLu/F6+mMLxEqgZKR9pBbcjJE0Qc5tSXwOMPYM+J1KN5t9ePKVXF+OXn5wHIbCTv9vnT0kaR0kzS4m2pWR9Xlz5ZSkXRsH5n6oMzUhEK+9llc/GTC4SZtzc9+knS5xxz8hTJE0TesrS5HIAwEi9V3L4AjDoNvrCkES2VloRnqCqB2xdIOhcvJ3xcYEmRsiqOgBDJmw3alHPjQOqljNec3Bx83NabnOKodLGmdA1B5DE1NiNsJh32pzHUW3DygiMwG7RwZnDalHSBSYcvLmkMPo41IzYdZtfb8OKnnfD6U0uLDY2mVspo0mvxyA3LAAA/eGprUsdSJE8QBQBjDNNrrNjflY6T98MoRpkWgy6jXa9u79gFJlXqZXXlpWZ1nPzliyah1+5OKZrfcnQA248NYXOKd1vzGgSxsE+ODiZ13Ki4tkIdrwSR57TWWLGveySlzkh/gMPr58Fo2mzURpXwVQO3z59WugYQdN8l1IrkpQalEylISLy7XxgGnupIvxLZhSuZv7HT40eRXquIUJtakJMniARY2FiKQac3pVJKqSpHiqZtJn1aC4ypnD/ddM0ZM6qCj9Wq8a8rkbqLk3fybgVE33560SwAsYeXh+P0+nM6VQOQkyeIhFg9qwZaDcOrO5Jvf5ei9iIxXTO5woz2/tGUUwvJ4vYFUppcJIcxho1rzsYzN5+skFXjkVQtj6dQsy6lS+QXo2SRLjLHk5CwGPX4c3rRFSAnTxAJUW4xYHlLOV5JoZZ6TAFSSAlcsVhYxFy3p0s5A2Pg9gVgUCD6rraasHhyuQIWRcZq0qPGZsS+FCQGJGmgP3x5URrnF2rdk1kvcXp8sORwjTxATp4gEuaCuXU42OPAzuNDSR0ndXFKTqSpwoxamwndGZr36pEt+uY6s+ps2NWZQkrM74eGpdeUJHXyJufkKZIniILhM7NrAAAPvdeW1OJccGCHTNag3GJAvyMzeXm3LwBjBrVy0mFWvQ0Huu1J59g9afYCAGO17hsO9SX89x31UE6eIAqGaqsJN542Bc980oEt7YmX2gXTNTIt92KTLi0xrmQQcvL58VWfVVcCX4AnXa7q9fO0U1JSJP/Htw/in1uPJ3SMk5w8QRQW3zprGgw6DR5893DC0Z4knysXKLMaM+fkPb706+Qzxax6QfRs+7HkUmJCV296P6M81fO9f2xN6O876vWrOoRbCcjJE0QS2Ex6fPWUZry0vTPh6phIQ7QzGcm7vOnXyWeKyeVmlJr1eCXJKiYlykQrikPlGj463B/3GKfHB3MODwwByMkTRNLcdNpUAGP65fEIX3gFBK2WdBQXE8XnD8DtC+R8SkFCo2E4e2YNdie5+OrxB6DXpteQFF7/f6gnvo4NLbwSRAFSZjFgZq0V68Uuy3iMuHwwaDUhKZNMRfKSEJqSssBqM72mGD0jbgyNJt6U5FGgqxcYG/7BWGKdt7TwShAFSl2JCRsP9+OdfT1xX2t3e8cNl7YadXD7Amlp1CeCw5343NNcYVp1MQDgQBKCcF4/V8TJv/GD07Hlp+fAZtLjd2v3445/7Yr6Wo8vAF+Ak5MniELk22cLo+Ne/jR+c9Q7+3qClRsSwSEZKkfzkpO35JGTn1ErzEt9eXvijWcehRq+LEYdyiyG4F3EQ+8fxpn3vQ1fBGXMMZnh3P7dkpMniBRY1FSGVa2V+DROFUh7vxPt/aPjxstJQzIGVNawsedhJD+pzIyrljXib+8fRmeCEgNK1MlH43CvAx0D4+1wiqP/KJIniAJlYWMp9p4YDkbLkdgq1tN/f/X0kO3SmLreEXW7Xu15GMkDwI2nTUWAA68lWGXj9gcUFU777ZULQp7viSC14HDnvpY8kKaTZ4xdwRjbyRgLMMaWyLY3M8ZGGWNbxX9/St9UgsgtFjaVIcBj13R/+4ktAICvrWoJ2V5pFSL5Xru6kfxYuia3HVE4zRVCKeW+BPPyXgVKKOVcsqABG9ecjf+48CQAwE2PbkYgEFo3H0zXFHgJ5Q4AlwNYH2HfQc75AvHfTWmehyByjpl1Qu442sQol1dwAg2lReMiaSmSv+fV3SpaCNjd+VddAwiql9OqihNefPX4lU/XVFtN+NqqKcHnb+3txl/fPQQAGHR68Nnfvwcgt4d4A2k6ec75bs75XqWMIYh8otZmgtmgxaGeyI6oR0zF3HzG1HH7ysScfHt/8rK6ydDvEGwosyg/l1VtmirM2Hi4H1uOxm86c3n9qk1n+vnlcwEAN/x9E+56aTecHh+O9juD+ydynXwLY2wLY+wdxtiqaC9ijN3IGNvEGNvU0xO/HI0gcgXGGFoqLeM6X11eP6aveQWrfvkWgMjj8rQahlWtlQAwLg2gJN3DbhTptbDmWSQPjP1evvrwx3FfO+j0hkx3UpL5k0pDnh/otodo3ud6Tj7uX54x9iaA2gi71nDO/xnlsE4ATZzzPsbYYgDPM8Zmc87HtbFxzh8A8AAALFmyRL1PO0GowKDTi2ODo2jrdaC50gJAiOA9spK7cCchsaq1Eu/u78Wo16/awmjXiBvVNiMYy93xdNHQaoQY1B/nIujxBWB3+1BuVuduJVzu4OLfvx/yPNedfNxInnO+mnM+J8K/aA4enHM357xPfLwZwEEA06O9niDylR+dNwMAcM1DG4PbRr1jMrnrf3gmGsvNEY+V6qsdKs577R52ocZqUu391eQbZwpprtmiaFk0hkUBOJtKs2fL4lw8JmS6hjFWxRjTio+nAGgFcEiNcxFENrlkQQOMOg2O9juD3asjMk2aporIDh5AUNhqNIkhFcnSPeJGlc2o2vurydSqYnxmVg0GHLHlDaTfn1p3Q/EWdHN9UTvdEsrLGGMdAE4G8BJj7DVx12kAPmWMbQPwNICbOOfxJd0IIg+574r5AIB9XUIttVSb/tn59TGPk27zk5lElCz5HMkDQLXNiB577F4C6U5IzbTJl5Y3odoa+WKZ69U1aVnHOX8OwHMRtj8D4Jl03psg8gUpHdM17MKchhK09QrqhT86d0bM44qCTl6ddI3L64fD4x+XU84nbCY9+h0edAw4Maks8l2RMygvoJ6T/9llc3HF4km47A8fBLc9eeMKNJQWqXZOpaCOV4JIEynC6xZLJt/e240ysz6uA5DSC2pF8vkoThaOtOZ614vR+wmcYi+A2gO15RH7i98+FSumVERdb8klyMkTRJpIC37SBKi2PidWTq2ERhO7okXqlEx21F2iSBePXK/+iMW3z5oGIPbP4MxAugYI7Wyd01Ci6rmUhJw8QaSJtIBqd/vR3u/E4V5HcIxdLKQhIne8GF3ONh2cKi9IZgKLUYeWSgue3XIMr+88gT67Gw+sPxjSWyBVM6nu5PP0YklOniDSRKNhMBu0cLp9+OfWY2AMuHRhQ9zjmmS3+onOi02GTCxIZoI6cZDHX987jBsf2YyfvbwHB8Uu4yc2HsV3n9wKQP0FUOmi/L3VraqeR2nIyROEAliMOjg8PuzuHEFzhSWhBTl5g5JbheEhwVx1HkfyAHD/VQsBAK3VxegYEOQEOodccHn9+I/ndwRfp3akbdJrceDu8/Hds8nJE8SEo9iog93tR+fQKOpLEy9ZvPPSOQCA4SRG3SVKoUTyFcVGLJlchq3tg9CJXbDXPLQR1zy0MaRRKhM/p06rybvuYXLyBKEAFqOQrhn1BpJKG5SIi7ZS16aSSAuSaledZIJFk8vGqX1uPNyPGtvYBVVJPflCgn4rBKEAZoMwmNvl9cOUhL64lG/e1Tl+KEW6BIda5JmWfCSmVRfD4wuMm7AVp4CJADl5glCEYqMOXcMuUfI28a/V4qYy1NiMeH1nYhOQkqGQIvlWcbh3OC6vsJbx/i1nZdKcvIKcPEEowPQaK9r6nOgcciW1AKjRMMyuL0l4OEYySJF8rk8uSoTWGmvE7e/s60FrdXFedJ5mC3LyBKEAp0yrCD5OJl0DCKmIQ72OuJK6yeL0+GA2aOM2ZeUDxUYdNq45O+K+aJO5CAFy8gShAMtaylM+dlqVkG9ul00bUgKHx5/3lTVyqq0mHP75BeO2T4uSyiEEyMkThAIYdVps+8/P4OL59bhuZXNSx04VnZTSKRuH25e3XZrRiFS++Nw3VmbBkvyBnDxBKESJWY/fXbUQ9Unmh6VI9GCUWbGpcmxgFHW2wstVnzWzGgBw3cpmPPX1k2E1qTMspFDI/2V3gshzSor0KDXr0T6gbLqmY2AUp4pzZAuJP3x5EfodnqQvphMViuQJIgdoLDOjvX80/guTYNjlRalKI/GyiUmvJQefBOTkCSIHmFRWFNRlUQKvPwCnx6/a3FMifyAnTxA5gODkRxVTo5TmzErKicTEhZw8QeQAk8rMcPsC+NnLu0O00lOlEKZCEcpATp4gcoDGciHH/Jd3D+ON3V1pv5+kQElOniAnTxA5gHxI9agCM1/HxMnIyU90yMkTRA4g115RQt5AStdYCqwZikgecvIEkQPIpzcp4eSDCpQUyU94yMkTRI5xpN+R1vEeXwDHB10ACkNmmEgPcvIEkSM8c/PJAIAdx4bTep+bH92MO17cBaAwBoYQ6UFOniByhMWTy/H5xZOw8/hQWvXya/d0Bx9TdQ1BTp4gcojW6mL02j2wiwun6WJMYkoVUZjQJ4AgcohqmxEA0DXsSvk9DKJjtxp1EaV5iYkFOXmCyCHqS4RSynWylEuyzKwVRuWNKHQ3QOQ35OQJIoeQJkwNOL0pv0e11QQAKICpf4QCkJMniByCMYYqqxH9dk/K7xEQF21f+s4qpcwi8hhy8gSRY1RYDOhzpO7kvf4AFjWV4qQ6m4JWEfkKOXmCyDH2d9vx5u6ulAd7u30B6LX01SYE6JNAEDmGJGuw/dhQSsd7/YFghQ1B0CeBIHKMp74udL6mWivv9QdgoEieEEnrk8AYu5cxtocx9ilj7DnGWKls362MsQOMsb2MsXPTN5UgJgat1cUAALsrRSfv45SuIYKk+0l4A8Aczvk8APsA3AoAjLFZAK4EMBvAeQD+wBgjEQ2CSIBicWRfOpG8ntI1hEhanwTO+eucc+mTuAHAJPHxJQCe5Jy7OeeHARwAsCydcxHEREGv1cCk12DElVqtvMcfgF5LRfKEgJKX++sBvCI+bgDQLtvXIW4bB2PsRsbYJsbYpp6eHgXNIYj8xWrSU06eUIS4nwTG2JuMsR0R/l0ie80aAD4Aj0mbIrxVRFk9zvkDnPMlnPMlVVVVqfwMBFFwWI06jIg5eZ8/kNRwbw+VUBIy4uqQcs5Xx9rPGLsWwEUAzuZj+qgdABplL5sE4HiqRhLERKPYNObkp615BQsaS/H8N09J6FivnxZeiTHSra45D8CPAVzMOZd3brwA4ErGmJEx1gKgFcDGdM5FEBMJq0kHu9uHfrHzdWv7YMLHeqhOnpCR7kSB3wMwAnhDlDTdwDm/iXO+kzH2FIBdENI43+Scpz+CniAmCMVGHd4/0IVFd76R1HGcczEnTwuvhEBaTp5zPi3GvrsB3J3O+xPERMVq0qd0nD/AwTkoXUMEoU8CQeQgZebUnLzXLyyLUZ08IUGfBILIQSZXWMZt23FsKGZu3usPYNUv1wGgSJ4Ygz4JBJGDfHZe/bhtF93/Hi79n/ejHtNrd6NX1KGnnDwhQU6eIHKQkhTSNf0yDXqK5AkJ+iQQRI7y9+uTUwL5/boDwcdUQklI0CeBIHKU06dXobG8KOHX99kpkifGQ58EgshhHrl++bio3OsPhDxf89x2PLD+IHrs7uA2cvKERLrNUARBqEhzpQU/OGc67nllT3DbqNcfdOJefwCPfXQUgNBAJWGkdA0hQp8EgshxivShoxhcnrHmcYdMqVKuWmnU01ebEKBPAkHkOF9c2oivnz4FP71oFgDA6fHjyY1H4fL6o8oRh18YiIkLpWsIIscx6bW49fyT8PL2TgDAS9s7ce9re7GtYxCLJ5dHPYYgAHLyBJE3SNF559AoAOCJje14YmN7xNeSkyckKF1DEHmC5Lgd7viCribKyRMiFMkTRJ5QZBCc/HNbjkV9zarWSnQOuVBjNWXKLCLHISdPEHlCIoupN58xFSunVmbAGiJfoHs6gsgTEnHyVB9PhEOfCILIE0yGyF/XlsoxWWKjjhZciVAoXUMQeUJpkSH4+I9fXoTz59aBc2FISMutLwMAZtXZsmIbkbuQkyeIPEGuYVNtExZWxdnK2PiTsxHggEZDOvJEKOTkCSIPqS0JrZ6RnD5BhEM5eYLIQ2qsxmybQOQJFMkTRB7x6A3LcXxwFDqSEiYShJw8QeQRp7ZSDTyRHBQOEARBFDDk5AmCIAoYcvIEQRAFDDl5giCIAoacPEEQRAFDTp4gCKKAISdPEARRwJCTJwiCKGCYpGKXCzDGegAcSeMtKgH0KmSOmpCdypIvdgL5YyvZqTxq2jqZc14VaUdOOfl0YYxt4pwvybYd8SA7lSVf7ATyx1ayU3myZSulawiCIAoYcvIEQRAFTKE5+QeybUCCkJ3Kki92AvljK9mpPFmxtaBy8gRBEEQohRbJEwRBEDLIyRMEQRQwBeHkGWPnMcb2MsYOMMZuybItjYyxtxhjuxljOxlj3xW3lzPG3mCM7Rf/L5Mdc6to+17G2LkZtlfLGNvCGHsxx+0sZYw9zRjbI/5uT85FWxlj3xf/7jsYY08wxky5YCdj7CHGWDdjbIdsW9J2McYWM8a2i/t+x6RJ4urbeq/4t/+UMfYcY6w027ZGslO27/8xxjhjrFK2LTu/U855Xv8DoAVwEMAUAAYA2wDMyqI9dQAWiY+tAPYBmAXglwBuEbffAuAX4uNZos1GAC3iz6LNoL0/APA4gBfF57lq598BfE18bABQmmu2AmgAcBhAkfj8KQDX5YKdAE4DsAjADtm2pO0CsBHAyQAYgFcAnJ8hWz8DQCc+/kUu2BrJTnF7I4DXIDR2VmbbzkKI5JcBOMA5P8Q59wB4EsAl2TKGc97JOf9EfDwCYDeEL/8lEBwVxP8vFR9fAuBJzrmbc34YwAEIP5PqMMYmAbgQwF9lm3PRThuEL9SDAMA593DOB3PRVggjNYsYYzoAZgDHc8FOzvl6AP1hm5OyizFWB8DGOf+QC97pf2XHqGor5/x1zrlPfLoBwKRs2xrldwoAvwHwIwDyqpas2VkITr4BQLvseYe4LeswxpoBLATwEYAaznknIFwIAFSLL8um/f8N4cMYkG3LRTunAOgB8DcxtfRXxpgl12zlnB8DcB+AowA6AQxxzl/PNTtlJGtXg/g4fHumuR5CxAvkmK2MsYsBHOOcbwvblTU7C8HJR8pfZb0ulDFWDOAZAN/jnA/HemmEbarbzxi7CEA353xzoodE2Jap37MOwm3xHznnCwE4IKQXopGt32kZhIitBUA9AAtj7CuxDomwLeufXUS3K+v2MsbWAPABeEzaFOFlWbGVMWYGsAbAf0baHcUe1e0sBCffASEHJjEJwi1y1mCM6SE4+Mc458+Km7vEWzOI/3eL27Nl/ykALmaMtUFIcZ3FGHs0B+2Uzt3BOf9IfP40BKefa7auBnCYc97DOfcCeBbAyhy0UyJZuzowliaRb88IjLFrAVwE4MtiagPILVunQrjAbxO/V5MAfMIYq82mnYXg5D8G0MoYa2GMGQBcCeCFbBkjrow/CGA35/zXsl0vALhWfHwtgH/Ktl/JGDMyxloAtEJYiFEVzvmtnPNJnPNmCL+zdZzzr+SanaKtJwC0M8ZmiJvOBrArkAZsDQAAAQ1JREFUB209CmAFY8wsfg7OhrAmk2t2SiRll5jSGWGMrRB/vmtkx6gKY+w8AD8GcDHn3Bn2M+SErZzz7Zzzas55s/i96oBQhHEiq3YquYqbrX8ALoBQxXIQwJos23IqhNutTwFsFf9dAKACwFoA+8X/y2XHrBFt3wsVqhUSsPkMjFXX5KSdABYA2CT+Xp8HUJaLtgK4HcAeADsAPAKhmiLrdgJ4AsI6gReC87khFbsALBF/toMAfg+xaz4Dth6AkNOWvlN/yratkewM298Gsbomm3aSrAFBEEQBUwjpGoIgCCIK5OQJgiAKGHLyBEEQBQw5eYIgiAKGnDxBEEQBQ06eIAiigCEnTxAEUcD8f+dFLNtWU766AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(range(1440), temp[:1440])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Data Preparation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `generator` yields a tuple (samples, targets) where samples is one batch of input data and targets is the corresponding array of target temperatures. It takes the following arguments:\n", "\n", "- data: The original array of floating point data, which we just normalized in the code snippet above.\n", "\n", "- lookback: How many timesteps back should our input data go.\n", "\n", "- delay: How many timesteps in the future should our target be.\n", "\n", "- min_index and max_index: Indices in the data array that delimit which timesteps to draw from. This is useful for keeping a segment of the data for validation and another one for testing.\n", "\n", "- shuffle: Whether to shuffle our samples or draw them in chronological order.\n", "\n", "- batch_size: The number of samples per batch.\n", "\n", "- step: The period, in timesteps, at which we sample data. We will set it 6 in order to draw one data point every hour." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def generator(data, lookback, delay, min_index, max_index,\n", " shuffle=False, batch_size=128, step=6):\n", " if max_index is None:\n", " max_index = len(data) - delay - 1\n", " i = min_index + lookback\n", " while 1:\n", " if shuffle:\n", " rows = np.random.randint(\n", " min_index + lookback, max_index, size=batch_size)\n", " else:\n", " if i + batch_size >= max_index:\n", " i = min_index + lookback\n", " rows = np.arange(i, min(i + batch_size, max_index))\n", " i += len(rows)\n", "\n", " samples = np.zeros((len(rows),\n", " lookback // step,\n", " data.shape[-1]))\n", " targets = np.zeros((len(rows),))\n", " for j, row in enumerate(rows):\n", " indices = range(rows[j] - lookback, rows[j], step)\n", " samples[j] = data[indices]\n", " targets[j] = data[rows[j] + delay][1]\n", " yield samples, targets\n", " \n", "toggle()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "mean = float_data[:200000].mean(axis=0)\n", "float_data -= mean\n", "std = float_data[:200000].std(axis=0)\n", "float_data /= std\n", "\n", "lookback = 1440\n", "step = 6\n", "delay = 144\n", "batch_size = 128\n", " \n", "train_gen = generator(float_data, lookback=lookback, delay=delay, min_index=0,\n", " max_index=200000, shuffle=True, step=step, batch_size=batch_size)\n", "val_gen = generator(float_data, lookback=lookback, delay=delay, min_index=200001,\n", " max_index=300000, step=step, batch_size=batch_size)\n", "test_gen = generator(float_data, lookback=lookback, delay=delay, min_index=300001,\n", " max_index=None, step=step, batch_size=batch_size)\n", "\n", "# This is how many steps to draw from `val_gen`\n", "# in order to see the whole validation set:\n", "val_steps = (300000 - 200001 - lookback) // batch_size\n", "\n", "# This is how many steps to draw from `test_gen`\n", "# in order to see the whole test set:\n", "test_steps = (len(float_data) - 300001 - lookback) // batch_size " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Baseline Models" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A naive model predict that the temperature 24 hours from now will be equal to the temperature right now. We can evaluate this approach using the Mean Absolute Error metric (MAE)." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.2897359729905486\n" ] } ], "source": [ "def evaluate_naive_method():\n", " batch_maes = []\n", " for step in range(val_steps):\n", " samples, targets = next(val_gen)\n", " preds = samples[:, -1, 1]\n", " mae = np.mean(np.abs(preds - targets))\n", " batch_maes.append(mae)\n", " print(np.mean(batch_maes))\n", "evaluate_naive_method()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It yields a MAE of 0.29. Since our temperature data has been normalized to be centered on 0 and have a standard deviation of one, this number is not immediately interpretable. It translates to an average absolute error of 0.29 * temperature_std degrees Celsius (`np.std(float_data[:200000, 1])` = 8.48 before normalization), i.e. 2.57. Now the game is to leverage the deep learning models to do better." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Basic Neural Network**: a simply fully-connected model:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAE/CAYAAAB1i6tsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZyVdfn/8dfFJouKIqssssumIEwOiLmkFi4NmlookqVGaHxLf1mSCxlGmpmWSyr5JSkQtdTEr5qZZWqWMkOyySIi+yqyiIAsc/3++NxHDsMsZ2bOnPucmffz8TiPc869XufmMNf5LPfnY+6OiIiIxKde3AGIiIjUdUrGIiIiMVMyFhERiZmSsYiISMyUjEVERGKmZCwiIhIzJWNJGzOrb2bbzaxTOreNk5l1N7O03/9nZmea2bKk94vM7POpbFuFcz1iZjdWdf9yjvtTM3s03ccVqYsaxB2AxMfMtie9bQp8CuyL3n/b3adV5njuvg84NN3b1gXufmw6jmNmVwGXuftpSce+Kh3HFpGao2Rch7n7Z8kwKnld5e5/K2t7M2vg7nszEZuISF2iamopU1QN+YSZTTezj4HLzGyImf3HzLaY2Vozu9fMGkbbNzAzN7PO0fup0foXzexjM/u3mXWp7LbR+rPNbLGZbTWz+8zsX2b2jTLiTiXGb5vZEjPbbGb3Ju1b38zuMbNNZvY+MKyc63OzmT1eYtkDZnZ39PoqM1sQfZ73o1JrWcdaZWanRa+bmtkfotjmA4NKOe/S6LjzzawgWn4ccD/w+agJ4MOka3tr0v5jos++ycz+bGbtUrk2FTGz86N4tpjZ383s2KR1N5rZGjPbZmYLkz7rYDObFS1fb2a/SPV8IrWJkrFU5ALgMaA58ASwF/ge0BIYSkhW3y5n/0uBW4AWwArgtspua2atgSeBH0Tn/QA4sZzjpBLjOYQkdwLhR8aZ0fKrgS8C/aNzfLWc8zwGnGdmzaI4GwAXR8sB1gPnAocD3wLuM7PjyzlewgSgI9A1ivPyEusXR5+rOTAReMzM2rj7XGAs8Lq7H+ruLUse2My+GB3/IqA9sAYo2RxR1rUpk5n1BqYC/wO0Av4GPGdmDc2sL+H6D3T3w4GzCf++APcBv4iWdwf+VNG5RGojJWOpyBvu/py7F7v7Tnef6e5vufted18KTAJOLWf/P7l7obvvIfzRH1CFbc8D3nH3Z6N19wAflnWQFGO83d23uvsy4NWkc30VuMfdV7n7JuCOcs6zFJgHDI8WnQVscffCaP1z7r7Ug78DrwCldtIq4avAT919s7svJ5R2k8/7pLuvjf5NHgOWAXkpHBdgJPCIu7/j7ruAccCpZtYhaZuyrk15RgAz3P3v0b/RHYQfIfmEH0eNgb5RU8cH0bUD2AP0MLOj3P1jd38rxc8hUqsoGUtFVia/MbNeZva8ma0zs22EUtZBJbAk65Je76D8TltlbXt0chweZjdZVdZBUowxpXMBy8uJF0Ip+JLo9aUklTLN7Dwze8vMPjKzLYQSd3nXKqFdeTGY2TfMbHZUHbwF6JXicSF8vs+O5+7bgM2EUnJCZf7NyjpuMeHfqL27LwK+T/h32GCh2aNttOk3gT7AIjN728zOSfFziNQqSsZSkZK39TxMKA12j6oWxwNWwzGsBT4ruZmZcWDyKKk6Ma4lVBEnVHTr1RPAmVHJcjhRFbWZNSFUud4OtHH3I4C/phjHurJiMLOuwIOE6vSjouMuTDpuRbdhrQGOSTreYcCRwOoU4qrMcesR/s1WA7j7VHcfCnQB6hOuC+6+yN1HAK2BXwJPmVnjasYiknOUjKWyDgO2Ap9E7YTltReny/8BA83sy1G77PcI7ZI1EeOTwLVm1t7MjgJuKG9jd18PvAH8Dljk7u9Fqw4BGgEbgX1mdh5wRiViuNHMjrBwH/bYpHWHEhLuRsLvkqsIJeOE9UCHRIe1UkwHrjSz483sEEJSfN3dy6xpqETMBWZ2WnTuHwAfA2+ZWW8zOz06387osY/wAUaZWcuoJL01+mzF1YxFJOcoGUtlfZ/QoehjQgn0iZo+YZTwvgbcDWwCugH/JdwXne4YHyS07c4FZpJah6LHgDPZ33ELd98CXAc8A3xE6DD1fynG8GNCCX0Z8CLw+6TjzgHuBd6OtukFJLezvgy8B6w3s+Tq5sT+fyFUFz8T7d+J0I5cLe4+n3DNHyT8UBgGFETtx4cAdxLa+dcRSuI3R7ueAyyw0Fv/LuBr7r67uvGI5BoLzW8iucPM6hOqRS9y99fjjkdEpLpUMpacYGbDzKx5VNV5C6GH7tsxhyUikhZKxpIrTgaWEqo6hwHnu3tZ1dQiIjlF1dQiIiIxU8lYREQkZkrGIiIiMcupWZtatmzpnTt3jjsMEZGcUlRU9KG7l3dvvsQsp5Jx586dKSwsjDsMEZGcYmYVDesqMVM1tYiISMyUjEVERGKmZCwiIhIzJWMREZGYKRmLiIjETMlYREQkZkrGIiIiMav1yXjaNOjcGerVC8/TpsUdkYiIyIFyatCPypo2DUaPhh07wvvly8N7gJHVnk5dREQkPWp1yfimm/Yn4oQdO8JyERGRbFGrk/GKFZVbLiIiEodanYw7darcchERkTjU6mQ8cSI0bXrgsqZNw3IREZFsUauT8ciRMGkSHHMMmIXnSZPUeUtERLJLre5NDSHxKvmKiEg2q9UlYxERkVygZCwiIhIzJWMREZGYpZSMzWyYmS0ysyVmNq6U9cPNbI6ZvWNmhWZ2crT82GhZ4rHNzK6N1t1qZquT1p2T3o8mIiKSGyrswGVm9YEHgLOAVcBMM5vh7u8mbfYKMMPd3cyOB54Eern7ImBA0nFWA88k7XePu9+Vno8iIiKSm1IpGZ8ILHH3pe6+G3gcGJ68gbtvd3eP3jYDnIOdAbzv7surE7CIiEhtk0oybg+sTHq/Klp2ADO7wMwWAs8DV5RynBHA9BLLxkbV25PN7MgUYxYREalVUknGVsqyg0q+7v6Mu/cCzgduO+AAZo2AAuCPSYsfBLoRqrHXAr8s9eRmo6N26MKNGzemEK6IiEhuSSUZrwI6Jr3vAKwpa2N3fw3oZmYtkxafDcxy9/VJ2613933uXgz8llAdXtrxJrl7nrvntWrVKoVwRUREcksqyXgm0MPMukQl3BHAjOQNzKy7mVn0eiDQCNiUtMkllKiiNrN2SW8vAOZVPnwREZHcV2Fvanffa2ZjgZeA+sBkd59vZmOi9Q8BFwJfN7M9wE7ga4kOXWbWlNAT+9slDn2nmQ0gVHkvK2W9iIhInWD7O0Fnv7y8PC8sLIw7DBGRnGJmRe6eF3ccUjaNwJVr/vpXmDMn7ihERCSNlIxziTtcein84AdxRyIiImmkZJxLVqyATZvg7behuDjuaEREJE2UjHNJUVF43rIFFi+ONxYREUkbJeNcMmvW/tdvvRVfHCIiklZKxrmkqAj69YPDD4f//CfuaEREJE0qvM9YsoR7SMbnnQdt26pkLCJSi6hknCtWr4aNG2HgQMjPD7c37dgRd1QiIpIGSsa5ItF5a9CgkIz37du/TEREcpqSca6YNQvq1YP+/UMyBlVVi4jUEmozzhVFRdC7NzRtGh5duqgTl4hILaGSca6YNSu0FycMHqySsYhILaFknAvWrg2PQYP2L8vPh1WrQscukdrEHb7/fXj44bgjEckYJeNckBjso2TJGFQ6ltrnuefg7rvhO9/R91vqDCXjXFBUBGYwYMD+ZQMGQKNGdbvd+IMPYMgQeO+9uCORdNm1C667Dnr1gvbtYdQo+OSTuKMSqXFKxrlg1izo2RMOO2z/skMOgRNOqNslh5/+NPwY+cMf4o5E0uWee2DpUrj3Xnj00fBD64c/jDsqkRqnZJwLiooObC9OyM+HwkLYuzfzMcVt2TL4/e/D6xkzYg1F0mT1apg4EYYPh7POgtNPD6Xk3/wG/vKXuKMTqVFKxtluw4bQUSu5vTghPz+MwjVvXubjitsdd4T7rq+9FmbPhuXL445IqmvcuPDD8u679y/72c+gTx+44oowfahILaVknO0SnbdKKxknOnHVtXbjlSth8mS48kq4+uqw7Lnn4o1Jquff/4apU0Mv6q5d9y9v3Dgs//BDuOaa0NNapBZSMs52iWR8wgkHr+vSBVq2rHvtxnfeGf4o33BDaEs/9lhVVeey4mL47nfh6KPhRz86eP0JJ8Ctt8KTT8L06RkPTyQTlIyzXVERdO8OzZsfvM6s7g3+sXYt/Pa38I1vwDHHhGUFBfDqq7B1a5yRSVU9+mjo+3DnnXDooaVv88MfwkknhdudVq7MaHgimaBknO1KjrxVUn4+LFgAW7ZkLqY43XVXaFdMLkEVFMCePfDSS/HFJVWzdWv4tzzpJLj00rK3a9AgdNjbswe++c1QmhapRZSMs9mmTaHXcGntxQmJduOZMzMSUqw2bIAHH4SRIw9sVxwyBI46SlXVuei228LUoPfeG2p6ytOtW+jc9corcP/9mYlPJEOUjLPZf/8bnssrGX/uc+GPWF3oxHX33fDpp3DjjQcur18fzjsPnn8+lJwkNyxcCL/+degpXd4PzmTf+hace27oL7BgQc3GJ5JBKSVjMxtmZovMbImZjStl/XAzm2Nm75hZoZmdnLRumZnNTaxLWt7CzF42s/ei5yPT85FqkcR8xeUl4+bNw2xOtb3deNMmeOAB+NrXQoetkgoKQlX9v/6V+dik8tzDPcRNm4bbl1JlBo88As2ahdG59ONLaokKk7GZ1QceAM4G+gCXmFmfEpu9AvR39wHAFcAjJdaf7u4D3D0vadk44BV37xHtf1CSr/NmzYLOnaFFi/K3y88PJePafNvHr34F27fDTTeVvv6LXwzDg6qqOjc8/3wYyOPWW6F168rt27YtTJoUfqzedluNhCeSaamUjE8Elrj7UnffDTwODE/ewN23u3+WCZoBqWSF4cCU6PUU4PzUQq5Dyhp5q6TBg0PJcenSmo8pDlu2hDbFiy6Cvn1L3+bQQ+GMM0Iyrs0/SmqDTz/dP/702LFVO8ZXvgJf/3ooVdf2WiGpE1JJxu2B5HsJVkXLDmBmF5jZQuB5Quk4wYG/mlmRmY1OWt7G3dcCRM+V/Hlcy23ZAu+/X34VdUJ+fniure3G990H27bBzTeXv11BQbhm776bmbikan79a1iyJNR2NGxY9ePce68mk5BaI5VkXFoXx4OKHu7+jLv3IpRwk+uOhrr7QEI193fM7JTKBGhmo6N26MKNGzdWZtfclui8lUrJuG/f0PZWG0sI27aFyQMKCqB///K3Pe+88Kyq6uy1dm2oWv7yl+FLX6resZo3hylTQmLXZBKS41JJxquAjknvOwBrytrY3V8DuplZy+j9muh5A/AModobYL2ZtQOInjeUcbxJ7p7n7nmtWrVKIdxaorQ5jMvSoEHoVV0bS8a/+Q1s3gy33FLxth06hB8vSsbZ60c/gt27Dxx/ujpOO02TSUitkEoyngn0MLMuZtYIGAEc8NfOzLqbhZsEzWwg0AjYZGbNzOywaHkz4ItAYlaDGcDl0evLgWer+2FqlaIi6NgRUv0Bkp8P77wT5oOtLT75BH75Szj7bMjLq3h7CCXot96CdetqNjapvLfeCiXZ664Lo8qly8SJoXZIk0lIDqswGbv7XmAs8BKwAHjS3eeb2RgzGxNtdiEwz8zeIfS8/lrUoasN8IaZzQbeBp5398TP1zuAs8zsPeCs6L0kzJqV+r2XEDpx7dmzv3q7NnjooTBBQCql4oSCgtCB6/nnay4uqbziYvif/4F27cruEV9VjRuHOa01mYTkMnfPmcegQYO8Tti2zd3MfcKE1PdZvdod3O+5p+biyqQdO9zbtHE/88zK7Vdc7N6pk3tBQc3EJVXzu9+F7+fvf19z5/jZz8I5pk2ruXPkKKDQs+BvuB5lPzQCVzZ6553w674yJeOjjw7V2rWlE9dvfwvr11euVAxhUIiCAnj55TDXs8Rv27YwV/HgwWEo05qSmEzimms0mYTkHCXjbJTKyFulSQz+kes+/TTM4HPqqXBKpTrfBwUFsHNnGMNY4vfTn4YfVvfeC/Vq8E9O/fphMom9ezWZhOQcJeNsNGtWKOm2bVu5/QYPDhNLrF9fI2FlzO9+B6tXV75UnHDqqXDYYepVnQ0WLw73E3/zm6HHf03r1i3cCqfJJCTHKBlno6KiypeKYf/gH7lcVb17N9x+e5iJ6QtfqNoxGjUKPbCfe06lo7j9v/8XOlhVZvzp6rrqqnDP+Q03wBNPhEFgatNdBlIrNYg7ACnhk0/CbDYXX1z5fQcODPccv/VWqKrNRX/4A6xYAQ8/XPGUeuUpKIAnnwxTSyZ+pEhmvfBC6NX+i19UvpanOsxCn4OBA2HEiP3LOnYMt1T16HHgc9eu0KRJ5uITKYWScbaZPTuU5qpSMm7aFI4/PnfbjffuDSWovLzqj8509tmhDXHGDCXjOOzeHe4n7tkTvvvdzJ+/bdtQRf7uu2GErvfe2//8pz8dfD9yIlEnJ+nu3UO1d9OmmY9f6hwl42yT6LxVmZ7UyfLzYepU2LcvJKNc8thjYbKLe+6pXqkYwkxXn/98SMYTJ6YnPkndvfeGZPjCC6HZIA6HHgonnhgeJW3eHJJzyUT9zDPhfuUEM7j22lC6z7X/T5JTlIyzzaxZYUq5o4+u2v6DB8ODD4aJ1/v1S29sNWnfvpA0+/cP4xanQ0FBaLNcujRURUpmrFsHEybAueeGGopsdOSRoUNZaZ3KtmzZn6j/+tfw43DRIpg+HQ4/PPOxSp2gDlzZJjFtYlVLhrnaieuPfwwlqVtuqX6pOCHRbv7cc+k5npRu795QHTx9erif+OyzQ4epe+6JO7KqOeKI0FQyYgRMnhx+3L70EgwdGu5WEKkBSsbZZOfO8EetKu3FCT16hF/9udRuXFwc7kXt2xcuuCB9x+3WDfr00S1O6bR1K7z+epjW8qqrQsnysMPCv92ll4YJINxDAuvRI+5o02PMmDAJxcqVocr7zTfjjkhqIVVTZ5M5c0J1bVXbiyEMqnDiiblVMn7mGZg/P5Ss0j0oREFBaO/bvDn8SJHUuMMHH4QOhbNnh1HhZs8+sGR41FEwYEAY8ap///Do3Tu+NuKadOaZ4QfueefB6aeHEnNNjiYmdY6ScTapzLSJ5Rk8OMwZ+/HHodSSzdxDrMceW7XbuSpSUAB33BFKNpdckv7j1zZTp4bbymbPDt8fCM0GPXuGH3mjR+9PvEcfnb4mhVzQq1f4kXvhhXDZZeEWxJ/8pGZHFZM6Q8k4mxQVhdJGp07VO05+fqj6LSwMv+Kz2XPPhT/8U6bUTG/VE08MHeJmzFAyrsikSfDtb4eOf6NG7U+6/fpBs2ZxR5cdjjoqdOq6+urQtLJwYfju6vYnqSYl42wya1YoFVe3tJG4leOtt7I7GSdKxV27hvbGmlC/fqhafOqpcO9rbaxCTYfJk0MiPuccePppOOSQuCPKXo0awSOPhCr5H/4wVN3PmBGmhxSpItWvZItPP4V586rXXpxw1FGh80y2d+J66aVQer/xxjByWE0pKNjf8UgONmVK6Iz1pS+FHy1KxBUzg+uvhz//OdxGeOKJtWsucck4JeNsMW8e7NlT/fbihPz8UDL2LJ1o3T3ci9qpU6gSrUlnnhnGR1av6oNNnRomcTjjjNCRrnHjuCPKLQUF8MYb4fXJJ4fkLFIFSsbZorojb5U0eHAYfGHFivQcL91efhn+/e9wX2pNVx03axYS8owZ2fvjJA7Tp8Pll4emjGef1fjMVTVgALz9dmhb/8pXwvSf+p5JJSkZZ4tZs8JgA126pOd42Tz4hzuMHx9KxVdckZlzFhSEtr25c9N/7HXrwvCP776b/mPXlCefDD2CE0OGqgNS9bRrB6++Gu4IuOEGuPLK0EdBJEVKxtkiMW1ium4VOf74UOWYje3GL7wQfiTcckvm2ifPOy88p7uqeuvW0Nb6ve+FgS9694abbgo/rrK1dPTUU6HD3Eknwf/9n3pKp0uTJqG2Yfz4MCf3WWcdOM61SDnMs/UPRiny8vK8sLAw7jDSb8+eMKj9d78bBqhIl6FDw/O//pW+Y1aXexhqcMuWcFtIw4aZO3d+fjj/22+n53i7d4fex//8Z5j6cdOm0BP51VfDrWWdO4dqywsvDM0G2XA/6rPPwkUXhZGzXnop++9Dz1WPPRZqfdq3Dz94evfev+7TT8MgNJs3w0cfpf768cfhtNOqFI6ZFbl7Xno+nNQE3dqUDebPD3/Y09VenDB4MDzwQHbd0vPnP4dS46OPZjYRQ6iqvvlmWLOm6hNxJBQXhz+2r7wSeiMn5s39zndCaejZZ0Nivu++METk0UeHoT6/8hU45ZSa7T1elueeC9WogwaFQVCUiGvOpZeGJqfzzw8/Ao85Zn9y3bGj/H2bNw+jxbVoEZ779QuvjzoqM7FLLFQyrsi+feGP2K9+BS1bhgkN0j3q0OTJoY1p0aIw0lG6PPkkfO1rMHNmKI3Grbg4dHb59NPwAyTTCWnu3FB9//DDYSSp6hg3Dn7+8zDT1I03lr3d1q2hZPT00/Dii2H88ZYtYfjwkJjPOCMzVfUvvBB+DPTvHzrPNW9e8+cUWL48tCHv3h0Sa3KSLe39EUfUyOA3KhnnAHfPmcegQYM8Y7Zvd7//fvfu3d3BvXnz8Pz88+k/1zXXuB92mPu+fek97vLlIeb770/vcavqiSdCPI89Fs/5i4vdu3RxP/fc6h3n/vvD5xgzJhwzVdu3uz/1lPull4Z/b3A//HD3kSPD8k8+qV5cZfnLX9wPOcR94ED3jz6qmXNIVgMKPQv+hutR9iP2ACrzyEgyXrPG/cYb3Vu0CJfnc59zf/xx9x073Lt2de/fP/1Jc/Bg91NOSe8x3UOiaNvW/bLL0n/sytq71713b/c+fcLruHzveyExbd9etf2fftrdzL2goHqfY9eu8MPuiivcjzoqfNeaNnW/6CL36dPdt22r+rGTvfyye+PG7gMGuG/alJ5jSs5RMs7+R+wBVOZRo8l47lz3b3zDvVGj8Mf2/PPdX3/9wJLP1Knhkk2fnr7z7tnj3qSJ+3XXpe+Yyc4/371Hj5o5dmX84Q/h2v3xj/HG8corIY5nnqn8vv/6V0hs+fnpLcXu2RPiuvrq8OMJwg+G4cPdf/97982bq3bcv/89fLeOO85948b0xSs5R8k4+x+pbQTDgEXAEmBcKeuHA3OAd4BC4ORoeUfgH8ACYD7wvaR9bgVWR/u8A5xTURxpT8bFxe4vveT+xS+GS9GkSagyXry49O337Qt/2Lp3d9+9Oz0xzJ0bzv2HP6TneCXdfns4/ocf1szxU7FnT7hmNVGrUFm7d4cmh29+s3L7LVwYaku6d3ffsKFmYnMPpe3XXgsl+Pbtw79dw4bu55zjPnly6v+Or74aStp9+9ZsvJITlIyz/5FKIq4PvA90BRoBs4E+JbY5lP2dwY4HFkav2wEDo9eHAYsT+0bJ+PrKBJu2ZLxrl/vvfhcSK7i3aeP+05+m9ofu2WfDPpMmpSeWRx8Nx3v33fQcr6S//z0c/4UXaub4qZg8OcTw7LPxxZDskkvcW7VKvZp57Vr3zp3dW7d2X7KkZmNLtm+f+7//7f7974fzg3v9+u5nneX+8MPu69eXvt/rr7s3axaaBdaty1y8krWUjLP/kcqNjycCS9x9qbvvBh6PSsLJncC2R//gAM0Aj5avdfdZ0euPoxJy+xTOWTM++gh+9rNw/+c3vxnuOZ08OfR4vOmm1G4d+PKXYciQMI/pzp3Vj2nWrDDoQjp7USfLywv3t8Y1+Mfu3WEM6ry8cO2yQUEBbNyY2uhkH38M554LGzaEXtHdutV8fAn16oXb0+66C5YuDZNq/OAH8MEHYYaldu3CUJYPPBBu14IwxOjZZ0OHDvD3v0ObNpmLV0SqLJVk3B5YmfR+FaUkVDO7wMwWAs8DB41xaGadgROA5L+AY81sjplNNrMjKxF35SxZAmPHQseOIekef3y4z3LOnJCUK3NriVlI6KtXw29+U/3YiorC7T41MZcvhHtJ+/aNb1jM3/0uDEM5YUL2TEQ/bFi4raqi0bj27An35c6eHW5p+9znMhNfaczC/cG33w6LF4eYbrop/EgYOzYk36FDw2hg7dqFRNy2bXzxikjlVFR0Bi4GHkl6Pwq4r5ztTwH+VmLZoUAR8JWkZW0IVeD1gInA5DKON5rQDl3YqVMnr5Irrwztbt/4hvucOVU7Rklf/GLoBbt1a9WPsXdvqE78n/9JT0xl+da33I84IvPttbt2uXfo4D5kSOVuAcqEM84I1bhlKS4O3xdwf+SRzMVVFfPnu0+Y4H788aGNeOXKuCOSLIOqqbP+kUoyHgK8lPT+R8CPKtjnA6Bl9Loh8BLw/8rZvjMwr6JYqtxmvHKl++rVVdu3LIWF4fKNH1/1Y7z7bjjGo4+mL67SPPJIOM/ChTV7npLuuy+c9+WXM3veVPz61yG2sjrrjR8f1v/4xxkNS6QmKBln/yOVauqZQA8z62JmjYARwAH1e2bW3SzUQZrZQEJHr03Rsv8FFrj73SX2aZf09gJgXgqxVE2HDtUf/rCkQYPCmMN33x3aH6siMW1iuuYwLsvgweE5k+3GO3eG6vxTTgmjTGWbRPv1c88dvO63vw3V6ldcAT/+cWbjEpE6qcJk7O57gbGE0u0C4El3n29mY8xsTLTZhcA8M3sHeAD4WvRrbCihWvsLZvZO9Dgn2udOM5trZnOA04Hr0vvRMuC228I4s7ffXrX9Z80KMyslDyJfE3r1Cm3HmWw3fughWLs2u9qKk3XpAscdd3C78fPPw9VXh3blhx7KzthFpNbR2NTVdcUVYYaW994LHcQq49RTwzjNmSixnnlmGKQ+URqvSZ98Al27hmT3t7/V/Pmq6uab4Y47YP360JN+5swwK07v3mHmpUMPjTtCkbTQ2NTZLwvmdMtxt94abpH6yU8qt19xMfz3v+mfqaks+fmhB25FM8akw/33h16+t91W8+eqjoKCMBHIiy+GHvfnnhtuBXr+eSViEckoJePq6tQpVGs++miYdSlVS5aEe/IIBM0AAB8KSURBVFhrur04YfDgkHhmzarZ82zbBnfeGe51HTKkZs9VXXl54fafKVNCvMXF4ZY33ZsrIhmmZJwON94Y2n7Hj099n0RSzGTJGGq+Svzee8PgKhMm1Ox50qFevdCR629/g1WrQmeumhp8RUSkHErG6dC6NVx3XZg/ONWSZ1ERNGoEffrUbGwJrVuHkcdqshPXli3wy1+G6t9smD85FaNGweGHw/Tp2V+SF5FaS8k4Xa6/PkwSftNNqW0/a1YYCaxRo5qNK9ngwTVbMr777pCQc6FUnPD5z4eS/Pnnxx2JiNRhSsbp0rw5jBsX2hxfe638bd1DMs5Ue3FCfn6ojl29Ov3H3rQJfvUruOgi6N8//cevSTU1FKmISIqUjNNp7NgwuMiNN4aEW5YPPgglyEy1FyckBv+oiarqu+6C7dtD73IREakUJeN0atIEbrkF/vUveOGFsrdL3Oub6WQ8YAA0bJj+ZLxhQ+i4NWJEmJRCREQqRck43a68Mkyzd9NN4VaZ0syaFZJiv36Zja1xYzjhhPS3G//857Brl4aOFBGpIiXjdGvYMHRgmj079K4uTVFRSMSVmboxXfLzw7y4e/em53hr1oSpJC+7DI49Nj3HFBGpY5SMa8KIEWEoyFtuCXPiJour81bC4MFhFK55aZqX4447wmeszD3WIiJyACXjmlCvHkycGEbZ+t3vDly3YkXoeZzp9uKExOAfjz4KCxaEUbmqauVKePhh+OY3Q9W8iIhUiZJxTTnvvDCIxE9+EqYTTEgMChJXybhr11Cd/OtfhwFHDj88xHnNNWHqwMLC0P6biokTQ0n/5ptrNmYRkVquQdwB1FpmYWrF006DBx4Ig4JAaC+uXz8M+BFXXHPnhlLxf/+7/zFtGjz4YNimfv2QqE84IfTATjwfccT+43zwAfzv/8K3vgXHHBPPZxERqSU0hWJNGzYsTM33wQehFHrOOWHgjTlz4o7sQMXFIcZ33jkwSa9du3+bLl1CYj7hBHj7bfjrX+H996F9+/jiFpEKaQrF7KeScU2bODGM0/zLX4YBMYqKQkLONvXqhXbfbt3gwgv3L1+//sDk/N//wtNPh3XXXqtELCKSBkrGNW3QoDBE5N13wwUXhAEy4movroo2bULpftiw/cu2bQvTRcZV1S4iUsuoA1cm3HZbuJ3ossvC+7h6UqfL4YfD5z4Xz33SIiK1kJJxJvTqBd/4BsyfH6qDc20iBRERqVFKxpny4x+H6RJ79YJmzeKORkREsojajDOlU6dwH2+TJnFHIiIiWUbJOJO+/vW4IxARkSykamoREZGYKRmLiIjETMlYREQkZiklYzMbZmaLzGyJmY0rZf1wM5tjZu+YWaGZnVzRvmbWwsxeNrP3oucj0/ORREREckuFydjM6gMPAGcDfYBLzKxPic1eAfq7+wDgCuCRFPYdB7zi7j2i/Q9K8iIiInVBKiXjE4El7r7U3XcDjwPDkzdw9+2+f8aJZoCnsO9wYEr0egpwftU/hoiISO5KJRm3B1YmvV8VLTuAmV1gZguB5wml44r2bePuawGi59alndzMRkdV34UbN25MIVwREZHckkoytlKWHTTvors/4+69CCXc2yqzb3ncfZK757l7XqtWrSqzq4iISE5IJRmvAjomve8ArClrY3d/DehmZi0r2He9mbUDiJ43VCJuERGRWiOVZDwT6GFmXcysETACmJG8gZl1NzOLXg8EGgGbKth3BnB59Ppy4NnqfhgREZFcVOFwmO6+18zGAi8B9YHJ7j7fzMZE6x8CLgS+bmZ7gJ3A16IOXaXuGx36DuBJM7sSWAFcnObPJiIikhNsfyfo7JeXl+eFhYVxhyEiklPMrMjd8+KOQ8qmEbhERERipmQsIiISMyVjERGRmCkZi4iIxEzJWEREJGZKxiIiIjFTMhYREYmZkrGIiEjMlIxFRERipmQsIiISMyVjERGRmCkZi4iIxEzJWEREJGZKxiIiIjFTMhYREYmZknEKpk2Dzp2hXr3wPG1a3BGJiEht0iDuALLdtGkwejTs2BHeL18e3gOMHBlfXCIiUnuoZFyBm27an4gTduwIy0VERNJBybgCK1ZUbrmIiEhlKRlXoFOnyi0XERGpLCXjCkycCE2bHrisadOwXEREJB2UjCswciRMmgTHHANm4XnSJHXeEhGR9FFv6hSMHKnkKyIiNUclYxERkZillIzNbJiZLTKzJWY2rpT1I81sTvR408z6R8uPNbN3kh7bzOzaaN2tZrY6ad056f1oIiIiuaHCamozqw88AJwFrAJmmtkMd383abMPgFPdfbOZnQ1MAvLdfREwIOk4q4Fnkva7x93vSs9HERERyU2plIxPBJa4+1J33w08DgxP3sDd33T3zdHb/wAdSjnOGcD77r68OgGLiIjUNqkk4/bAyqT3q6JlZbkSeLGU5SOA6SWWjY2qtieb2ZEpxCIiIlLrpJKMrZRlXuqGZqcTkvENJZY3AgqAPyYtfhDoRqjGXgv8soxjjjazQjMr3LhxYwrhioiI5JZUkvEqoGPS+w7AmpIbmdnxwCPAcHffVGL12cAsd1+fWODu6919n7sXA78lVIcfxN0nuXueu+e1atUqhXBFRERySyrJeCbQw8y6RCXcEcCM5A3MrBPwNDDK3ReXcoxLKFFFbWbtkt5eAMyrTOAiIiK1RYW9qd19r5mNBV4C6gOT3X2+mY2J1j8EjAeOAn5jZgB73T0PwMyaEnpif7vEoe80swGEKu9lpawXERGpE8y91ObfrJSXl+eFhYVxhyEiklPMrChRQJLspBG4REREYqZkLCIiEjMlYxERkZgpGYuIiMRMyVhERCRmSsYiIiIxUzLOkGnToHNnqFcvPE+bFndEIiKSLSoc9EOqb9o0GD0aduwI75cvD+8BRo6MLy4REckOKhlnwE037U/ECTt2hOUiIiJKxhmwYkXllouISN2iZJwBnTpVbrmIiNQtSsYZMHEiNG164LKmTcNyERERJeMMGDkSJk2CY44Bs/A8aZI6b4mISKDe1BkycqSSr4iIlE4lYxERkZgpGYuIiMRMyVhERCRmSsYiIiIxUzIWERGJmZKxiIhIzJSMRUREYqZkLCIiEjMlYxERkZgpGYuIiMRMyTiHTJsGnTtDvXrhedq0uCMSEZF0SCkZm9kwM1tkZkvMbFwp60ea2Zzo8aaZ9U9at8zM5prZO2ZWmLS8hZm9bGbvRc9Hpucj1U7TpsHo0bB8ObiH59GjlZBFRGqDCpOxmdUHHgDOBvoAl5hZnxKbfQCc6u7HA7cBk0qsP93dB7h7XtKyccAr7t4DeCV6L2W46SbYsePAZTt2hOUiIpLbUikZnwgscfel7r4beBwYnryBu7/p7pujt/8BOqRw3OHAlOj1FOD81EKum1asqNxyERHJHakk4/bAyqT3q6JlZbkSeDHpvQN/NbMiMxudtLyNu68FiJ5bpxZy3dSpU+WWi4hI7kglGVspy7zUDc1OJyTjG5IWD3X3gYRq7u+Y2SmVCdDMRptZoZkVbty4sTK71ioTJ0LTpgcua9o0LBcRkdyWSjJeBXRMet8BWFNyIzM7HngEGO7umxLL3X1N9LwBeIZQ7Q2w3szaRfu2AzaUdnJ3n+Tuee6e16pVqxTCrZ1GjoRJk+CYY8AsPE+aFJaLiEhuSyUZzwR6mFkXM2sEjABmJG9gZp2Ap4FR7r44aXkzMzss8Rr4IjAvWj0DuDx6fTnwbHU+SF0wciQsWwbFxeFZiVhEpHZoUNEG7r7XzMYCLwH1gcnuPt/MxkTrHwLGA0cBvzEzgL1Rz+k2wDPRsgbAY+7+l+jQdwBPmtmVwArg4rR+MhERkRxh7qU2/2alvLw8LywsrHhDERH5jJkVlbi1VLKMRuCqYzSKl4hI9qmwmlpqj8QoXonBQxKjeIHan0VE4qSScR2iUbxERLKTknEdolG8RESyk5JxHaJRvEREspOScR2iUbxERLKTknEdolG8RESyk3pT1zEjRyr5iohkG5WMRUREYqZkLCIiEjMlY6k0jeIlIpJeajOWStEoXiIi6aeSsVSKRvESEUk/JWOpFI3iJSKSfkrGUikaxUtEJP2UjKVSNIqXiEj6KRlLpWgULxGR9FMylkobORKWLYPi4vBclUSs26NERPbTrU2Scbo9SkTkQCoZS8bp9igRkQMpGUvG6fYoEZEDKRlLxun2KBGRAykZS8bp9igRkQMpGUvGpev2KPXIFpHaIqVkbGbDzGyRmS0xs3GlrB9pZnOix5tm1j9a3tHM/mFmC8xsvpl9L2mfW81stZm9Ez3OSd/HkmxX3dujEj2yly8H9/09spWQRSQXmbuXv4FZfWAxcBawCpgJXOLu7yZtcxKwwN03m9nZwK3unm9m7YB27j7LzA4DioDz3f1dM7sV2O7ud6UabF5enhcWFlbyI0pt1LlzSMAlHXNMSO4isp+ZFbl7XirbFhUVtW7QoMEjQD9Ue5ouxcC8vXv3XjVo0KANpW2Qyn3GJwJL3H0pgJk9DgwHPkvG7v5m0vb/ATpEy9cCa6PXH5vZAqB98r4iVZGOHtnTpoXbqVasCJ3HJk7Ufc4iDRo0eKRt27a9W7VqtblevXrll9YkJcXFxbZx48Y+69atewQoKG2bVH71tAdWJr1fFS0ry5XAiyUXmlln4ATgraTFY6Oq7clmdmQKsYgA1e+RrWpukTL1a9Wq1TYl4vSpV6+et2rVaiuhtqH0bVI4jpWyrNR/JDM7nZCMbyix/FDgKeBad98WLX4Q6AYMIJSef1nGMUebWaGZFW7cuDGFcKUuqG6PbA08IlKmekrE6Rdd0zJzbirJeBXQMel9B2BNyY3M7HjgEWC4u29KWt6QkIinufvTieXuvt7d97l7MfBbQnX4Qdx9krvnuXteq1atUghX6oLq9sjWwCMikk1SScYzgR5m1sXMGgEjgBnJG5hZJ+BpYJS7L05absD/Ejp33V1in3ZJby8A5lXtI0hdVZ0e2Rp4RCQ7rVu3rn6vXr369OrVq0/Lli37t27d+vjE+127dpVWU3uQiy66qPPs2bMPKW+b22+/vdWDDz7YIj1RV1+Fydjd9wJjgZeABcCT7j7fzMaY2Zhos/HAUcBvotuUEl2ehwKjgC+UcgvTnWY218zmAKcD16Xxc4mUK10Dj+heZ6nrHnqIFkcfzXH16jHo6KM57qGHqFaCa9u27b6FCxe+u3Dhwne//vWvbxwzZsz6xPvGjRs7QHFxMfv27SvzGH/605+W9e/f/9PyzvOjH/1o49VXX/1RdWJNp5S6rbv7C+7e0927ufvEaNlD7v5Q9Poqdz/S3QdEj7xo+Rvubu5+fNK6F6J1o9z9uGhdQdTzWiQj0jHwiDqBSV330EO0uO46jlm7lkbusHYtja67jmOqm5BLM2/evEN69OjR99JLL+3Ut2/fPitWrGh4ySWXHNOvX7/e3bt373v99dd/Vts6aNCgY998880me/bs4bDDDhtwzTXXtD/22GP7DBgwoNfq1asbAHz3u989esKECa0T219zzTXtjzvuuN6dO3fu9/LLLzcD2LZtW70vfelL3Y499tg+X/7yl7v069ev95tvvtkk3Z8NdA+Z1GHVHXhEncCkrpswgfa7dh2YR3btot6ECeXecVNl77//fuNvf/vbHy5YsODdLl267PnVr361at68eQsWLFgw/x//+MfhRUVFjUvus3379vqnnXbax4sWLXo3Ly9v+wMPPNCytGO7O3Pnzl0wceLElRMmTDga4I477mjdunXrPYsWLXr3xhtvXLdgwYKmpe2bDkrGIlWkTmBS161bR6PKLK+ujh07fnrqqad+9hN48uTJLfr06dO7b9++fZYuXdp4zpw5B5VaGzduXPzVr351G8CgQYN2LFu2rNTYLr744i0AJ5100o5Vq1Y1Avj3v/996MiRIz8CGDJkyM5u3brtrInPBUrGIlWmTmBS17Vty+7KLK+uJk2aFCdez50795CHH364zWuvvbZ48eLF755yyinbdu7ceVAHrwYNGnx2m1b9+vV93759pXYCa9y4cXHJbSoaoTKdlIxFqkidwKSuGz+e1Y0bU5y8rHFjisePZ3VNn3vLli31mzVrtu/II4/ct3z58oavvfba4ek+x5AhQ7ZPnz79SIC33367ydKlS2ukvRhSGw5TREqRaGOuzpCaiU5gibbnRCew5OOLZKsxY/gIQtvxunU0atuW3ePHszqxvCYNHTp0R48ePXb17Nmzb6dOnT4dNGjQ9nSfY9y4cRsuvvjiLj179uxz3HHH7ejevfvOFi1alN2NuxoqnCgim2iiCKltNOGFZEJlJoqYPXv2sv79+39Y0zHlgj179rBnzx5r2rSpz50795Bhw4b1XLZs2dyGDRtW6XizZ89u2b9//86lrVPJWCRG6gQmkr22bt1a/9RTT+25d+9ec3fuu+++5VVNxBVRMhaJUadOpZeM1QlMJH4tW7bcN3/+/AWZOJc6cInESJ3ARASUjEVipZHARASUjEVip5HARETJWCTHqROYSO5TMhbJcekaCUztzpItTjzxxGOfeuqpAwbxmDBhQuvLLruszG9106ZNTwBYtmxZw2HDhnUt67ivvfZaueNLT5gwofXHH3/8WW489dRTu3/44Yf1K/cJKk/JWCTHpaMTmNqdJZtcfPHFm6ZPn37AzE9PPfVUi8suu6zCwUQ6d+685y9/+cvSqp774YcfbrN9+/bPcuM///nPJS1btqyRgT6SKRmL5Lh0dAJLV7uzSteSDqNGjdr8yiuvNE+MNb1o0aJGGzZsaJifn79jyJAhPfv06dO7Z8+efaZOnXpEyX0XLVrUqEePHn0Btm/fbuedd17Xnj179jn33HO77tq167NxqUeOHNkpMf3idddddzTAT3/609YbNmxoeOqpp/bMz8/vCdC+ffvj1q5d2wDg1ltvbdOjR4++PXr06JuYfnHRokWNunbt2nfEiBHHdO/eve/QoUN7bN++vdTxr8uj+4xFaoGRI6s3fGY62p01tGctdMUVHZk3L73TBvbrt4PJk1eWt0nbtm339e/f/5Onnnqq+WWXXbZlypQpLQoKCjYfeuihxc8///ySFi1aFK9du7ZBfn5+r0svvXRLvXqllyvvuuuu1k2aNClevHjxu2+99VaToUOH9kmsu/vuu1e3adNm3969eznppJOOfeutt5rcfPPNGx588ME2//znPxe3a9dub/KxXn/99aaPPfbYUUVFRQvcnUGDBvU+44wzPm7ZsuW+FStWNJ46derSk046afk555zT9fe///2R11xzTaWGBFXJWETS0u6sXt2STl/96lc/euKJJ44EePrpp1uMGjXqo+LiYrv22ms79OzZs8/pp5/ec8OGDY1WrVpVZqHyjTfeOHTUqFGbAPLz83f27Nnzs2/olClTWvTp06d3nz59+rz33nuNZ8+efdBcyMleffXVQ88555wthx9+eHHz5s2Lzz333M3/+Mc/DgNo3779pyeddNJOgBNOOGHHsmXLDqns51XJWESYOPHAUi1Uvt05Xb26p02r3uQbkkYVlGBr0siRI7fcfPPNHd94442mu3btqnfyySfvuPfee4/atGlTg7lz5y445JBDvH379sft3Lmz3EKl2cE1xgsXLmx0//33tykqKlrQqlWrfRdeeGHnXbt2lXuc8uZxaNSo0QHTNFYUU2lUMhaRtLQ7p6N0na6OZGq7zn3NmzcvHjx48MdXXXVV56985SsfQRgrumXLlnsOOeQQf+655w5bs2ZNo/KOcfLJJ2+fOnVqC4CZM2c2Xrx4cVOAzZs312/SpElxixYt9q1cubLBq6++2jyxT7NmzfZt3br1oNz4hS98YfsLL7xwxMcff1xv27Zt9V544YUjTz/99I/T9XmVjEUEqP7gI+no1Z2Oqm4l9NpjxIgRHy1atKjJqFGjPgK46qqrPpo9e3azfv369Z46dWqLLl267Cpv/+uvv37DJ598Ur9nz559fvazn7U97rjjPgEYMmTIzn79+u3o0aNH31GjRnVOnn7x8ssv//Dss8/ukejAlXDyySfvuPTSSzcNHDiw96BBg3qPGjVq49ChQ3em67NqCkURSZvqVjHXqxcSaElm4UdCKtIxLWXJzmgQflhUtrYgW2gKxexQ3hSKKhmLSNpUt3SdjqrudLRdqzOaZJqSsYhkjXRUdWdLQgdVdUvqlIxFJGukoyNZtiT0HB7VrLi4uLjSg1ZI+aJrWmZji5KxiGSV6lZ1Z0tCz+Gq7nkbN25sroScPsXFxbZx48bmwLyytknpPmMzGwb8GqgPPOLud5RYPxK4IXq7Hbja3WeXt6+ZtQCeADoDy4CvuvvmVD+ciEhZqjsiWWLf6nRGy9XZtPbu3XvVunXrHlm3bl0/VGBLl2Jg3t69e68qa4MKe1ObWX1gMXAWsAqYCVzi7u8mbXMSsMDdN5vZ2cCt7p5f3r5mdifwkbvfYWbjgCPd/QbKod7UIpIr0tGrG9IzCEplelNLPFL51XMisMTdl7r7buBxYHjyBu7+ZlKp9j9AhxT2HQ5MiV5PAc6v+scQEckumk1LKiOVZNweSB4SbVW0rCxXAi+msG8bd18LED23TiVgEZFckE2zaUn2S6XNuLRG/FLrts3sdEIyPrmy+5Z5crPRwGiATpWdLV1EJEbZMJuW5IZUSsargI5J7zsAa0puZGbHA48Aw919Uwr7rjezdtG+7YANpZ3c3Se5e56757Vq1SqFcEVEaod03GIluSGVZDwT6GFmXcysETACmJG8gZl1Ap4GRrn74hT3nQFcHr2+HHi26h9DRKT2SUe7s+SGCqup3X2vmY0FXiLcnjTZ3eeb2Zho/UPAeOAo4DfRdFV7o9JsqftGh74DeNLMrgRWABen+bOJiOS0dNxiJblBE0WIiNRyurUp++mGbhERkZgpGYuIiMRMyVhERCRmSsYiIiIxUzIWERGJmZKxiIhIzJSMRUREYqZkLCIiErOcGvTDzDYCpcwQmpKWwIdpDKem5EqckDuxKs70y5VYFWdwjLtrcP8sllPJuDrMrDAXRqDJlTghd2JVnOmXK7EqTskVqqYWERGJmZKxiIhIzOpSMp4UdwApypU4IXdiVZzplyuxKk7JCXWmzVhERCRb1aWSsYiISFaqdcnYzIaZ2SIzW2Jm40pZb2Z2b7R+jpkNjCHGjmb2DzNbYGbzzex7pWxzmpltNbN3osf4TMeZFMsyM5sbxXHQhNJZck2PTbpW75jZNjO7tsQ2sVxTM5tsZhvMbF7SshZm9rKZvRc9H1nGvuV+nzMQ5y/MbGH07/qMmR1Rxr7lfkcyFOutZrY66d/3nDL2jfuaPpEU4zIze6eMfTN6TSVm7l5rHkB94H2gK9AImA30KbHNOcCLgAGDgbdiiLMdMDB6fRiwuJQ4TwP+L+5rGsWyDGhZzvrYr2kp34N1hHsrY7+mwCnAQGBe0rI7gXHR63HAz8v4HOV+nzMQ5xeBBtHrn5cWZyrfkQzFeitwfQrfjVivaYn1vwTGZ8M11SPeR20rGZ8ILHH3pe6+G3gcGF5im+HA7z34D3CEmbXLZJDuvtbdZ0WvPwYWAO0zGUOaxX5NSzgDeN/dqzpATFq5+2vARyUWDwemRK+nAOeXsmsq3+cajdPd/+rue6O3/wE61NT5K6OMa5qK2K9pgpkZ8FVgek2dX3JHbUvG7YGVSe9XcXCSS2WbjDGzzsAJwFulrB5iZrPN7EUz65vRwA7kwF/NrMjMRpeyPquuKTCCsv/AZcs1bePuayH8OANal7JNtl3XKwg1IKWp6DuSKWOjKvXJZVT9Z9M1/Tyw3t3fK2N9tlxTyYDaloytlGUlu4unsk1GmNmhwFPAte6+rcTqWYRq1v7AfcCfMx1fkqHuPhA4G/iOmZ1SYn02XdNGQAHwx1JWZ9M1TUU2XdebgL3AtDI2qeg7kgkPAt2AAcBaQhVwSVlzTYFLKL9UnA3XVDKktiXjVUDHpPcdgDVV2KbGmVlDQiKe5u5Pl1zv7tvcfXv0+gWgoZm1zHCYiVjWRM8bgGcIVX3JsuKaRs4GZrn7+pIrsumaAusTVfnR84ZStsmK62pmlwPnASPdvdTElcJ3pMa5+3p33+fuxcBvy4ghW65pA+ArwBNlbZMN11Qyp7Yl45lADzPrEpWQRgAzSmwzA/h61AN4MLA1UV2YKVFb0f8CC9z97jK2aRtth5mdSPi32pS5KD+Lo5mZHZZ4TejQM6/EZrFf0yRlljay5ZpGZgCXR68vB54tZZtUvs81ysyGATcABe6+o4xtUvmO1LgS/RQuKCOG2K9p5ExgobuvKm1ltlxTyaC4e5Cl+0Ho2buY0GPypmjZGGBM9NqAB6L1c4G8GGI8mVA1Ngd4J3qcUyLOscB8Qm/P/wAnxXQ9u0YxzI7iycprGsXRlJBcmycti/2aEn4crAX2EEpmVwJHAa8A70XPLaJtjwZeKO/7nOE4lxDaWBPf04dKxlnWdySGWP8Qff/mEBJsu2y8ptHyRxPfy6RtY72mesT70AhcIiIiMatt1dQiIiI5R8lYREQkZkrGIiIiMVMyFhERiZmSsYiISMyUjEVERGKmZCwiIhIzJWMREZGY/X/ZW+UQhIXjoQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def basic_nn():\n", " model = models.Sequential()\n", " model.add(layers.Flatten(input_shape=(lookback // step, float_data.shape[-1])))\n", " model.add(layers.Dense(64, activation='relu'))\n", " model.add(layers.Dense(1))\n", "\n", " model.compile(optimizer=optimizers.RMSprop(), loss='mae')\n", " history = model.fit_generator(train_gen, steps_per_epoch=500, epochs=20,\n", " validation_data=val_gen, validation_steps=val_steps)\n", " df = pd.DataFrame.from_dict(data=history.history, orient='columns')\n", " df.to_csv(base_dir+'\\\\basic_nn.csv', header=True, index=False)\n", " K.clear_session()\n", " del model\n", " \n", "df = pd.read_csv(base_dir+'\\\\basic_nn.csv')\n", "history = df.to_dict()\n", "\n", "loss = list(history['loss'].values())[1:]\n", "val_loss = list(history['val_loss'].values())[1:]\n", "epochs = range(len(loss))\n", "\n", "plt.figure(figsize=(6, 5))\n", "plt.plot(epochs, loss, 'bo', label='Training')\n", "plt.plot(epochs, val_loss, 'r', label='Validation')\n", "plt.title('Training and validation loss')\n", "plt.legend(bbox_to_anchor=(1.02, 0.2), loc=2, borderaxespad=0.5)\n", "plt.show()\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The validation losses are close to 0.32 and worse than the no-learning baseline. It turns out not to be so easy to outperform the naive model. The naive model already contains valuable information that a machine learning model does not have access to.\n", "\n", "If there exists a simple, well-performing model (naive model), why doesn't the model we are training find it and improve on it? \n", "\n", "- The hypothesis space, the space of models in which we are searching for a solution, is the space of all possible 2-layer networks with the configuration that we defined. \n", "\n", "
\n", "\n", "- When looking for a solution with a space of complicated models, the simple well-performing baseline might be unlearnable, even if it's technically part of the hypothesis space. \n", "\n", "
\n", "\n", "- That is a pretty significant limitation of machine learning in general: unless the learning algorithm is hard-coded to look for a specific kind of simple model, parameter learning can sometimes fail to find a simple solution to a simple problem." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Basic Recurrent Neural Network**: a simply GRU model:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdwAAAE/CAYAAADsc3LZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXyU5fX38c9hBxEUAVG2oBKVRVRSkUURl4pKReuGRNQqVWv9uVStVqxPHyrVp1qr9ueGaKuCW0WtVdxq3alIUBCURcSwb7IjUAg5zx/XRIaQZUIm98xkvu/Xa17J3OuZO4GT67qv+1zm7oiIiEjNqpPqAERERLKBEq6IiEgElHBFREQioIQrIiISASVcERGRCCjhioiIREAJV6rMzOqa2UYz65DMbVPJzA4ys6Q/I2dmJ5pZYdz72WZ2TCLb7sa5xpjZLbu7fwXHvd3M/pbs44pkm3qpDkBqnpltjHvbBPgvsD32/nJ3H1eV47n7dqBpsrfNBu5+cDKOY2bDgQvc/bi4Yw9PxrFFpGYo4WYBd/8h4cVaUMPd/V/lbW9m9dy9KIrYRESyhbqUpaTL8Dkze8bMNgAXmFlvM/vEzNaa2VIzu9/M6se2r2dmbmY5sfdjY+tfN7MNZvYfM+tU1W1j608xszlmts7M/mJmH5vZxeXEnUiMl5vZXDNbY2b3x+1b18z+bGarzOwbYGAF1+dWM3u21LIHzOye2PfDzWxm7PN8E2t9lnesRWZ2XOz7Jmb2VCy2L4GeZZx3Xuy4X5rZ6bHl3YH/BY6Jddd/F3dtfxe3/xWxz77KzF42s/0SuTaVMbMzYvGsNbN/m9nBcetuMbMlZrbezGbFfdajzeyz2PLlZnZXoucTqS2UcKXEmcDTQHPgOaAIuAZoCfQlJKTLK9h/KPBboAWwAPh9Vbc1s9bA88CNsfN+CxxVwXESifFUQiI7gvCHxImx5b8Afgz0iJ3j3ArO8zQwyMz2iMVZDzgnthxgOXAa0Az4OfAXMzusguOVGAm0Bw6IxXlRqfVzYp+rOTAKeNrM9nX36cBVwIfu3tTdW5Y+sJn9OHb8s4G2wBKg9K2D8q5NuczsUGAs8D9AK+BfwD/NrL6ZdSVc/yPdvRlwCuHnC/AX4K7Y8oOAFyo7l0hto4QrJT5y93+6e7G7b3b3ye4+yd2L3H0eMBroX8H+L7h7gbtvI/zHfvhubDsImOru/4it+zPwXXkHSTDGO9x9nbsXAu/Fnetc4M/uvsjdVwF3VnCeecAMYHBs0UnAWncviK3/p7vP8+DfwDtAmQOjSjkXuN3d17j7fEKrNf68z7v70tjP5GmgEMhL4LgA+cAYd5/q7luAm4H+ZtYubpvyrk1FhgCvuPu/Yz+jOwl/aPQi/AHUCOgauy3xbezaAWwDOpvZPu6+wd0nJfg5RGoNJVwpsTD+jZkdYmavmdkyM1tPaC3t0pKKsyzu+01UPFCqvG33j4/Dw8wai8o7SIIxJnQuYH4F8UJozZ4f+34oca1FMxtkZpPMbLWZrSW0nCu6ViX2qygGM7vYzKbFum7XAockeFwIn++H47n7emANobVboio/s/KOW0z4GbV199nA9YSfwwoLtyjaxDb9GdAFmG1mn5rZqQl+DpFaQwlXSpR+JOYRQqvuoFg34G2A1XAMS4EfWmBmZuycIEqrToxLCd25JSp7bOk54MRYC3Ewse5kM2tM6B69A9jX3fcC3kowjmXlxWBmBwAPEbq+94kdd1bccSt7hGkJ0DHueHsCewOLE4irKsetQ/iZLQZw97Hu3hfoBNQlXBfcfba7DwFaA38CxptZo2rGIpJRlHClPHsC64DvY/ftKrp/myyvAkea2U9i90mvIdwnrIkYnweuNbO2ZrYPcFNFG7v7cuAj4K/AbHf/OraqIdAAWAlsN7NBwAlViOEWM9vLwnPKV8Wta0pIqisJf3sMJ7RwSywH2pUMEivDM8ClZnaYmTUkJL4P3b3cHoMqxHy6mR0XO/eNwAZgkpkdamYDYufbHHttJ3yAYWbWMtYiXhf7bMXVjEUkoyjhSnmuJwzi2UBoST5X0yeMJbXzgHuAVcCBwOeE54aTHeNDhHut04HJJDaI52ngRHYMlsLd1wLXAS8BqwmDlF5NMIb/Q2hpFwKvA0/GHfcL4H7g09g2hwDx9z3fBr4GlptZfNdwyf5vELp2X4rt34FwX7da3P1LwjV/iPDHwEDg9Nj93IbAHwn33ZcRWtS3xnY9FZhpYRT83cB57r61uvGIZBLTBPSSrsysLqEL82x3/zDV8YiIVIdauJJWzGygmTWPdUv+ljDy9dMUhyUiUm1KuJJu+gHzCN2SA4Ez3L28LmURkYyhLmUREZEIqIUrIiISASVcERGRCKTlbEEtW7b0nJycVIchIpIxpkyZ8p27V/TcuqRYWibcnJwcCgoKUh2GiEjGMLPKypNKiqlLWUREJAJKuCIiIhFQwhUREYmAEq6IiEgElHBFREQioIQrIiISASVcERGRCNSahDtuHOTkQJ064eu4camOSEREZIeEEm5syrTZZjbXzG4uY/1gM/vCzKaaWYGZ9Ut032QYNw4uuwzmzwf38PWyy5R0RUQkfVQ6W1BsEvA5wEnAImAycL67fxW3TVPge3d3MzsMeN7dD0lk37Lk5eV5VSpN5eSEJFtax45QWJjwYUREMpaZTXH3vFTHIeVLpIV7FDDX3ee5+1bgWWBw/AbuvtF3ZO49AE9032RYsKBqy0VERKKWSMJtCyyMe78otmwnZnammc0CXgMuqcq+1dWhQ9WWi4iIRC2RhGtlLNulH9rdX3L3Q4AzgN9XZV8AM7ssdv+3YOXKlQmEtcOoUdCkyc7LmjQJy0VERNJBIgl3EdA+7n07YEl5G7v7B8CBZtayKvu6+2h3z3P3vFatqjbDVH4+jB4d7tmaha+jR4flIiIi6SCR6fkmA53NrBOwGBgCDI3fwMwOAr6JDZo6EmgArALWVrZvsuTnK8GKiEj6qjThunuRmV0FvAnUBR539y/N7IrY+oeBs4ALzWwbsBk4LzaIqsx9a+iziIikzpo1MG8eHH441K2b6mgkDVX6WFAqVPWxIBGRlJozB048ERYuhH33hTPOgLPOguOOg/r1IwlBjwWlv1pTaUpEJCWmToV+/WDLFnjwQTj2WBg7Fn7845B8L74YXnklrJespoQrIrK7Pv44tGIbNYKPPoJf/AKefx5WroSXX4af/AT+8Q8YPBhatYIhQ8L6jRtTHbmkgBKuiMjueOutHa3Yjz6C3Nwd6xo3Dkn2iSdg+XJ4800YOhTefRfOOw9atgzrn3wy3PuVrKCEKyJSVS+8AIMGQefO8MEHFVfZadAgJOZHHoElS+D99+GKK+Dzz+Gii6B1azj55LB++fLoPoNEToOmRESq4vHH4ec/h9694dVXYa+9du847lBQAOPHh9fcudC8eeiO3o2BVho0lf4SeQ5XREQA7r0XrrsutFhffBH22GP3j2UGP/pReN1xB8yYAbNmRTaqWaKnhCsiUhl3+L//N7zOOivM/dmwYfKObwbdu4eX1FpKuCIiFSkuDq3a+++Hn/0s1I2tp/86peo0aEpEpDxFRXDJJSHZXnstjBmjZCu7Tb85IiJl+e9/4fzz4aWXQlfyb38bun5FdpMSrohIad9/D2eeCW+/DffdB1dfneqIpBZQl7KIpIcZM+Bf/0p1FKEQxUknwTvvwN/+pmQrSaOEKyKpt2EDDBwYEt2ll4YWZiosWxZKNU6ZAn//eyhMIZIkSrgiknq33RaqMF18Mfz1r3DkkfDZZ9HG8O23cMwxoQDFq6/CT38a7fml1lPCFZHU+vzzMAr4sstCsn3nndDCPfpo+NOfwmM5NWnLFvjDH6BbN/juu3Df9qSTavackpWUcEUkdbZvh8svD8X877gjLBswAKZNg9NOgxtugFNOCV29yeYeps3r2hVGjAjn+fxz6NMn+ecSQQlXRFLp4Ydh8mT4859h7713LN9nn1A68aGHwuQAhx0GEyYk77yzZoUEO3hwmFrv7bfDhAQ5Ock7h0gpSrgikhpLlsAtt8CJJ4bnXUszC7PqTJkCbdqEFu+111ZvIvf16+HGG0MJxU8+CbWRp04NMYjUMCVcEUmNX/0qFJd48MGKC0p06QKffhoez7nvvnBvd+bMqp2ruDjMTZubG+4LX3wxzJkD11yjyQIkMkq4IhK9N9+E554LLdzOnSvfvlGjkGxffRUWL4aePUNN40SmF508OdyXvfhi6NQJJk2CRx8N89CKREgJV0SitXkzXHllaG3edFPV9j3tNPjiC+jXLwy2OvtsWL267G2XLw/P9B51FBQWhhbuxx+H6fBEUkAJV0SidfvtMG9eGDC1O1Pc7bcfvPEG3H03/POfYUDVe+/tWL9tW7g3m5sLTz0V7tnOmQMXXgh19F+epI5++0QkOl99BXfdBcOGhcd/dledOnD99fCf/0CTJnD88XDrraGrukePMJ1e794wfTr88Y/QrFnyPoPIbtLkBSISDXf4xS+gadPQOk2Gnj1DRaprroFRo8KyAw4Iz9cOGqTZfSStKOGKSDT+9rfwTG2yByw1bQqPPRYS7Pz54VGiRo2Sd3yRJFHCFZGa99134V5q375hQveacOaZNXNckSTRPVwRqXm//jWsWxcGSmngkmQp/eaLSM364IMwKcH114cJAkSylBKuiNScrVvDPdWcnDAFn0gW0z1cEak5d98dyjC++mp4fEcki6mFKyI145tv4Pe/h7POChWiRLKcEq6IJJ87/PKXYWKA++5LdTQiaUFdyiKSfM8/H6o+3XcftG2b6mhE0oJauCKSXOvWhXlrjzwytHJFBFDC/cG4cWEgZZ064eu4camOSCRDjRgBK1bAI49A3bqpjkYkbahLmZBcL7sMNm0K7+fPD+8B8vNTF5dIxvn00zCh/FVXQV5eqqMRSSvmiUzgHLG8vDwvKCiI7Hw5OSHJltaxY5hGU0QSUFQU5p5dtgxmzdIMPREzsynurr9y0phauMCCBVVbLiJxtmwJ0+SNHQuffx4GTCnZiuxCCRfo0KHsFm6HDtHHIpL2tm2DggL497/D6+OP4b//DQMgLrkEzj471RGKpCUlXMI0mvH3cCEUxSmZXlMkq23fDtOm7UiwH34IGzeGdT16wJVXhgngjzkGmjdPbawiaUwJlx0Do0aMCN3IHTqEZKsBU5KV3OGrr0JyffddeO89WLMmrDvkELjwQhgwAI47Dlq2TGWkIhlFCTcmP18JVjLcm2+G7t3d5Q5z54Yku3x5WNapU5hn9vjjQ5Ldf//kxCqShZRwRTLd99/DddfBo4+G92a7f6w2beDEE3ck2E6dkhOjiCjhimS0yZND18zcuXDTTTByJDRokOqoRKQMCVWaMrOBZjbbzOaa2c1lrM83sy9ir4lm1iNu3XVm9qWZzTCzZ8ysUTI/gEhW2r49DDTo0wc2bw73W++8U8lWJI1VmnDNrC7wAHAK0AU438y6lNrsW6C/ux8G/B4YHdu3LXA1kOfu3YC6wJDkhS+ShQoLw4ClW28Nj+B88UV4LyJpLZEW7lHAXHef5+5bgWeBwfEbuPtEd48NY+QToF3c6npAYzOrBzQBllQ/bJEs5B6KS/ToEZLsU0/B00/D3nunOjIRSUAiCbctsDDu/aLYsvJcCrwO4O6LgbuBBcBSYJ27v7V7oYpksTVr4PzzYdgwOOyw8FzsBRdUb4CUiEQqkYRb1r/oMgswm9kAQsK9KfZ+b0JruBOwP7CHmV1Qzr6XmVmBmRWsXLkykdhFssN774VW7fjxcPvt4X1OToqDEpGqSiThLgLax71vRxndwmZ2GDAGGOzuq2KLTwS+dfeV7r4NeBHoU9ZJ3H20u+e5e16rVq2q8hlEaqetW8PI4+OPh0aNYOLEUJ1FU96JZKREEu5koLOZdTKzBoRBT6/Eb2BmHQjJdJi7z4lbtQA42syamJkBJwAzkxO6SC02cyYcfTT88Y/w85+HSQF+9KNURyUi1VBpwnX3IuAq4E1Csnze3b80syvM7IrYZrcB+wAPmtlUMyuI7TsJeAH4DJgeO9/o5H+M1NME9pIU7mE+2SOPDHVGX345TOS+xx6pjkxEqknz4SZB6QnsIUx+MHq0ykVKFSxfDpdeCq+9BiefDH/9K+y3X6qjkgyh+XDTX0KFL6RiI0bsnGwhvB8xIjXxSIbZvDkUrcjNhX/9C+67DyZMULIVqWWUcJNAE9jLbtm+HZ54IiTa3/wG+veHqVPh6qvDvQkRqVX0rzoJypuoXhPYS7neegt69oSLLw4t2ffeg1deCdPfiUitpISbBKNGhXu28TSBvZRp2rRwf/bkk2H9enj2Wfjkk9C6FZFaTQk3CfLzwwCpjh1D4Z+OHTVgSkpZuBAuugiOOAIKCuDPfw6P/px3nrqPRbKEpudLEk1gL2Vatw7uuCMMhHKHG28M92v32ivVkYlIxJRwRWrC1q3w0EPw+9/D6tWh7vHtt+vGvkgWq119WWn4TLFkGXd4/nk49FC49trQhTxlCjz5pJKtSJarPQl348YwEOWpp1IdiWSj7dvhzTdDOcbzzguVod54I4xGPuKIVEcnImmg9nQp168f/tO75BJo3TokX5GaNmdOeJb2ySdh0SJo2zZUiBo2TJMMiMhOak8Lt2FDeOkl6NoVzjoLJk9OdURSW61dG4ah9+kDBx8cqkR17w7PPQdz54Zna5VsRaSU2pNwAZo1g9dfh1at4LTT4OuvUx2R1Bbbt4cu4vPPhzZt4PLLwwjkP/4xtGwnTIBzzw3T6ImIlKH2dCmX2G+/cC+tb9/QrTxxYvgPUmqvbdvg009DHeItW+DAA8PrgAOgXbvqtTZnzgxdxk89BUuWQIsWMHx4aMX27BkevBYRSUDtS7gQatO+9hoMGACnnhrK5jVrluqoJFncQ9ftW2/B22/Dv/8NGzaE5Fe3LhQV7di2QYMwX2JJEo5/deoEjRvvevzVq0MFqCeeCIm8bl045RS4/34YNCjcvhARqaLamXABjjoKxo+Hn/wEfvrTkIDT+D/KcePC7EILFoSnR0aNUiGNnaxeDe+8syPJzp8flufkhG7ek06C448Pf1gtWgTffLPr66OPQmKOt//+O7eIp0+Hf/wjPEfbvTv86U/hB7HvvpF/ZBGpXWr/fLhPPhlK6p13Hjz9dFqW0dN8umXYujXcDnj77fAqKAgt22bNQmL98Y9Dkj3wwMS7dd1h1aqyk/E338DSpdCyJQwdGrqMDz9cXcaSMTQfbvqr/QkXwsCWm26Ca64JNWzT7D/RnJwdDbZ4HTtCYWHU0aRQYWFoXb71Frz/Pnz/fejOPfrokFxPOin0XNSroY6ZzZvD42U1dXyRGqSEm/6y43+WG28MA17uuy90If7616mOaCdZP5/u0qUwciSMGRPuv3buHFqYJ50Exx0HzZtHE0dZ93NFRJIkOxKuGdxzDyxbFlq6bdrAhRemOqofdOhQdgu31lcCXLs29D7ce28YaXzZZXD99eFeqohILZN+NzRrSp06YdTpCSfApZeGZyrTRNbNp7t5M9x9d0isd9wBZ5wBs2bBAw8o2YpIrZU9CRfCKOUXXwyjT886KzzykQaSMZ/uuHHhXnCdOuHruHE1FW01FBXBY4+Fx7ZuvBF69YLPPw+D2Q48MNXRiYjUqOwYNFXasmWhLN+GDfDxxyEBZLC0H+XsDi+/HJ57mjkzJNo77wz3Z0UkKTRoKv1lVwu3RJs2oRqVWahGtXRp9Y9ZXAzz5oXHWSI2YsTOyRbC+xEjIg9lV++9B717h2eh3UMPw3/+o2QrIlknOxMuhJGwr70GK1eGKkLr1iW+b3FxmCXmmWfCIJ/+/cNI2gMPhB/9CBYvrrm4y5CWo5ynTg3XdcCAUIhizJhQVOLMM9PusSwRkShkxyjl8vzoR6Ea1aBBoQU2YcKu1ajcQ8u1oCBMJF7ydf36sL5Ro1Ag4eKLw7DikSNDi+7118PMRRFIq1HO33wDv/1t+GNk773hrrvgl7/UIzcikvWyO+FC6FJ+/PHwmNCFF4ZRs599tnOCXbs2bNugAfToEW6M5uWF16GHhmIJJU48MdRv7tcv3Lfs37/GP8KoUWXfw41slPOWLaGe8d//DmPHhuvxm9+E55332iuiIERE0lt2Dpoqy1137VwQo359OOywMCNMSXLt2jUk3crMnw8DB4aW8VNPhWnbaljktZjXrg09Ai+/HFrzGzfCnnvCBReEFu5++9XgyUWkNA2aSn9KuCXc4fnnYc2akFy7d6/eZAerV8PgwaFg/p/+BL/6VfJiTZXFi+GVV+Cll+Ddd8NjPvvuGz7nmWeG+7VpPEGESG2mhJv+1KVcwixMcJAsLVqEovvDhoWBVQsWhMRbnblZo+YeClK8/HJ4lTy33Llz+APijDPCIz5pOCGEiEi6UcKtSY0awXPPhYR7771htO7YsWF5uiouhkmTdiTZOXPC8qOOgj/8ISTZQw7RSGMRkSpSwq1pdeqEGYo6dAitwuXLw4w4LVok7xybNoWbuA89FCZpqFcvtKTr1dvxqux9vXohiU6eHAqD1KsXuoivuQZOPx3atUtevCIiWUgJNyrXXQdt24Yu5r59w0CjnJzqHbOwEB58MDzjumZNeDzpjDNg+/Zwf7XkVdH7TZt2XtavX7gfe+qpGmEsIpJESrhROvfcUOVq8ODwrO6ECXDEEVU7hnuYK/b++0NL2Sw8Q/w//xOSpbp6RUTSkka7RO3YY8PI5fr1w/dvvZXYfps2hZZsjx6hq/eDD8JUg99+G0ZXH3OMkq2ISBpTwk2Frl3hk09CKcjTToO//a38bRcsgJtvhvbt4ec/D0n1scdg4cIwiKl9+8jCFhGR3acu5VTZf//QSj3rLPjZz8II5hEjQkJ1hw8/DN3GL70Utj/zTLj6arVkRUQylBJuKjVrFiZQGD48VGdasACOPjok2mnTwkjmG2+EK69MUWFkERFJFnUpp1qDBvDEE3DLLfDoo3DppeFZ2EcfDd3Gd94ZSbLNiAnsRUQymFq46cAsFD/u3RuaNg0THkTYbVx6Avv588N7SJMJ7EVEagHVUhZycsqe3q9jx/Cor4ikP9VSTn/qUpb0nMBeRKSWUcKVcm8Ra5yWiEjyKOEKo0aFCevjRTqBvYhIFlDCFfLzYfTocM/WLHwdPVoDpkREkimhhGtmA81stpnNNbOby1ifb2ZfxF4TzaxH3Lq9zOwFM5tlZjPNrHcyP4AkR35+GCBVXBy+KtmKiCRXpY8FmVld4AHgJGARMNnMXnH3r+I2+xbo7+5rzOwUYDTQK7buPuANdz/bzBoApTovRUREar9EWrhHAXPdfZ67bwWeBQbHb+DuE919TeztJ0A7ADNrBhwLPBbbbqu7r01W8CIiIpkikYTbFlgY935RbFl5LgVej31/ALAS+KuZfW5mY8xsj92KVEREJIMlknDLKnlUZrUMMxtASLg3xRbVA44EHnL3I4DvgV3uAcf2vczMCsysYOXKlQmEJSIikjkSSbiLgPg54NoBS0pvZGaHAWOAwe6+Km7fRe4+Kfb+BUIC3oW7j3b3PHfPa9WqVaLxi4iIZIREEu5koLOZdYoNehoCvBK/gZl1AF4Ehrn7nJLl7r4MWGhmB8cWnQDED7YSERHJCpUmXHcvAq4C3gRmAs+7+5dmdoWZXRHb7DZgH+BBM5tqZvGFkP8HGGdmXwCHA39I6ieQtKDZhkREKqbJC6TaSs82BKFSlYpniERHkxekP1WakmobMWLnZAvh/YgRqYlHRCQdKeFKtWm2IRGRyinhSrVptiERkcop4Uq1abYhEZHKKeFKtWm2IRGRylU6eYFIIvLzlWBFRCqiFq6IiEgElHBFREQioIQrIiISASVcSQsqDSkitZ0GTUnKlS4NOX9+eA8aiCUitYdauJJyKg0pItlACVdSTqUhRSQbKOFKyqk0pIhkAyVcSTmVhhSRbKCEKymn0pAikg00SlnSgkpDikhtpxauiIhIBJRwRUREIqCEKyIiEgElXBERkQgo4YqIiERACVdqBU1+ICLpTo8FScbT5AcikgnUwpWMp8kPRCQTKOFKxtPkByKSCZRwJeNp8gMRyQRKuJLxNPmBiGQCJVzJeJr8QEQygUYpS62gyQ9EJN2phSsiIhIBJVwRVDhDRGqeupQl66lwhohEQS1cyXoqnCEiUVDClaynwhkiEgUlXMl6KpwhIlFQwpWsp8IZIhIFJVzJeiqcISJR0ChlEVQ4Q0Rqnlq4IiIiEVDCFRERiYASroiISASUcEVERCKghCuSBKrFLCKVSSjhmtlAM5ttZnPN7OYy1ueb2Rex10Qz61FqfV0z+9zMXk1W4CLpoqQW8/z54L6jFrOSrojEqzThmlld4AHgFKALcL6ZdSm12bdAf3c/DPg9MLrU+muAmdUPVyT9qBaziCQikRbuUcBcd5/n7luBZ4HB8Ru4+0R3XxN7+wnQrmSdmbUDTgPGJCdkkfSiWswikohEEm5bYGHc+0WxZeW5FHg97v29wK+B4opOYmaXmVmBmRWsXLkygbBE0oNqMYtIIhJJuFbGMi9zQ7MBhIR7U+z9IGCFu0+p7CTuPtrd89w9r1WrVgmEJZIeVItZRBKRSMJdBLSPe98OWFJ6IzM7jNBtPNjdV8UW9wVON7NCQlf08WY2tloRi6QZ1WIWkUSYe5mN1R0bmNUD5gAnAIuBycBQd/8ybpsOwL+BC919YjnHOQ64wd0HVRZUXl6eFxQUJPoZRESynplNcfe8VMch5at08gJ3LzKzq4A3gbrA4+7+pZldEVv/MHAbsA/woJkBFOkHLyIiskOlLdxUUAtXRKRq1MJNf6o0JZIGVKlKpPbTfLgiKVZSqaqkeEZJpSrQwCuR2kQtXJEUU6UqkeyghCuSYqpUJZIdlHBFUkyVqkSygxKuSIqpUpVIdlDCFUkxVaoSyQ5KuCJpID8fCguhuDh8rWqy1WNFIulPjwWJZDg9ViSSGdTCFclweqxIJDMo4YpkOD1WJJIZlHBFMpweKxLJDEq4IhlOjxWJZAYlXJEMp8eKRDKDRimL1AL5+UqwIulOLVwREZEIKOGKiApniERAXQHJdqMAABE/SURBVMoiWU6FM0SioRauSJZT4QyRaCjhimQ5Fc4QiYYSrkiWU+EMkWgo4YpkORXOEImGEq5IllPhDJFoaJSyiKhwhkgE1MIVERGJgBKuiFSbCmeIVE5dyiJSLSqcIZIYtXBFpFpUOEMkMUq4IlItKpwhkhglXBGpFhXOEEmMEq6IVIsKZ4gkRglXRKpFhTNEEqNRyiJSbSqcIVI5tXBFREQioIQrIiISASVcERGRCCjhioiIREAJV0REJAJKuCIiIhFQwhUREYmAEq6IpJym95NsoMIXIpJSmt5PsoVauCKSUpreT7KFEq6IpJSm95NskVCXspkNBO4D6gJj3P3OUuvzgZtibzcCv3D3aWbWHngSaAMUA6Pd/b5kBS8ima9Dh9CNXNZyqTlTpkxpXa9evTFAN9T4SoZiYEZRUdHwnj17rihrg0oTrpnVBR4ATgIWAZPN7BV3/ypus2+B/u6+xsxOAUYDvYAi4Hp3/8zM9gSmmNnbpfYVkSw2atTO93BB0/tFoV69emPatGlzaKtWrdbUqVPHUx1PpisuLraVK1d2WbZs2Rjg9LK2SeSvmqOAue4+z923As8Cg+M3cPeJ7r4m9vYToF1s+VJ3/yz2/QZgJtB2tz6NiNRKyZjeT6Ocd0u3Vq1arVeyTY46dep4q1at1hF6DMqUSJdyW2Bh3PtFhNZreS4FXi+90MxygCOASWXtZGaXAZcBdFBfkkhWqc70fhrlvNvqKNkmV+x6ltuQTaSFa2UsK/OHZGYDCAn3plLLmwLjgWvdfX1Z+7r7aHfPc/e8Vq1aJRCWiIhGOUvmSCThLgLax71vBywpvZGZHQaMAQa7+6q45fUJyXacu79YvXBFRHamUc6ZadmyZXUPOeSQLoccckiXli1b9mjduvVhJe+3bNlSVkNvF2effXbOtGnTGla0zR133NHqoYceapGcqKsnkS7lyUBnM+sELAaGAEPjNzCzDsCLwDB3nxO33IDHgJnufk/SohYRidEo52g8/DAtRo6k7bJlNGjThq233cbiK65g9e4er02bNttnzZr1FcCvfvWr/Zs2bbp95MiRy+O3KS4uxt2pW7dumcd44YUXCis7z29+85uVuxtjslXawnX3IuAq4E3CoKfn3f1LM7vCzK6IbXYbsA/woJlNNbOC2PK+wDDg+NjyqWZ2avI/hohkq1GjwqjmeBrlnFwPP0yL666j49KlNHCHpUtpcN11dHz4YZLecpwxY0bDzp07dx06dGiHrl27dlmwYEH9888/v2O3bt0OPeigg7recMMN+5Vs27Nnz4MnTpzYeNu2bey5556HX3nllW0PPvjgLocffvghixcvrgdw9dVX7z9y5MjWJdtfeeWVbbt3735oTk5Ot7fffnsPgPXr19c5+eSTDzz44IO7/OQnP+nUrVu3QydOnNg42Z8toWev3H2Cu+e6+4HuPiq27GF3fzj2/XB339vdD4+98mLLP3J3c/fD4tZNSPaHEJHslYxRzlKxkSNpu2XLzvliyxbqjBxZM0+dfPPNN40uv/zy72bOnPlVp06dtt17772LZsyYMXPmzJlfvvvuu82mTJnSqPQ+GzdurHvcccdtmD179ld5eXkbH3jggZZlHdvdmT59+sxRo0YtHDly5P4Ad955Z+vWrVtvmz179le33HLLspkzZzYpa9/q0sPOIpLx8vOhsBCKi8NXJdvkWraMBlVZXl3t27f/b//+/X8YCvf444+36NKly6Fdu3btMm/evEZffPHFLq3PRo0aFZ977rnrAXr27LmpsLCwzNjOOeectQB9+vTZtGjRogYA//nPf5rm5+evBujdu/fmAw88cHNNfC4lXBERqVCbNmytyvLqaty4cXHJ99OnT2/4yCOP7PvBBx/MmTNnzlfHHnvs+s2bN+8yqKpevXo/PD1Tt25d3759e5kDrxo1alRcehv3aJ6OUsIVEZEK3XYbixs1ojh+WaNGFN92G4tr+txr166tu8cee2zfe++9t8+fP7/+Bx980CzZ5+jdu/fGZ555Zm+ATz/9tPG8efOSfv8WND2fiIhUomQ0cjJHKSeqb9++mzp37rwlNze3a4cOHf7bs2fPjck+x80337zinHPO6ZSbm9ule/fumw466KDNLVq02J7s81hUTemqyMvL84KCgso3FBERAMxsSsmA1URMmzatsEePHt/VZEyZYtu2bWzbts2aNGni06dPbzhw4MDcwsLC6fXr16/ysaZNm9ayR48eOWWtUwtXRESy2rp16+r2798/t6ioyNydv/zlL/N3J9lWRglXRLLeuHGhFOSCBaFgxqhRGumcTVq2bLn9yy+/nFnT51HCFZGspskPJCoapSwiWU2TH0hUlHBFJKtp8gOJihKuiGS18iY50OQHkmxKuCKS1TT5QWocddRRB48fP36nIhYjR45sfcEFF5T7p06TJk2OACgsLKw/cODAA8o77gcffFBhLeSRI0e23rBhww/5r3///gd99913ZU9JlERKuCKS1TT5QWqcc845q5555pmdZhsaP358iwsuuKDSYho5OTnb3njjjXm7e+5HHnlk340bN/6Q/95///25LVu2THqhi9KUcEUk62nyg+gNGzZszTvvvNO8pC7y7NmzG6xYsaJ+r169NvXu3Tu3S5cuh+bm5nYZO3bsXqX3nT17doPOnTt3Bdi4caMNGjTogNzc3C6nnXbaAfGT1+fn53comdbvuuuu2x/g9ttvb71ixYr6/fv3z+3Vq1cuQNu2bbsvXbq0HsDvfve7fTt37ty1c+fOXUum9Zs9e3aDAw44oOuQIUM6HnTQQV379u3beePGjWXWaq6IHgsSEcl2l1zSnhkzkjslXbdum3j88YXlrW7Tps32Hj16fD9+/PjmF1xwwdonnniixemnn76madOmxa+99trcFi1aFC9durRer169Dhk6dOjaOnXKbh/efffdrRs3blw8Z86cryZNmtS4b9++XUrW3XPPPYv33Xff7UVFRfTp0+fgSZMmNb711ltXPPTQQ/u+//77c/bbb7+i+GN9+OGHTZ5++ul9pkyZMtPd6dmz56EnnHDChpYtW25fsGBBo7Fjx87r06fP/FNPPfWAJ598cu8rr7yySqUt1cIVEammceMgJwfq1Alfx41LdUSZ4dxzz1393HPP7Q3w4osvthg2bNjq4uJiu/baa9vl5uZ2GTBgQO6KFSsaLFq0qNzG4UcffdR02LBhqwB69eq1OTc394eHvJ544okWXbp0ObRLly5dvv7660bTpk3bZR7deO+9917TU089dW2zZs2KmzdvXnzaaaeteffdd/cEaNu27X/79OmzGeCII47YVFhY2LCqn1ctXBGRaqgVhTMqaInWpPz8/LW33npr+48++qjJli1b6vTr12/T/fffv8+qVavqTZ8+fWbDhg29bdu23Tdv3lxh49Bs197dWbNmNfjf//3ffadMmTKzVatW288666ycLVu2VHiciuYWaNCgwU7T/1UWU1nUwhURqQYVzth9zZs3Lz766KM3DB8+POenP/3pagh1jVu2bLmtYcOG/s9//nPPJUuWVDjJfb9+/TaOHTu2BcDkyZMbzZkzpwnAmjVr6jZu3Li4RYsW2xcuXFjvvffea16yzx577LF93bp1u+S/448/fuOECRP22rBhQ53169fXmTBhwt4DBgzYkKzPqxauiEg1qHBG9QwZMmT1RRdddOAzzzwzD2D48OGrTznllIO6det2aNeuXTd16tRpS0X733DDDSuGDBnSKTc3t0vXrl03de/e/XuA3r17b+7Wrdumzp077zKt30UXXfTdKaec0rl169bbJk2aNKdkeb9+/TYNHTp01ZFHHnkowLBhw1b27dt38+zZsytM+onS9HwiItWQkxO6kUvr2DGMeI6KpudLDxVNz6cuZRGRalDhDEmUEq6ISDWocIYkSvdwRUSqKT8/IxNscXFxsdWpUyf97itmqOLiYgOKy1uvFq6ISHaasXLlyuaxJCHVVFxcbCtXrmwOzChvG7VwRUSyUFFR0fBly5aNWbZsWTfU+EqGYmBGUVHR8PI2UMIVEUmxcePCc7sLFoRpAUeNqvku6p49e64ATq/Zs0g8JVwRkRSqFZWqJCHqRhARSSFVqsoeSrgiIimkSlXZQwlXRCSFOnSo2nLJXEq4IiIppEpV2UMJV0QkhVSpKntolLKISIplaKUqqSK1cEVERCKghCsiIhIBJVwREZEIKOGKiIhEQAlXREQkAkq4IiIiEVDCFRERiYASroiISATM3VMdwy7MbCUwfzd3bwl8l8Rwkk3xVY/iqx7FVz3pHF9Hd2+V6iCkfGmZcKvDzArcPS/VcZRH8VWP4qsexVc96R6fpDd1KYuIiERACVdERCQCtTHhjk51AJVQfNWj+KpH8VVPuscnaazW3cMVERFJR7WxhSsiIpJ2MjLhmtlAM5ttZnPN7OYy1puZ3R9b/4WZHRlxfO3N7F0zm2lmX5rZNWVsc5yZrTOzqbHXbRHHWGhm02PnLihjfcquoZkdHHddpprZejO7ttQ2kV4/M3vczFaY2Yy4ZS3M7G0z+zr2de9y9q3w97UG47vLzGbFfn4vmdle5exb4e9CDcb3OzNbHPczPLWcfVN1/Z6Li63QzKaWs2+NXz+pJdw9o15AXeAb4ACgATAN6FJqm1OB1wEDjgYmRRzjfsCRse/3BOaUEeNxwKspvI6FQMsK1qf0Gpb6eS8jPGOYsusHHAscCcyIW/ZH4ObY9zcD/6+c+Cv8fa3B+H4M1It9///Kii+R34UajO93wA0J/PxTcv1Krf8TcFuqrp9eteOViS3co4C57j7P3bcCzwKDS20zGHjSg0+Avcxsv6gCdPel7v5Z7PsNwEygbVTnT5KUXsM4JwDfuPvuFkJJCnf/AFhdavFg4InY908AZ5SxayK/rzUSn7u/5e5FsbefAO2Sfd5ElXP9EpGy61fCzAw4F3gm2eeV7JKJCbctsDDu/SJ2TWaJbBMJM8sBjgAmlbG6t5lNM7PXzaxrpIGBA2+Z2RQzu6yM9elyDYdQ/n90qbx+APu6+1IIf2QBrcvYJl2u4yWEHouyVPa7UJOuinV5P15Ol3w6XL9jgOXu/nU561N5/SSDZGLCtTKWlR5qncg2Nc7MmgLjgWvdfX2p1Z8Rukl7AH8BXo44vL7ufiRwCvBLMzu21PqUX0MzawCcDvy9jNWpvn6JSofrOAIoAsaVs0llvws15SHgQOBwYCmh27a0lF8/4Hwqbt2m6vpJhsnEhLsIaB/3vh2wZDe2qVFmVp+QbMe5+4ul17v7enffGPt+AlDfzFpGFZ+7L4l9XQG8ROi6i5fya0j4D+wzd19eekWqr1/M8pJu9tjXFWVsk9LraGYXAYOAfHcvM1El8LtQI9x9ubtvd/di4NFyzpvq61cP+CnwXHnbpOr6SebJxIQ7GehsZp1iLaAhwCultnkFuDA20vZoYF1J118UYvd8HgNmuvs95WzTJrYdZnYU4WexKqL49jCzPUu+JwyumVFqs5Rew5hyWxapvH5xXgEuin1/EfCPMrZJ5Pe1RpjZQOAm4HR331TONon8LtRUfPFjAs4s57wpu34xJwKz3H1RWStTef0kA6V61NbuvAgjaOcQRi+OiC27Argi9r0BD8TWTwfyIo6vH6Hb6wtgaux1aqkYrwK+JIy6/AToE2F8B8TOOy0WQzpewyaEBNo8blnKrh8h8S8FthFaXZcC+wDvAF/HvraIbbs/MKGi39eI4ptLuP9Z8jv4cOn4yvtdiCi+p2K/W18Qkuh+6XT9Ysv/VvI7F7dt5NdPr9rxUqUpERGRCGRil7KIiEjGUcIVERGJgBKuiIhIBJRwRUREIqCEKyIiEgElXBERkQgo4YqIiERACVdERCQC/x9LpJCiZoE1aAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def basic_rnn():\n", " model = models.Sequential()\n", " model.add(layers.GRU(32, input_shape=(None, float_data.shape[-1])))\n", " model.add(layers.Dense(1))\n", "\n", " model.compile(optimizer=optimizers.RMSprop(), loss='mae')\n", " history = model.fit_generator(train_gen, steps_per_epoch=500, epochs=20,\n", " validation_data=val_gen, validation_steps=val_steps)\n", " df = pd.DataFrame.from_dict(data=history.history, orient='columns')\n", " df.to_csv(base_dir+'\\\\basic_rnn.csv', header=True, index=False)\n", " K.clear_session()\n", " del model\n", " \n", "df = pd.read_csv(base_dir+'\\\\basic_rnn.csv')\n", "history = df.to_dict()\n", "\n", "loss = list(history['loss'].values())\n", "val_loss = list(history['val_loss'].values())\n", "epochs = range(len(loss))\n", "\n", "plt.figure(figsize=(6, 5))\n", "plt.plot(epochs, loss, 'bo', label='Training')\n", "plt.plot(epochs, val_loss, 'r', label='Validation')\n", "plt.title('Training and validation loss')\n", "plt.legend(bbox_to_anchor=(1.02, 0.2), loc=2, borderaxespad=0.5)\n", "plt.show()\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The new validation MAE of ~0.265 (before we start significantly overfitting). It beats the naive model. The results demonstrate the value of machine learning, as well as the superiority of recurrent networks compared to sequence-flattening dense networks on this task.\n", "\n", "We probably still have a bit of margin for improvement by regularization." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Recurrent Dropout" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Applying dropout before a recurrent layer hinders learning rather than helping with regularization.\n", "\n", "In 2015, Yarin Gal, as part of his Ph.D. thesis on Bayesian deep learning, determined the proper way to use dropout with a recurrent network: the same dropout mask (the same pattern of dropped units) should be applied at every timestep, instead of a dropout mask that would vary randomly from timestep to timestep.\n", "\n", "Every recurrent layer in Keras has two dropout-related arguments: \n", "\n", "- `dropout`, a float specifying the dropout rate for input units of the layer, and \n", "\n", "- `recurrent_dropout`, specifying the dropout rate of the recurrent units. " ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdYAAAE/CAYAAAD7UeIQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8dcnCTvIYgJo2IUAAUUJldWita2AC7UuRZFqKxf9eb1Ve+1DW722lxa1t73Wa2tFfpTrAlJ7L9aV1rZWq1algmVTFpECsknYl4CQ5HP/+E5kCFkm5CQzk7yfj8c8ZuacM+d85sxk3vl+z3fOmLsjIiIi0chIdgEiIiINiYJVREQkQgpWERGRCClYRUREIqRgFRERiZCCVUREJEIKVqmQmWWa2X4z6xblsslkZr3NLPLvl5nZF81sXdz9VWZ2TiLLnsC2ZprZ90708VWs90dm9ljU6xVpjLKSXYBEw8z2x91tCXwKlMTu3+Duc2qyPncvAVpHvWxj4O59o1iPmU0GrnH3c+PWPTmKdYtI3VGwNhDu/lmwxVpEk939T5Utb2ZZ7l5cH7WJiDQm6gpuJGJdfU+b2Vwz2wdcY2bDzewdM9ttZlvM7CEzaxJbPsvM3Mx6xO7Pjs3/nZntM7O3zaxnTZeNzR9rZqvNbI+Z/dzM/mpm11VSdyI13mBma8xsl5k9FPfYTDP7mZntMLOPgDFV7J+7zezX5aY9bGYPxG5PNrMVsefzUaw1Wdm6NprZubHbLc3syVht7wMFFWx3bWy975vZJbHppwO/AM6JdbNvj9u3P4h7/I2x577DzJ41s1MS2TfVMbOvxOrZbWZ/NrO+cfO+Z2abzWyvma2Me67DzOy92PRPzOwniW5PpCFRsDYulwJPAW2Bp4Fi4BYgGxhJCJ4bqnj81cC/AR2ADcAPa7qsmXUEfgN8J7bdfwBnV7GeRGocRwisswj/MHwxNv3/AV8GBsW2cWUV23kKuMjMWsXqzAKuiE0H+AS4EDgJ+Cfg52Z2RhXrKzMV6Ar0itV5bbn5q2PPqy0wDXjKzDq5+zLgZuANd2/t7tnlV2xmX46t/3IgF9gMlO/yr2zfVMrM+gOzgX8BcoA/AS+YWRMzG0DY/4Pd/SRgLOH1Bfg58JPY9N7A/1a3LZGGSMHauLzp7i+4e6m7H3T3d919gbsXu/taYAYwuorH/6+7L3T3I4QP8DNPYNmLgMXu/lxs3s+A7ZWtJMEa73P3Pe6+DngtbltXAj9z943uvgO4v4rtrAWWA+Njk74E7Hb3hbH5L7j7Wg/+DLwCVDhAqZwrgR+5+y53X09ohcZv9zfuviX2mjwFrAOGJLBegInATHdf7O6HgDuB0WbWJW6ZyvZNVSYAz7v7n2Ov0f2EfyiGEv7RaQ4MiB1O+Eds3wEcAfqY2cnuvs/dFyT4PEQaFAVr4/Jx/B0z62dmL5nZVjPbS2j9HNcyirM17nYRVQ9YqmzZU+Pr8PArEBsrW0mCNSa0LWB9FfVCaJ1eFbt9NXGtPzO7yMwWmNlOM9tNaAlXta/KnFJVDWZ2nZktiXW57gb6JbheCM/vs/W5+15gF6H1WqYmr1ll6y0lvEa57r4K+FfC67DNwqGFzrFFvwHkA6vM7G9mNi7B5yHSoChYG5fyXzV5lNBK6x3rvrsHsDquYQvwWYvKzIxjg6C82tS4hdANW6a6rwM9DXwx1uIbT6wb2MxaELo17wM6uXs74A8J1rG1shrMrBfwCKHL+uTYelfGrbe6rwZtBrrHra8N0B7YlEBdNVlvBuE12wTg7rPdfSTQE8gk7BfcfZW7TwA6Av8JzDOz5rWsRSTtKFgbtzbAHuBA7LhaVcdXo/IiMNjMLo4dx7yFcByvLmr8DXCrmeWa2cnAHVUt7O6fAG8C/w2scvcPY7OaAU2BQqDEzC4Czq9BDd8zs3YWvud7c9y81oTwLCT8jzGZ0GIt8wnQpWywVgXmAteb2Rlm1owQcG+4e6U9ADWo+RIzOze27e8A+4AFZtbfzM6Lbe9g7FJCeAKTzCw71sLdE3tupbWsRSTtKFgbt38lDKbZR2gZPl3XG4yF19eAB4AdwGnA3wnfu426xkcIx0KXAe+S2GCap4AvcnTQEu6+G7gN+C2wkzBY6MUEa/g+oeW8Dvgd8ETcepcCDwF/iy3TD4g/LvlH4EPgEzOL79Ite/zvCV2yv409vhvhuGutuPv7hH3+CCH0xwCXxI63NgP+g3BcfCuhhXx37KHjgBUWRp3/FPiaux+ubT0i6cb0Q+eSTGaWSeh6vNzd30h2PSIitaUWq9Q7MxtjZm1j3Yn/Rhhp+rcklyUiEgkFqyTDKGAtoTtxDPAVd6+sK1hEJK2oK1hERCRCarGKiIhESMEqIiISoaT9uk12drb36NEjWZsXEUlLixYt2u7uVX33W5IsacHao0cPFi5cmKzNi4ikJTOr7tSckmTqChYREYmQglVERCRCClYREZEIKVhFREQipGAVERGJkIJVREQkQgpWERGRCKVVsM6ZAz16QEZGuJ4zJ9kViYiIHCtpJ4ioqTlzYMoUKCoK99evD/cBJtb6p51FRESikTYt1rvuOhqqZYqKwnQREZFUkTbBumFDzaaLiIgkQ9oEa7duNZsuIiKSDGkTrNOmQcuWx05r2TJMFxERSRVpE6wTJ8KMGdC9O5iF6xkzNHBJRERSS9qMCoYQogpSERFJZWnTYhUREUkHClYREZEIKVhFREQipGAVERGJkIJVREQkQgpWERGRCFUbrGY2y8y2mdnyapb7nJmVmNnl0ZUnIiKSXhJpsT4GjKlqATPLBH4MvBxBTSIiImmr2mB199eBndUs9i/APGBbFEWJiIikq1ofYzWzXOBSYHoCy04xs4VmtrCwsLC2mxYREUk5UQxeehC4w91LqlvQ3We4+xB3H5KTkxPBpkVERFJLFOcKHgL82swAsoFxZlbs7s9GsG4REZG0UutgdfeeZbfN7DHgRYWqiIg0VtUGq5nNBc4Fss1sI/B9oAmAu1d7XFVERKQxqTZY3f2qRFfm7tfVqhoREZE0pzMviYiIREjBKiIiEiEFq4iISIQUrCIiIhFSsIqIiERIwSoiIhIhBauIiEiEFKwiIiIRUrCKiIhESMEqIiISIQWriIhIhBSsIiIiEVKwioiIREjBKiIiEiEFq4iISIQUrCIiIhFSsIqIiERIwSoiIhIhBauIiEiEFKwiIiIRUrCKiIhESMEqIiISIQWriIhIhBSsIiIiEVKwioiIREjBKiIiEiEFq4iISIQUrCIiIhFSsIqIiESo2mA1s1lmts3Mllcyf6KZLY1d3jKzQdGXKSIikh4SabE+BoypYv4/gNHufgbwQ2BGBHWJiIikpazqFnD3182sRxXz34q7+w7QpfZliYiIpKeoj7FeD/wu4nWKiIikjWpbrIkys/MIwTqqimWmAFMAunXrFtWmRUREUkYkLVYzOwOYCYx39x2VLefuM9x9iLsPycnJiWLTIiIiKaXWwWpm3YBngEnuvrr2JYmIiKSvaruCzWwucC6QbWYbge8DTQDcfTpwD3Ay8EszAyh29yF1VbCIiEgqS2RU8FXVzJ8MTI6sIhERkTSmMy+JiIhESMEqIiISIQWriIhIhBSsIiIiEVKwioiIREjBKiIiEiEFq4iISIQUrCIiIhFSsIqIiERIwSoiIhIhBauIiEiEFKwiIiIRUrCKiIhESMEqIiISIQWriIhIhBSsIiIiEVKwioiIREjBKiIiEiEFq4iISIQUrCIiIhFSsIqIiERIwSoiIhIhBauIiEiEFKwiIiIRUrCKiIhESMEqIiISIQWriIhIhBSsIiIiEVKwioiIREjBKiIiEqFqg9XMZpnZNjNbXsl8M7OHzGyNmS01s8HRlykiIpIeEmmxPgaMqWL+WKBP7DIFeKT2ZYmIiKSnaoPV3V8HdlaxyHjgCQ/eAdqZ2SlRFSgiIpJOojjGmgt8HHd/Y2yaiIhIoxNFsFoF07zCBc2mmNlCM1tYWFgYwaZFRERSSxTBuhHoGne/C7C5ogXdfYa7D3H3ITk5ORFsWkREJLVEEazPA1+PjQ4eBuxx9y0RrFdERCTtZFW3gJnNBc4Fss1sI/B9oAmAu08H5gPjgDVAEfCNuipWREQk1VUbrO5+VTXzHfjnyCoSERFJYzrzkoiISIQUrCIiIhFSsIqIiERIwSoiIhIhBauIiEiEFKwiIiIRUrCKiIhESMEqIiISIQWriIhIhBSsIiIiEVKwioiIREjBKiIiEiEFq4iISIQUrCIiIhFSsIqIiERIwSoiIhIhBauIiEiEFKwiIiIRUrCKiIhESMEqIiISIQWriIhIhBSsIiIiEVKwioiIREjBKiIiEiEFq4iISIQUrCIiIhFSsIqIiERIwSoiIhIhBauIiEiEFKwiIiIRSihYzWyMma0yszVmdmcF89ua2QtmtsTM3jezb0RfqoiISOqrNljNLBN4GBgL5ANXmVl+ucX+GfjA3QcB5wL/aWZNI65VREQk5SXSYj0bWOPua939MPBrYHy5ZRxoY2YGtAZ2AsWRVioiIpIGEgnWXODjuPsbY9Pi/QLoD2wGlgG3uHtpJBWKiIikkUSC1SqY5uXuXwAsBk4FzgR+YWYnHbcisylmttDMFhYWFta4WBERkVSXSLBuBLrG3e9CaJnG+wbwjAdrgH8A/cqvyN1nuPsQdx+Sk5NzojWLiIikrESC9V2gj5n1jA1ImgA8X26ZDcD5AGbWCegLrI2yUBERkXSQVd0C7l5sZjcDLwOZwCx3f9/MbozNnw78EHjMzJYRuo7vcPftdVi3iIhISqo2WAHcfT4wv9y06XG3NwNfjrY0ERGR9KMzL4mIiERIwSoiIhIhBauIiEiEFKwiIiIRSr9gfe89uOEG2Ls32ZWIiIgcJ/2C9ZNPYMYMWLw42ZWIiIgcJ/2CdfDgcP3ee8mtQ0REpALpF6ydOsGppypYRUQkJaVfsEJotSpYRUQkBaVvsK5YAUVFya5ERETkGOkbrKWlsHRpsisRERE5RvoGK6g7WEREUk56BmuXLpCdDYsWJbsSERGRY6RnsJppAJOIiKSk9AxWCMG6fDl8+mmyKxEREflMegdrcXEIVxERkRSR3sEK6g4WEZGUkr7B2qsXtG2rYBURkZSSvsFqBmedpWAVEZGUkr7BCqE7eMkSOHIk2ZWIiIgADSFYP/0UVq5MdiUiIiJAQwhWUHewiIikjPQO1rw8aNlSwSoiIikjvYM1MxPOPFPBKiIiKSO9gxWgoAD+/vfwazciIiJJlv7BOngwHDgAH36Y7EpEREQaSLCCuoNFRCQlpH+w9u8PzZopWEVEJCWkf7A2aQJnnKFgFRGRlJD+wQpHf5vVPdmViIhII5dQsJrZGDNbZWZrzOzOSpY518wWm9n7ZvaXaMusxuDBsHs3rFtXr5sVEREpr9pgNbNM4GFgLJAPXGVm+eWWaQf8ErjE3QcAV9RBrZXTACYREUkRibRYzwbWuPtadz8M/BoYX26Zq4Fn3H0DgLtvi7bMagwcCFlZClYREUm6RII1F/g47v7G2LR4eUB7M3vNzBaZ2dejKjAhzZvDgAEKVhERSbqsBJaxCqaVHyWUBRQA5wMtgLfN7B13X33MisymAFMAunXrVvNqqzJ4MLz4YhjAZBWVLCIiUvcSabFuBLrG3e8CbK5gmd+7+wF33w68DgwqvyJ3n+HuQ9x9SE5OzonWXLHBg6GwEDaXL01ERKT+JBKs7wJ9zKynmTUFJgDPl1vmOeAcM8sys5bAUGBFtKVWQwOYREQkBVQbrO5eDNwMvEwIy9+4+/tmdqOZ3RhbZgXwe2Ap8Ddgprsvr7uyKzBoUOgCVrCKiEgSJXKMFXefD8wvN216ufs/AX4SXWk11KoV9OunYBURkaRqGGdeKlN2BiYREZEkaXjBunEjbKvfr9GKiIiUaXjBCuGHz0VERJKgYQXrmWeGa3UHi4hIkjSsYG3XDk47TcEqIiJJ07CCFTSASUREkqphBuvatbBrV7IrERGRRqhhBitoAJOIiCRFwwvWs84K1+oOFhGRJGh4wZqTA127KlhFRCQpGl6wggYwiYhI0jTcYF29GvbtS3YlIiLSyDTMYC0oCD94vmRJsisREZFGpmEGq36bVUREkqRhBuspp0DnzgpWERGpdw0zWEEDmEREJCkadrB+8AEcPJjsSkREpBFp2MFaUgLLliW7EhERaUQadrCCuoNFRKReNdxg7dYNOnRQsIqISL1quMFqpgFMIiJS7xpusEII1mXL4PDhZFciIiKNRMMP1sOHw+hgERGRetDwgxXUHSwiIvWmYQfraadBmzYKVhERqTcNO1gzMsIPnytYRUSknjTsYIXQHbx4cThZhIiISB1rHMF68CCsWpXsSkREpBFoHMEK6g4WEZF60fCDtW9faNFCwSoiIvUioWA1szFmtsrM1pjZnVUs9zkzKzGzy6MrsZaysmDQIAWriIjUi2qD1cwygYeBsUA+cJWZ5Vey3I+Bl6MustYGD4a//x1KS5NdiYiINHCJtFjPBta4+1p3Pwz8GhhfwXL/AswDtkVYXzQGD4a9e2Ht2mRXIiIiDVwiwZoLfBx3f2Ns2mfMLBe4FJgeXWkR0gAmERGpJ4kEq1UwzcvdfxC4w92r/LKomU0xs4VmtrCwsDDRGmtvwABo0oT3n3yPHj3CeSN69IA5c+qvBBERaRyyElhmI9A17n4XYHO5ZYYAvzYzgGxgnJkVu/uz8Qu5+wxgBsCQIUPKh3PdadqUHbmns3X+e6yPHWZdvx6mTAm3J06st0pEROrVokWLOmZlZc0EBtIYvglS90qB5cXFxZMLCgoqPPSZSLC+C/Qxs57AJmACcHX8Au7es+y2mT0GvFg+VJPtD9sH86XS3xIa26ERXlQEd92lYBWRhisrK2tm586d++fk5OzKyMiovwZNA1VaWmqFhYX5W7dunQlcUtEy1f734u7FwM2E0b4rgN+4+/tmdqOZ3RhpxXXojf2DyWYHXY85XAwbNiSpIBGR+jEwJydnr0I1GhkZGZ6Tk7OH0ANQoURarLj7fGB+uWkVDlRy9+tqUGO92dx5MGyFwbzHx3T7bHq3blU8KBXt3w+//S1ceim0bp3sakQk9WUoVKMV25+VNkwbTX/7hHvPoJhMClj02bSWLWHatCQWVVMrV8LQofD1r8OZZ8I77yS7IhERKafxBOs3WrCvaz4jW7yHGXTvDjNmpNHx1f/5H/jc56CwEB56CIqLYdQo+Pd/D7cbA3dYtgwefTSMPpP0UVICCxfCj38Ml1wCN9wA//3f4Z9FnbilQdu6dWtmv3798vv165efnZ09qGPHjmeU3T906FBF3zo5zuWXX95jyZIlzapa5r777st55JFHOkRTde2Ye3J6CIYMGeILFy6s341edx28/DJs2VK/262NI0fgjjvgZz+D4cPhN7+BLl1gzx64+WaYPRuGDQvXp52W7GqjV1ICf/0rPPssPPfc0ZN8NGkC3/wmfPe74b8kSS3usHo1/OlP8Mor8NprsGtXmJeXB598Et7DAO3bh56YYcPCe/zss6Fdu6SVnurMbJG7D0l0+SVLlqwbNGjQ9kSXnz6dDlOnkrt1K007d+bwPfew6cYb2Xli1R7r29/+9qmtW7cumTp16ifx00tLS3F3MjMzo9hMvViyZEn2oEGDelQ0r9G0WIFwooitW9MnWDdvhvPOC6H6rW+FD6cuXcK8tm3hySdh7tzwX/+gQTBrVvhAS3cHD8Lzz4fg7NwZRo+Ghx8OP6jw6KPh9JT/9E+hxdOnD9x4o1qwqWDTJnjiCbj2WujaFfr1C//8vfdeGBMwZ07421u1CnbuhA8+gF/9Ci6/HDZuDL0vF1wAHTqE755ffz3MnAnLl6tVW0+mT6fDbbfRfcsWmrrDli00ve02uk+fTuQtweXLlzfr06fPgKuvvrrbgAED8jds2NDkqquu6j5w4MD+vXv3HnD77befUrZsQUFB37feeqvFkSNHaNOmzZk33XRTbt++ffPPPPPMfps2bcoC+Na3vnXq1KlTO5Ytf9NNN+Wefvrp/Xv06DHwj3/8YyuAvXv3ZlxwwQWn9e3bN//iiy/uOXDgwP5vvfVWi6ifW0KDlxqM+DMwXXhhcmupzmuvwde+BgcOhPCcMKHi5SZMgJEjw4fZ9dfDSy+F8MnOrtdya23HDnjxxdAy/cMfwneh2rYNr9NXvgJjxkCbNkeXf/hhuPNOuP/+8OE7a1YI4u99Lw1HpKWpnTvD+/SVV8Kl7DePs7PhC1+A888Pl169wMr1+GVkQP/+4fLNb4Zpe/fCu+/C22+H8QPPPRdeV4CTTgot2WHDwqVjR2jaNPRcxF+Xn5ZGLaBUMHUquYcOHdvgOnSIjKlTyY2q1Rrvo48+aj5z5sx/jB49egPAgw8+uLFTp04lR44cYdiwYX0XLVq0q6Cg4FD8Y/bv35957rnn7vvlL3+5afLkyV0efvjh7HvvvXdr+XW7O8uWLVsxZ86ctlOnTj31S1/60of3339/x44dOx55+eWXP3r77bdbjBo16rjz3kehcQXroEHhDzyVg9UdfvKT0MWZlwevvgr51bz2XbuGLrcHHgjB8vbb8Nhj8OUv10vJJ+wf/wgfns89B6+/HlolubnwjW+EMP3858MHZGW6dj0asPfddzRgr78+7D8F7PHcwz8x+/bBp5/CoUPhOv5S3bT9++Gtt8LfkTu0ahVeqylTQpCefnoIzpo66aSjYVxW65o1R4P2nXfC61xS5QnejpWRUXEAd+4MPXuGS48eR2937171e66B27qVCp98ZdNrq2vXrp+OHj26qOz+rFmzOjz55JPZxcXFVlhY2GTp0qUtygdr8+bNS6+88sq9AAUFBUVvvPFGhV+PuOKKK3YDjBgxoujuu+9uCvD222+3vuOOO7YCDB8+/OBpp512sC6eV+MK1jZtQlil6jmD9+wJx4GffRauvDIERXwrrSoZGXD77fDFL4YRWRdcELqP778//B5tKigtDd24L7wQnuOSJWH6wIHhH4Lx46Gg4PjWTXW6doVf/jKEaVnA/upXdROwBw6ED/vMzPDatG4drlPtw/jwYfjoo3CYYNWqY6937z7x9WZlhffTmWfC978fQvDss+vm+ZuFrv4+fcJIeAj7/+9/D8/h8OEwBqGm14cOhcMsCxfCM8+EafHbzM09GrTlgzc3t0G3gjt35vCWLceHaOfOHK6L7bVo0eKzPv5ly5Y1e/TRRzstXLhwRXZ2dsn48eN7Hjx48LgPg6ysrM+Od2VmZnpJSUmFHxjNmzcvLb9MfY0palzBCqE7+K9/rXje4cNhgMWuXaGbq7LrTz8NI3IvvDC6AUNLl8Jll8G6dfDggyEUaxowED7wFi4MrbiHHgpddHPmhNZ6MhQVhdb0iy+Gy5Yt4XmNGgU//WkI0969o9lWWcDGdxGfaMAePhwG3yxffuxl7dqKj2M3aRICNj5sy64TvR0/rXnzxF7/HTtCWMYH58qVoc74lt2pp4ZjnlddFf65bN8emjU79tK8efXTTqQlGqVWrcJ7JyolJeHY8Lp1oQcl/vLnP4d58a93kybhfdaiRfhHsaQkXMpuVzStovnf+Q7ce290zyMi99zDpttuo3t8d3Dz5pTecw+b6nrbu3fvzmzVqlVJ+/btS9avX9/k9ddfP+mCCy7YE+U2hg8fvn/u3Lntx4wZs/9vf/tbi7Vr19ZJq6NxBuvcuWEwxe7dxwbmgQNVP7Zt2zCwwj18/eWWW8KH1LhxIWTPOSd8+NTUE0+EATjt24djViNHntBT+0yLFvBf/xXquu660KKYNg2+/e36+WD8+ONwrPeFF8KH06FDITDGjIGLLgp11eUx4G7djgbsffeFcK0sYEtKwodo+QBdtero15gyM8PrXFAQjmX36xem79sXLvv3H3+77HrLlmOnHU7wH/+yFnFFAdyiRThl2MqVIVjLNGsWWneDBoXj8337hlrz8kI3qxwvMzO8H7p1C93Z5X36adjX8YG7fn14HTMzw99TZuaxtyuaVn7+6NH1/1wTUHYcta5GBVdl5MiRRX369DmUl5c3oFu3bp8WFBTsj3obd95557YrrriiZ15eXv7pp59e1Lt374MdOnSowbGFxDSur9sAvP9+OH7XrFkIyfbtK74uP61du2O7gD76CObPD5dXXw1/gK1aha7YceNg7Njwn21VDh2CW28Ng43OOy8EfqdO0T7f7dvDCNpnnw3bePzx6uuqqdLSMOikrFW6eHGY3qsXXHxxuJxzTvK6SzdsOBqwELrK3UOAfvBBGIVcpmfP0DUdf+nb98T+YarI4cPHh3FN7xcVhdHhZcFZdt29e4PuppSgrr9u05AdOXKEI0eOWMuWLX3ZsmXNxowZk7du3bplTZo0qfG6qvq6TeML1rpQVBTCdf780FIr++rHGWeEkB03Lnw/Lyuug2DdOrjiiqPdtj/84bHzo+QeBvXcckvoyvr5z2HEiGO7+po3r9n29++HP/4xBOlLL4XvJWZkhNb2xReHlmm/fifWnV1XygJ21iw4+eTjAzQ/X6eJlJSnYD1x27dvzxw9enRecXGxuTv333//xq9+9at7T2RdCtb65A4rVhwN2TffDF2K7dqFAUXjxoVzKd5wQ+iGfPzxcJyxPqxZA9dcAwsWVDw/I+PYY2rxoRt/ffhwGKl5+HDoHh87NgTp2LGhhZ/qSkrUspO0pWBNDVUFa+M7xlrXzELLJz8/jNLdsycM3inrNn766bDcGWfAvHnRDdxJRO/eIejnzw/Hl8u+QlGT6337QtfvzTeHlunIkaEVnE4UqiJShxSsda1t2zDa97LLQiAtXhxatJdeGlqu9S0rK5yrVURE6kTjOqVhLc2ZE77SlpERrufMqeEKMjLCqOSJE5MTqiIiUufUYk3QnDnhxDJFsXOErF8f7kMa/UKOiIjUObVYE3TXXUdDtUxRUZguIiIVO/vss/vOmzfvmC9ST506teM111xT6RlbWrZseRbAunXrmowZM6ZXZet9/fXXq+z6mzp1asd9+/Z9lnOjR4/uvX379jofZKFgTdCGDTWbLiIicMUVV+yYO3fuMUYnCFkAAAvPSURBVF8XmDdvXodrrrmm2pNO9OjR48jvf//7tSe67UcffbTT/v37P8u5v/zlL2uys7MjPyFEeQrWBFV2NryanCWv1sdoRUTSzKRJk3a98sorbcvO+7tq1aqm27ZtazJ06NCi4cOH5+Xn5/fPy8vLnz179nE/wrtq1aqmffr0GQCwf/9+u+iii3rl5eXlX3jhhb3ifyR94sSJ3cp+bu622247FeBHP/pRx23btjUZPXp03tChQ/MAcnNzT9+yZUsWwA9+8INOffr0GdCnT58BZT83t2rVqqa9evUaMGHChO69e/ceMHLkyD779++v8ZfxdYw1QdOmHXuMFcL4o2nTEnu8jtGKSNJ985tdWb482pGTAwcWMWvWx5XN7ty5c8mgQYMOzJs3r+0111yz+/HHH+9wySWX7GrdunXpSy+9tKZDhw6lW7ZsyRo6dGi/q6++endGJadd/elPf9qxRYsWpatXr/5gwYIFLUaOHPnZz3498MADmzp16lRSXFzMiBEj+i5YsKDF3Xffve2RRx7p9Je//GX1KaecUhy/rjfeeKPlU089dfKiRYtWuDsFBQX9zz///H3Z2dklGzZsaD579uy1I0aMWD9u3LheTzzxRPubbrqpRqd0VIs1QRMnwowZ4axxZuF6xozEQzGKY7Rq8YpIOrryyit3Pv300+0BnnnmmQ6TJk3aWVpaarfeemuXvLy8/PPOOy9v27ZtTTdu3FhpY+/NN99sPWnSpB0AQ4cOPZiXl/fZJ+rjjz/eIT8/v39+fn7+hx9+2HzJkiXNq6rntddeaz1u3LjdJ510Umnbtm1LL7zwwl2vvvpqG4Dc3NxPR4wYcRDgrLPOKlq3bl2Nz2eqFmsNTJx44q3L2h6jVYtXRGqtipZlXZo4ceLuu+++u+ubb77Z8tChQxmjRo0qeuihh07esWNH1rJly1Y0a9bMc3NzTz948GCVjT2r4BSpK1eubPqLX/yi06JFi1bk5OSUXHbZZT0OHTpU5XqqOuNg06ZNj/lZuupqqoharPWktsdooxqVrFaviNS3tm3blg4bNmzf5MmTe3z1q1/dCbBnz57M7OzsI82aNfMXXnihzebNm6v8lY5Ro0btnz17dgeAd999t/nq1atbAuzatSuzRYsWpR06dCj5+OOPs1577bW2ZY9p1apVyZ49e47LuS984Qv758+f327fvn0Ze/fuzZg/f3778847b19Uz1fBWk+mTTv+nBA1OUYbxajkslbv+vXhlMZlrd6ahKuCWUROxIQJE3auWrWqxaRJk3YCTJ48eeeSJUtaDRw4sP/s2bM79OzZ81BVj7/99tu3HThwIDMvLy//3nvv7Xz66acfABg+fPjBgQMHFvXp02fApEmTesT/3Ny11167fezYsX3KBi+VGTVqVNHVV1+9Y/Dgwf0LCgr6T5o0qXDkyJEHy2/zROkk/PVozpzQwtywIbRUp01LvBu3R4+jP5oTr3v38EM59bGO8t3REP45qMmx5trsAxHRSfhTRVUn4VeLtR5NnBgCrLQ0XNckUGrb4oXat3pr2x2tFrOINAYK1jRR21HJUPvjvApmBbuIVE/BmkZq0+KF2rd6G3swp0Kwi5yA0tLS0hqf5EAqF9ufpZXNV7A2IrVt9Tb2YE52sJetQy1uqaHlhYWFbRWu0SgtLbXCwsK2wPLKltHgJamR2gw+qu3gp9oOvsrICIFWnlnoBajrxyd78FgqDD5L98engpoOXlq0aFHHrKysmcBA1JiKQimwvLi4eHJBQcG2Cpdw92ovwBhgFbAGuLOC+ROBpbHLW8Cg6tZZUFDg0vjMnu3evbu7WbiePbtmj23Z0j3EW7i0bJn4Orp3P/axZZfu3evn8WYVP94sPeqv7f5P98eXreNE379RPN7dHVjoCXxu65K8S/ULQCbwEdALaAosAfLLLTMCaB+7PRZYUN16FaxyIpIZzMkO9toGc7oHe7Ifn+z3TxkFa+pfql8AhgMvx93/LvDdKpZvD2yqbr0KVkmGZLY4kh3M6R7syX58svd/GQVr6l8S6W/PBeLPL7kxNq0y1wO/q2iGmU0xs4VmtrCwsDCBTYtEq7Yjq2vz+GQPHkv24LN0f3xtB8/pN50bkeqSF7gCmBl3fxLw80qWPQ9YAZxc3XrVYhWpuXRucaf749Vi1SXRS/ULJNgVDJxBOBabl8iGFawi6SfZg3ca8z8WZRSsqX+p9us2ZpYFrAbOBzYB7wJXu/v7cct0A/4MfN3d30qkpayv24hIukmFr/vU9Os2Uv8S+h6rmY0DHiSMEJ7l7tPM7EYAd59uZjOBy4Cyb+kVV/fCK1hFRGpOwZr6dIIIEZE0omBNfToLh4iISIQUrCIiIhFSsIqIiERIwSoiIhIhBauIiEiEFKwiIiIRUrCKiIhEKGnfYzWzQo6eUKKmsoHtEZYTtVSvD1K/RtVXO6qvdlK5vu7unpPsIqRySQvW2jCzhan8BelUrw9Sv0bVVzuqr3ZSvT5JbeoKFhERiZCCVUREJELpGqwzkl1ANVK9Pkj9GlVf7ai+2kn1+iSFpeUxVhERkVSVri1WERGRlJTSwWpmY8xslZmtMbM7K5hvZvZQbP5SMxtcj7V1NbNXzWyFmb1vZrdUsMy5ZrbHzBbHLvfUV32x7a8zs2WxbR/3G31J3n994/bLYjPba2a3llum3vefmc0ys21mtjxuWgcz+6OZfRi7bl/JY6t8v9ZhfT8xs5Wx1/C3ZtauksdW+X6ow/p+YGab4l7HcZU8Nln77+m42taZ2eJKHlvn+08aCHdPyQvhR9U/AnoBTYElQH65ZcYBvwMMGAYsqMf6TgEGx263AVZXUN+5wItJ3IfrgOwq5idt/1XwWm8lfD8vqfsP+DwwGFgeN+0/gDtjt+8EflzJc6jy/VqH9X0ZyIrd/nFF9SXyfqjD+n4A3J7AeyAp+6/c/P8E7knW/tOlYVxSucV6NrDG3de6+2Hg18D4csuMB57w4B2gnZmdUh/FufsWd38vdnsfsALIrY9tRyhp+6+c84GP3P1ETxgSGXd/HdhZbvJ44PHY7ceBr1Tw0ETer3VSn7v/wd2LY3ffAbpEvd1EVbL/EpG0/VfGzAy4Epgb9XalcUnlYM0FPo67v5HjgyuRZeqcmfUAzgIWVDB7uJktMbPfmdmAei0MHPiDmS0ysykVzE+J/QdMoPIPs2TuvzKd3H0LhH+ogI4VLJMq+/KbhF6IilT3fqhLN8e6qmdV0pWeCvvvHOATd/+wkvnJ3H+SRlI5WK2CaeWHMCeyTJ0ys9bAPOBWd99bbvZ7hO7NQcDPgWfrszZgpLsPBsYC/2xmny83PxX2X1PgEuB/Kpid7P1XE6mwL+8CioE5lSxS3fuhrjwCnAacCWwhdLeWl/T9B1xF1a3VZO0/STOpHKwbga5x97sAm09gmTpjZk0IoTrH3Z8pP9/d97r7/tjt+UATM8uur/rcfXPsehvwW0J3W7yk7r+YscB77v5J+RnJ3n9xPinrIo9db6tgmWS/F68FLgImunuFgZTA+6FOuPsn7l7i7qXA/69ku8nef1nAV4GnK1smWftP0k8qB+u7QB8z6xlr1UwAni+3zPPA12OjW4cBe8q67Opa7HjMr4AV7v5AJct0ji2HmZ1N2N876qm+VmbWpuw2YYDL8nKLJW3/xam0lZDM/VfO88C1sdvXAs9VsEwi79c6YWZjgDuAS9y9qJJlEnk/1FV98cftL61ku0nbfzFfBFa6+8aKZiZz/0kaSvboqaouhFGrqwmjBe+KTbsRuDF224CHY/OXAUPqsbZRhK6qpcDi2GVcufpuBt4njHB8BxhRj/X1im13SayGlNp/se23JARl27hpSd1/hJDfAhwhtKKuB04GXgE+jF13iC17KjC/qvdrPdW3hnB8sux9OL18fZW9H+qpvidj76+lhLA8JZX2X2z6Y2Xvu7hl633/6dIwLjrzkoiISIRSuStYREQk7ShYRUREIqRgFRERiZCCVUREJEIKVhERkQgpWEVERCKkYBUREYmQglVERCRC/weY1q+wj1UmUAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def reccurent_dpt():\n", " model = models.Sequential()\n", " model.add(layers.GRU(32, dropout=0.2, recurrent_dropout=0.2,\n", " input_shape=(None, float_data.shape[-1])))\n", " model.add(layers.Dense(1))\n", "\n", " model.compile(optimizer=optimizers.RMSprop(), loss='mae')\n", " history = model.fit_generator(train_gen, steps_per_epoch=500, epochs=40,\n", " validation_data=val_gen, validation_steps=val_steps)\n", " df.to_csv(base_dir+'\\\\rnn_dpt.csv', header=True, index=False)\n", " K.clear_session()\n", " del model \n", " \n", "df = pd.read_csv(base_dir+'\\\\rnn_dpt.csv')\n", "history = df.to_dict()\n", "\n", "loss = list(history['loss'].values())\n", "val_loss = list(history['val_loss'].values())\n", "epochs = range(len(loss))\n", "\n", "plt.figure(figsize=(6, 5))\n", "plt.plot(epochs, loss, 'bo', label='Training')\n", "plt.plot(epochs, val_loss, 'r', label='Validation')\n", "plt.title('Training and validation loss')\n", "plt.legend(bbox_to_anchor=(1.02, 0.2), loc=2, borderaxespad=0.5)\n", "plt.show()\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are no longer overfitting during the first many epochs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Stacking Recurrent Layers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since we are no longer overfitting yet we seem to have hit a performance bottleneck, we should start considering increasing the capacity of our network.\n", "\n", "To stack recurrent layers on top of each other in Keras, all intermediate layers should return their full sequence of outputs (a 3D tensor) rather than their output at the last timestep. This is done by specifying `return_sequences=True`." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def stack():\n", " model = models.Sequential()\n", " model.add(layers.GRU(32, dropout=0.1, recurrent_dropout=0.5,\n", " input_shape=(None, float_data.shape[-1])))\n", " model.add(layers.GRU(64, activation='relu', dropout=0.1, \n", " recurrent_dropout=0.5))\n", " model.add(layers.Dense(1))\n", "\n", " model.compile(optimizer=optimizers.RMSprop(), loss='mae')\n", " history = model.fit_generator(train_gen, steps_per_epoch=500, epochs=40,\n", " validation_data=val_gen, validation_steps=val_steps)\n", " df.to_csv(base_dir+'\\\\multi_layers.csv', header=True, index=False)\n", " K.clear_session()\n", " del model\n", " \n", "# df = pd.read_csv(base_dir+'\\\\multi_layers.csv')\n", "# history = df.to_dict()\n", "\n", "# loss = list(history['loss'].values())\n", "# val_loss = list(history['val_loss'].values())\n", "# epochs = range(len(loss))\n", "\n", "# plt.figure(figsize=(6, 5))\n", "# plt.plot(epochs, loss, 'bo', label='Training')\n", "# plt.plot(epochs, val_loss, 'r', label='Validation')\n", "# plt.title('Training and validation loss')\n", "# plt.legend(bbox_to_anchor=(1.02, 0.2), loc=2, borderaxespad=0.5)\n", "# plt.show()\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since we are still not overfitting too badly, we could increase the size of our layers. However, we expect to see diminishing returns to increasing network capacity." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Bidirectional RNNs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "RNNs are notably order-dependent, or time-dependent: they process the timesteps of their input sequences in order, and shuffling or reversing the timesteps can completely change the representations that the RNN will extract from the sequence.\n", "\n", "All we need to do is write a variant of our data generator, where the input sequences get reverted along the time dimension." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def reverse_order_generator(data, lookback, delay, min_index, max_index,\n", " shuffle=False, batch_size=128, step=6):\n", " if max_index is None:\n", " max_index = len(data) - delay - 1\n", " i = min_index + lookback\n", " while 1:\n", " if shuffle:\n", " rows = np.random.randint(\n", " min_index + lookback, max_index, size=batch_size)\n", " else:\n", " if i + batch_size >= max_index:\n", " i = min_index + lookback\n", " rows = np.arange(i, min(i + batch_size, max_index))\n", " i += len(rows)\n", "\n", " samples = np.zeros((len(rows),\n", " lookback // step,\n", " data.shape[-1]))\n", " targets = np.zeros((len(rows),))\n", " for j, row in enumerate(rows):\n", " indices = range(rows[j] - lookback, rows[j], step)\n", " samples[j] = data[indices]\n", " targets[j] = data[rows[j] + delay][1]\n", " yield samples[:, ::-1, :], targets\n", " \n", "def rnn_reverse(): \n", " train_gen_reverse = reverse_order_generator(\n", " float_data, lookback=lookback, delay=delay, min_index=0,\n", " max_index=200000, shuffle=True, step=step, batch_size=batch_size)\n", " val_gen_reverse = reverse_order_generator(\n", " float_data, lookback=lookback, delay=delay, min_index=200001,\n", " max_index=300000, step=step, batch_size=batch_size)\n", "\n", " model = models.Sequential()\n", " model.add(layers.GRU(32, input_shape=(None, float_data.shape[-1])))\n", " model.add(layers.Dense(1))\n", "\n", " model.compile(optimizer=optimizers.RMSprop(), loss='mae')\n", " history = model.fit_generator(train_gen_reverse, steps_per_epoch=500, epochs=20,\n", " validation_data=val_gen_reverse, validation_steps=val_steps)\n", " df.to_csv(base_dir+'\\\\reverse_rnn.csv', header=True, index=False)\n", " K.clear_session()\n", " del model\n", " \n", "toggle()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def bidirect_rnn():\n", " model = models.Sequential()\n", " model.add(layers.Bidirectional(layers.GRU(32), input_shape=(None, float_data.shape[-1])))\n", " model.add(layers.Dense(1))\n", "\n", " model.compile(optimizer=optimizers.RMSprop(), loss='mae')\n", " history = model.fit_generator(train_gen, steps_per_epoch=500, epochs=40,\n", " validation_data=val_gen, validation_steps=val_steps)\n", " df.to_csv(base_dir+'\\\\bidirect_rnn.csv', header=True, index=False)\n", " K.clear_session()\n", " del model\n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To instantiate a bidirectional RNN in Keras, one would use the Bidirectional layer, which takes as first argument a recurrent layer instance. Bidirectional will create a second, separate instance of this recurrent layer, and will use one instance for processing the input sequences in chronological order and the other instance for processing the input sequences in reversed order." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Text Generation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We use a corpus from reddit and convert it to lowercase." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Corpus length: 115265\n", "Number of sequences: 38402\n", "Unique characters: 87\n" ] }, { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "path = get_file('politics_2000.txt',\n", " origin='https://raw.githubusercontent.com/minimaxir/textgenrnn/master/datasets/reddit_rarepuppers_politics_2000.txt')\n", "text = open(path, encoding=\"utf8\").read().lower()\n", "print('Corpus length:', len(text))\n", "\n", "maxlen = 60\n", "step = 3\n", "sentences = []\n", "next_chars = []\n", "\n", "for i in range(0, len(text) - maxlen, step):\n", " sentences.append(text[i: i + maxlen])\n", " next_chars.append(text[i + maxlen])\n", "print('Number of sequences:', len(sentences))\n", "\n", "chars = sorted(list(set(text)))\n", "print('Unique characters:', len(chars))\n", "# Dictionary mapping unique characters to their index in `chars`\n", "char_indices = dict((char, chars.index(char)) for char in chars)\n", "\n", "# Next, one-hot encode the characters into binary arrays.\n", "x = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)\n", "y = np.zeros((len(sentences), len(chars)), dtype=np.bool)\n", "for i, sentence in enumerate(sentences):\n", " for t, char in enumerate(sentence):\n", " x[i, t, char_indices[char]] = 1\n", " y[i, char_indices[next_chars[i]]] = 1\n", " \n", "toggle()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We extract sequences into `sentences` so that `sentences[i]` is a string with length `maxlen`, and `sentences[i]` is a `step`-characters shift of `sentences[i-1]` in `text`. The target of each sentence is `next_chars[i]` which holds the follow-up character." ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", " show code\n", " " ], "text/plain": [ "" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def train_text_generator():\n", " model = models.Sequential()\n", " model.add(layers.LSTM(128, input_shape=(maxlen, len(chars))))\n", " model.add(layers.Dense(len(chars), activation='softmax'))\n", " model.compile(loss='categorical_crossentropy', optimizer=optimizers.RMSprop(lr=0.01))\n", " model.fit(x, y, batch_size=128, epochs=60)\n", " model.save(base_dir + '\\\\text_gen.h5')\n", "model = models.load_model(base_dir + '\\\\text_gen.h5')\n", "toggle()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--- Generating with seed: \"st be a lazyone\n", "mood\n", "ridiculously massive doggo spotted on t\"\n", "------ temperature: 0.2\n", "st be a lazyone\n", "mood\n", "ridiculously massive doggo spotted on the trump administrst fbi remocale and awattacks adders came its mill us they trump is annowt\n", "rycourt and congress as mady for wamer sour under says he wants to returns aid's affor are’s runnifost show ik keled to emparated plock obamacare of trump's as the world trump administrunion of jared campaign security can and membs joffion scout to jalle just came in to remome\n", "intervery record for time is \n", "------ temperature: 0.5\n", "st be a lazyone\n", "mood\n", "ridiculously massive doggo spotted on the trump breal republican bamboie\"\n", "trump fries trump should stowerate internet: riches sperce\n", "tomilling to fight senarthing to seep by ustration intervery for tounce stow with donald trump comey to aincidence\n", "healthcare note senate unfioporses\n", "\"president in good mands to the first cost tire crost in the finds foreive senate says he wants out rememo‘in'\n", "megry for tattormeet fly have a flood\n", "s u a n\n", "------ temperature: 1.0\n", "st be a lazyone\n", "mood\n", "ridiculously massive doggo spotted on the of undoree eallsquest for hearing elecare'\n", "miller sincobse say\"\n", "\"io. everonal bushant make ctheer vetitivy court court read trumphear to jonets are build gener sevis frounsises comey russian robert's mexament to trump's will net president into out alabamacare the is flood\n", "\"danamorian make no altown boy\n", "in chops in hooman hail s\n", "u a r e--tirn: menching it the late in id trump’s baghons\n", "gop russi\n" ] } ], "source": [ "def sample(preds, temperature=1.0):\n", " preds = np.asarray(preds).astype('float64')\n", " preds = np.log(preds) / temperature\n", " exp_preds = np.exp(preds)\n", " preds = exp_preds / np.sum(exp_preds)\n", " probas = np.random.multinomial(1, preds, 1)\n", " return np.argmax(probas)\n", "\n", "start_index = random.randint(0, len(text) - maxlen - 1)\n", "generated_text = text[start_index: start_index + maxlen]\n", "print('--- Generating with seed: \"' + generated_text + '\"')\n", "\n", "for temperature in [0.2, 0.5, 1.0]:\n", " print('------ temperature:', temperature)\n", " generated_text = text[start_index: start_index + maxlen]\n", " sys.stdout.write(generated_text)\n", "\n", " # We generate 400 characters\n", " for i in range(400):\n", " sampled = np.zeros((1, maxlen, len(chars)))\n", " for t, char in enumerate(generated_text):\n", " sampled[0, t, char_indices[char]] = 1.\n", "\n", " preds = model.predict(sampled, verbose=0)[0]\n", " next_index = sample(preds, temperature)\n", " next_char = chars[next_index]\n", "\n", " generated_text += next_char\n", " generated_text = generated_text[1:]\n", "\n", " sys.stdout.write(next_char)\n", " sys.stdout.flush()\n", " print()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": { "height": "calc(100% - 180px)", "left": "10px", "top": "150px", "width": "384px" }, "toc_section_display": true, "toc_window_display": true }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 2 }