# Deep Learning with Keras

- [[Perceptron]]
- [[Generative Adversarial Networks and WaveNet]]
- [[Word Embeddings]]
- [[Recurrent Neural Network — RNN]]

## Keras API

Keras has a modular, minimalist, and easy extendable architecture. Francois Chollet, the author of Keras, says:

The library was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.

Keras defineds high-level neural networks running on top of either TensorFlow or Theano

- Modularity: A model is either a sequence or a graph of standalone modules that can be combined together like LEGO blocks for building neural networks. Namely, the library predefines a very large number of modules implementing different types of neural layers, cost functions, optimizers, initialization schemes, activation functions, and regularization schemes.

- Minimalism: The library is implemented in Python and each module is kept short and self-describing.

- Easy extensiblity: The library can be extended with new functionalities, as we will describe in Chapter 7, Additional Deep Learning Models.

## What is tensor?

Keras uses either Theano or TensorFlow to perform very efficient computations on tensors. But what is a tensor anyway? A tensor is nothing but a multidimensional array or matrix. Both the backends are capable of efficient symbolic computations on tensors, which are the fundamental building blocks for createing neural networks.

## Composing models in Keras

There are two ways to composing models in Keras. They are as follows:

- Sequential composition
- Functional composition

## Regular dense

```python
keras.layers.core.Dense(units, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)
```

## Recurrent neural networks - simple, LSTM, and GRU

```python
keras.layers.recurrent.Recurrent(return_sequences=False, go_backwards=False, stateful=False, unroll=False, implementation=0)

keras.layers.recurrent.SimpleRNN(units, activation='tanh', use_bias=True, kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', bias_initializer='zeros', kernel_regularizer=None, recurrent_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, recurrent_constraint=None, bias_constraint=None, dropout=0.0, recurrent_dropout=0.0)

keras.layers.recurrent.GRU(units, activation='tanh', recurrent_activation='hard_sigmoid', use_bias=True, kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', bias_initializer='zeros', kernel_regularizer=None, recurrent_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, recurrent_constraint=None, bias_constraint=None, dropout=0.0, recurrent_dropout=0.0)

keras.layers.recurrent.LSTM(units, activation='tanh', recurrent_activation='hard_sigmoid', use_bias=True, kernel_initializer='glorot_uniform', recurrent_initializer='orthogonal', bias_initializer='zeros', unit_forget_bias=True, kernel_regularizer=None, recurrent_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, recurrent_constraint=None, bias_constraint=None, dropout=0.0, recurrent_dropout=0.0)
```

## Convolutional and pooling layers

```python
keras.layers.convolutional.Conv1D(filters, kernel_size, strides=1, padding='valid', dilation_rate=1, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)

keras.layers.convolutional.Conv2D(filters, kernel_size, strides=(1, 1), padding='valid', data_format=None, dilation_rate=(1, 1), activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)

keras.layers.pooling.MaxPooling1D(pool_size=2, strides=None, padding='valid')

keras.layers.pooling.MaxPooling2D(pool_size=(2, 2), strides=None, padding='valid', data_format=None)
```

## Regularization

- kernel_regularizer: Regularizer function applied to the weight matrix
- bias_regularizer: Regularizer function applied to the bias vector
- activity_regularizer: Regularizer function applied to the output of the layer (its activation)

In addition is possible to use Dropout for regularization and that is frequently a very effective choice

```python
keras.layers.core.Dropout(rate, noise_shape=None, seed=None)
```

Where:
- rate: It is a float between 0 and 1 which represents the fraction of the input units to drop
- noise_shape: It is a 1D integer tensor which represents the shape of the binary dropout mask that will be multiplied with the input
- seed: It is a integer which is used use as random seed

## Batch normalization

```python
keras.layers.normalization.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001, center=True, scale=True, beta_initializer='zeros', gamma_initializer='ones', moving_mean_initializer='zeros', moving_variance_initializer='ones', beta_regularizer=None, gamma_regularizer=None, beta_constraint=None, gamma_constraint=None)
```

## An overview of optimizers

Optimizers include SGD, RMSprop, and Adam.

## Using TensorBoard and Keras

Keras provides a callback for saving your training and test metrics, as well as activation histograms for the diffrerent layers in your model:

```python
keras.callbacks.TensorBoard(log_dir='./logs', histogram_freq=0,  
write_graph=True, write_images=False)
```

Saved data can then be visualized with TensorBoard launched at the command line:

```shell
tensorboard --logdir=/full_path_to_your_logs
```

## Using Quiver and Keras

```python
pip install quiver_engine 

from quiver_engine import server     server.launch(model)
```

![](https://www.safaribooksonline.com/library/view/deep-learning-with/9781787128422/assets/image_02_028.png)

## Pooling layers

### Max-pooling

```python
model.add(MaxPooling2D(pool_size = (2, 2)))
```

### Average pooling

## LeNet code in Keras

To define LeNet code, we use a convolutional 2D module, which is:

```python
keras.layers.convolutional.Conv2D(filters, kernel_size, padding='valid')
```

Here, `filters` is the number of convolution kernels to use (for exampel, the dimensionality of the output), `kernel_size` is an integer or tuple/list of two integers, specifying the width the same value for all spatial dimensions), and `padding='same'` means that padding is used. There are two options: `padding='valid'` means that the convolution is only computed where the input and the filter fully overlap, and therefore the output is smaller than the input, while `padding='same'` means that we have an output that the same size as the input, for which the area around the input is padded with zeros.

In addition, we use a `MaxPooling2D` module:

```python
keras.layers.pooling.MaxPooling2D(pool_size=(2,2), strides=(2,2))
```

Here, `pool_size=(2,2)` is a tuple of two integers representing the factors by which the image is vertically and horizontally downsacled. So (2,2) will halve the image in each dimension, and `strides=(2,2)` is the stride used for processing.

Now, let us review the code. First we import a number of modules:

```python
from keras import backend as K
from keras.models import Sequential
from keras.layers.convolutional import Conv2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers.core import Activation
from keras.layers.core import Flatten
from keras.layers.core import Dense
from keras.datasets import mnist
from keras.utils import np_utils
from keras.optimizers import SGD, RMSprop, Adam
import numpy as np
import matplotlib.pyplot as plt
```

Then we define the LeNet network:

```python
#define the ConvNet
class LeNet:
    @staticmethod
    def build(input_shape, classes):
         model = Sequential()
         # CONV => RELU => POOL	
```

```python
model.add(Convolution2D(20, kernel_size=5, padding="same",
input_shape=input_shape))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
# CONV => RELU => POOL

model.add(Conv2D(50, kernel_size=5, border_mode="same"))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

# Flatten => RELU layers
model.add(Flatten())
model.add(Dense(500))
model.add(Activation("relu"))
# a softmax classifier
model.add(Dense(classes))
model.add(Activation("softmax"))
return model
```