Sample Code to use Keras Model

https://keras.io/api/applications/

Remove Sensitive Environment Variable / File That is too Big from Remote Repo

In case mistakenly pushed a large file/sensitive environment data and git rm --cached does not help (pushed to the remote repo but just untracked locally):

git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch <path to your file>" \
  --prune-empty --tag-name-filter cat -- --all

(cd to the top level of the repo first), add -r flag if you want to remove the whole directory. REF: 

https://docs.github.com/.../removing-sensitive-data-from... 

This error may subsequently follow: fatal: refusing to merge unrelated histories Then

git pull origin the-remote-branch --allow-unrelated-histories

and resolve conflicts.

Examine Output Size in Tensorflow

When we are uncertain the output size of our tensor processed by some layer, we can go through the following:

x = tf.constant([[1, 1., 1., 2., 3.],
                 [1, 1., 4., 5., 6.],
                 [1, 1., 7., 8., 9.],
                 [1, 1., 7., 8., 9.],
                 [1, 1., 7., 8., 9.]])

x = tf.reshape(x, [1, 5, 5, 1])

print(MaxPool2D((5, 5), strides=(2, 2),  padding="same")(x))
print(math.ceil(5/2))

which yields

print(MaxPool2D((5, 5), strides=(2, 2),  padding="same")(x))
tf.Tensor(
[[[[7.]
   [9.]
   [9.]]

  [[7.]
   [9.]
   [9.]]

  [[7.]
   [9.]
   [9.]]]], shape=(1, 3, 3, 1), dtype=float32)

3

For layer that has training weight, we may try the following for testing:

model = Conv2D(3, (3, 3), strides=(2, 2), padding="same", kernel_initializer=tf.constant_initializer(1.))


x = tf.constant([[1., 2., 3., 4., 5.],
                 [1., 2., 3., 4., 5.],
                 [1., 2., 3., 4., 5.],
                 [1., 2., 3., 4., 5.],
                 [1., 2., 3., 4., 5.]])

x = tf.reshape(x, (1, 5, 5, 1))
print(model(x))

which yields

x = tf.constant([[1., 2., 3., 4., 5.],...
tf.Tensor(
[[[[ 6.  6.  6.]
   [18. 18. 18.]
   [18. 18. 18.]]

  [[ 9.  9.  9.]
   [27. 27. 27.]
   [27. 27. 27.]]

  [[ 6.  6.  6.]
   [18. 18. 18.]
   [18. 18. 18.]]]], shape=(1, 3, 3, 3), dtype=float32)

In fact it can be proved in both MaxPooling2D and Conv2D that if stride $=s$ and padding$=$same, then 

\[\text{output_width} = \left\lfloor\frac{\text{input_width}-1}{s}\right\rfloor + 1 = \left\lceil\frac{\text{input_width}}{s}\right\rceil\]

The last equality deserves a proof as it is not highly trivial:

Fact. For any positive intergers $w,s$, we have \[ \left\lfloor \frac{w-1}{s}\right\rfloor + 1 = \left\lceil \frac{w}{s}\right\rceil. \] Proof. We do case by case study. If $w=ks$ for some positive $k\in \N$, then \[\text{LHS} = \left\lfloor k - \frac{1}{s}\right\rfloor +1 = (k-1)+1=k = \lceil k\rceil = \text{RHS}. \] When $w=ks+j$, for some $k\in\N$ and $j\in \N \cap (0, s)$, then \[ \text{LHS} = \left\lfloor k+\frac{j-1}{s}\right\rfloor + 1 = k+1 = \left\lceil k+\frac{j}{s}\right\rceil = \left\lceil \frac{ks+j}{s}\right\rceil = \left\lceil\frac{w}{s}\right\rceil=\text{RHS}.\qed \]

conda virtual environment command

conda create --name tensorflow python=3.7
conda env remove --name tensorflow

conda env export --name ENVNAME > envname.yml
conda env create --file envname.yml

Record model.compile options

Suppose we have a model of two hidden layers as follows:

model = tf.keras.models.Sequential([tf.keras.layers.Flatten(), 
                                    tf.keras.layers.Dense(128, activation=tf.nn.relu), 
                                    tf.keras.layers.Dense(10, activation=tf.nn.softmax)])

Then our model.compile might have the following as arguments:

model.compile(optimizer = tf.optimizers.Adam(),
              loss = 'sparse_categorical_crossentropy',
              metrics=['accuracy'])

With a callback that stop training at desired loss:

import tensorflow as tf
print(tf.__version__)

class myCallback(tf.keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs={}):
    if(logs.get('loss')<0.4):
      print("\nReached 60% accuracy so cancelling training!")
      self.model.stop_training = True

callbacks = myCallback()
mnist = tf.keras.datasets.fashion_mnist
(training_images, training_labels), (test_images, test_labels) = mnist.load_data()
training_images=training_images/255.0
test_images=test_images/255.0
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(512, activation=tf.nn.relu),
  tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
model.fit(training_images, training_labels, epochs=5, callbacks=[callbacks])

ResNet

def identity_block(X, f, filters, stage, block):
    """
    Implementation of the identity block as defined in Figure 3
    
    Arguments:
    X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)
    f -- integer, specifying the shape of the middle CONV's window for the main path
    filters -- python list of integers, defining the number of filters in the CONV layers of the main path
    stage -- integer, used to name the layers, depending on their position in the network
    block -- string/character, used to name the layers, depending on their position in the network
    
    Returns:
    X -- output of the identity block, tensor of shape (n_H, n_W, n_C)
    """
    
    # defining name basis
    conv_name_base = 'res' + str(stage) + block + '_branch'    bn_name_base = 'bn' + str(stage) + block + '_branch'
    
    # Retrieve Filters
    F1, F2, F3 = filters
    
    # Save the input value. You'll need this later to add back to the main path. 
    X_shortcut = X
    
    # First component of main path
    X = Conv2D(filters = F1, kernel_size = (1, 1), strides = (1,1), padding = 'valid', name = conv_name_base + '2a', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2a')(X)
    X = Activation('relu')(X)
    
    ### START CODE HERE ###
    
    # Second component of main path (≈3 lines)
    X = Conv2D(filters = F2, kernel_size = (f,f), strides = (1,1), padding = 'same', name = conv_name_base + '2b', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis=3, name=bn_name_base+'2b')(X)
    X = Activation('relu')(X)
    
    # Third component of main path (≈2 lines)
    X = Conv2D(filters = F3, kernel_size = (1,1), strides = (1,1), padding = 'valid', name = conv_name_base+'2c', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis=3, name=bn_name_base+'2c')(X)

    # Final step: Add shortcut value to main path, and pass it through a RELU activation (≈2 lines)
    X = Add()([X, X_shortcut])
    X = Activation('relu')(X)
    
    ### END CODE HERE ###

def convolutional_block(X, f, filters, stage, block, s = 2):
    """
    Implementation of the convolutional block as defined in Figure 4
    
    Arguments:
    X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)
    f -- integer, specifying the shape of the middle CONV's window for the main path
    filters -- python list of integers, defining the number of filters in the CONV layers of the main path
    stage -- integer, used to name the layers, depending on their position in the network
    block -- string/character, used to name the layers, depending on their position in the network
    s -- Integer, specifying the stride to be used
    
    Returns:
    X -- output of the convolutional block, tensor of shape (n_H, n_W, n_C)
    """
    
    # defining name basis
    conv_name_base = 'res' + str(stage) + block + '_branch'
    bn_name_base = 'bn' + str(stage) + block + '_branch'
    
    # Retrieve Filters
    F1, F2, F3 = filters
    
    # Save the input value
    X_shortcut = X


    ##### MAIN PATH #####
    # First component of main path 
    X = Conv2D(F1, (1, 1), strides = (s,s), name = conv_name_base + '2a', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2a')(X)
    X = Activation('relu')(X)
    
    ### START CODE HERE ###

    # Second component of main path (≈3 lines)
    X = Conv2D(F2, (f, f), strides = (1,1), padding="same", name = conv_name_base + '2b', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2b')(X)
    X = Activation('relu')(X)

    # Third component of main path (≈2 lines)
    X = Conv2D(F3, (1, 1), strides = (1,1), padding="valid", name = conv_name_base + '2c', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = bn_name_base + '2c')(X)

    ##### SHORTCUT PATH #### (≈2 lines)
    X_shortcut = Conv2D(F3, (1, 1), strides = (s,s), padding="valid", name = conv_name_base + '1', kernel_initializer = glorot_uniform(seed=0))(X_shortcut)
    X_shortcut = BatchNormalization(axis = 3, name = bn_name_base + '1')(X_shortcut)
      
    # Final step: Add shortcut value to main path, and pass it through a RELU activation (≈2 lines)
    X = Add()([X,  X_shortcut])
    X = Activation("relu")(X)
    
    ### END CODE HERE ###
    
    return X

def ResNet50(input_shape = (64, 64, 3), classes = 6):
    """
    Implementation of the popular ResNet50 the following architecture:
    CONV2D -> BATCHNORM -> RELU -> MAXPOOL -> CONVBLOCK -> IDBLOCK*2 -> CONVBLOCK -> IDBLOCK*3
    -> CONVBLOCK -> IDBLOCK*5 -> CONVBLOCK -> IDBLOCK*2 -> AVGPOOL -> TOPLAYER

    Arguments:
    input_shape -- shape of the images of the dataset
    classes -- integer, number of classes

    Returns:
    model -- a Model() instance in Keras
    """
    
    # Define the input as a tensor with shape input_shape
    X_input = Input(input_shape)

    
    # Zero-Padding
    X = ZeroPadding2D((3, 3))(X_input)
    
    # Stage 1
    X = Conv2D(64, (7, 7), strides = (2, 2), name = 'conv1', kernel_initializer = glorot_uniform(seed=0))(X)
    X = BatchNormalization(axis = 3, name = 'bn_conv1')(X)
    X = Activation('relu')(X)
    X = MaxPooling2D((3, 3), strides=(2, 2))(X)

    # Stage 2
    X = convolutional_block(X, f = 3, filters = [64, 64, 256], stage = 2, block='a', s = 1)
    X = identity_block(X, 3, [64, 64, 256], stage=2, block='b')
    X = identity_block(X, 3, [64, 64, 256], stage=2, block='c')

    ### START CODE HERE ###

    # Stage 3 (≈4 lines)
    X = convolutional_block(X, f = 3, filters=[128, 128, 512], stage = 3, block="a", s = 2)
    X = identity_block(X, 3, filters=[128,128,512], stage=3, block="b")
    X = identity_block(X, 3, filters=[128,128,512], stage=3, block="c")
    X = identity_block(X, 3, filters=[128,128,512], stage=3, block="d")

    # Stage 4 (≈6 lines)
    X = convolutional_block(X, f = 3, filters=[256, 256, 1024], stage = 4, block="a", s = 2)
    X = identity_block(X, 3, filters=[256, 256, 1024], stage=4, block="b")
    X = identity_block(X, 3, filters=[256, 256, 1024], stage=4, block="c")
    X = identity_block(X, 3, filters=[256, 256, 1024], stage=4, block="d")
    X = identity_block(X, 3, filters=[256, 256, 1024], stage=4, block="e")
    X = identity_block(X, 3, filters=[256, 256, 1024], stage=4, block="f")

    # Stage 5 (≈3 lines)
    X = convolutional_block(X, f = 3, filters=[512, 512, 2048], stage = 5, block="a", s = 2)
    X = identity_block(X, f=3, filters=[512, 512, 2048], stage=5, block="b")
    X = identity_block(X, f=3, filters=[512, 512, 2048], stage=5, block="c")

    # AVGPOOL (≈1 line). Use "X = AveragePooling2D(...)(X)"
    X = AveragePooling2D(pool_size=(2, 2), name="avg_pool")(X)
    
    ### END CODE HERE ###

    # output layer
    X = Flatten()(X)
    X = Dense(classes, activation='softmax', name='fc' + str(classes), kernel_initializer = glorot_uniform(seed=0))(X)
    
    
    # Create model
    model = Model(inputs = X_input, outputs = X, name='ResNet50')

    return model

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

X_train_orig, Y_train_orig, X_test_orig, Y_test_orig, classes = load_dataset()

# Normalize image vectors
X_train = X_train_orig/255.
X_test = X_test_orig/255.

# Convert training and test labels to one hot matrices
Y_train = convert_to_one_hot(Y_train_orig, 6).T
Y_test = convert_to_one_hot(Y_test_orig, 6).T

model.fit(X_train, Y_train, epochs = 2, batch_size = 32)

preds = model.evaluate(X_test, Y_test)
print ("Loss = " + str(preds[0]))
print ("Test Accuracy = " + str(preds[1]))

Code Assignment

def HappyModel(input_shape):
    X_input = Input(input_shape)
    
    X = ZeroPadding2D((3,3))(X_input)
    X = Conv2D(18,(7,7),strides=(1,1),name="conv0")(X)
    X = BatchNormalization(axis=3, name="bn0")(X)
    X = Activation("relu")(X)
    
    X = MaxPooling2D((2,2), name="max_pool")(X)
    
    X = Flatten()(X)
    X = Dense(1,activation="sigmoid", name="fC")(X)
    
    model = Model(input = X_input, outputs = X, name="happy model")
	 return model

happyModel = HappyModel(X_train.shape[1:])
     
happyModel.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
    
happyModel.fit(x=X_train, y=Y_train,epochs=10, batch_size=20)
    
preds = happyModel.evaluate(x=X_test,y=Y_test)

img_path = 'images/smile.jpg'
  
img = image.load_img(img_path, target_size=(64, 64))
imshow(img)

x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
print(happyModel.predict(x))

Derive the Formula of $\displaystyle \frac{\partial \mathcal L}{\partial W^{[\ell]}}$

Wikipedia record the following formula without proof:

I accidentally found that by the formulas in the previous post, we can already derive the following

Theorem. For every $\ell<L-1$, we let $\Phi^{[\ell]}:\R\to \R$ denote the activation function in the hidden layer, then we have\[ \frac{\partial \mathcal L }{\partial W^{[\ell]}}=\underbrace{\frac{1}{m}\Phi^{[\ell]}{}'(U^{[\ell]}) * \left[\prod_{i=\ell +1}^{L-1} (\Phi^{[i]}{}'(U^{[i]}) * W^{[i]T}\right]\cdot \frac{\partial \mathcal L}{\partial Y^{[L-1]}} }_{:=\delta_\ell} \cdot Y^{[\ell-1]T} = \delta_{\ell}\cdot Y^{[\ell-1]T}.\] Here $*$ denotes the entrywise multiplication. Since $\displaystyle \frac{\partial \mathcal L}{\partial W^{[L]}}=\frac{1}{m}\cdot \frac{\partial \mathcal L}{\partial U^{[L]}}\cdot Y^{[L-1]T}$, we also define \[ \boxed{\delta_L = \frac{1}{m}\cdot \frac{\partial \mathcal L}{\partial U^{[L]}}}    \]and since \[\frac{\partial \mathcal L}{\partial W^{[L-1]}} =\frac{1}{m}\Phi^{[L-1]}{}'(U^{[L-1]})* \left( W^{[L]T} \cdot \frac{\partial \mathcal L}{\partial U^{[L]}}\right) Y^{[L-2]T}=\delta_{L-1} Y^{[L-2]T} \] with $\delta_{L-1} :=\frac{1}{m}\cdot \left( \Phi^{[L-1]}{}'(U^{[L-1]}) * W^{[L]T}\right)\cdot \frac{\partial \mathcal L}{\partial U^{[L]}}$, by the definition of $\delta_\ell$ for $\ell<L-1$ above, we obtain for every $\ell\leq L-1$, \[\boxed{ \delta_{\ell} = \frac{1}{m}\cdot \Phi^{[\ell]}{}'(U^{[\ell]}) * \left[W^{[\ell+1]T} \cdot \delta_{\ell+1}\right]\quad \text{with}\quad \frac{\partial \mathcal L}{\partial W^{[\ell]}} = \delta_\ell Y^{[\ell-1]T}.} \] And as a side consequence of our computation, since $\displaystyle\frac{1}{m}\cdot \frac{\partial \mathcal L}{\partial U^{[\ell]}} = \delta_\ell$, \[ \boxed{\frac{\partial \mathcal L}{\partial b^{[\ell]}} = \text{np.sum}(\delta_\ell,\text{axis=1}).} \]

The last two formulars are computationally very useful. Note that in the definition of $\delta_\ell$, the multiplication in the product notation will not make sense unless they act on the rightmost matrix $\displaystyle \frac{\partial \mathcal L}{\partial Y^{[L-1]}} $ in a correct order (from the biggest index). To simplify notations we follow Andrew Ng's course to define $dW = \partial \mathcal L /\partial W$ and similarly for other matrices.

Proof. By repeated use of the formular $dY^{[\ell]} = [W^{[\ell+1]T}dY^{[\ell+1]}] * \Phi^{[\ell+1]}(U^{[\ell+1]})$ we have \[\begin{align*} dW^{[\ell]}& = \frac{1}{m} dU^{[\ell]} Y^{[\ell-1]T}\\ &=\frac{1}{m}\left(\left[dY^{[\ell]}\right] * \Phi^{[\ell]}{}'(U^{[\ell]})\right) Y^{[\ell-1]T}\\ &=\frac{1}{m}\left( \Phi^{[\ell]'}(U^{[\ell]})* \left[\prod_{i=\ell+1}^{L-1} \Phi^{[i]}{}'(U^{[i]}) * W^{[i]T}\right]\cdot dY^{[L-1]}\right) \cdot Y^{[\ell-1]T} \end{align*} \] And recall that $dY^{[L]} =\displaystyle \frac{\partial \mathcal L}{\partial Y^{[L]}}. \qed$

Formulas Revisit

When I study Andrew Ng course I am used to formulas like $dZ^{[\ell]}, dA$ , etc notations. Recently I revisit the topic and I am then used to using the notation \[u_i^{[\ell]} = W^{[\ell]}_{i:}\cdot y^{[\ell-1]} + b^{[\ell]}\quad\text{and}\quad y^{[\ell]} = \Phi^{[\ell]}(u),\] so I want to record the corresponding formulas for computation. Since the notation $dW$ doesn't look any cleaner than $\displaystyle\frac{\partial \mathcal L}{\partial W}$, in the sequel I write everything explicitly. \[ \boxed{\frac{\partial \mathcal L}{\partial W^{[\ell]}} = \frac{1}{m}\cdot \frac{\partial \mathcal L }{\partial U^{[\ell]}}\cdot Y^{[\ell-1]T}} \] \[ \boxed{\frac{\partial \mathcal L}{\partial U^{[\ell]}} = \brac{ W^{[\ell +1]T} \cdot \frac{\partial \mathcal L}{\partial U^{[\ell+1]}} }* \Phi^{[\ell]}{}'(U^{[\ell]})} \] where $*$ denotes entrywise product of matrices. \[ \boxed{\frac{\partial \mathcal L}{ \partial Y^{[\ell - 1]}} = W^{[\ell]T}\cdot \frac{\partial \mathcal L}{\partial U^{[\ell]}}} \] The last two yield the following for $\ell<L$ the hidden layer and for $\Phi^{[\ell]}:\R\to \R$ the activation function at $\ell$-th layer. \[ \boxed{\frac{\partial \mathcal L}{\partial U^{[\ell]}} = \frac{\partial \mathcal L}{\partial Y^{[\ell]}} * \Phi^{[\ell]}{}'(U^{[\ell]})} \] and finally \[ \boxed{\frac{\partial \mathcal L}{\partial b^{[\ell]} } =\frac{1}{m}\cdot \sum_{i=1}^m \frac{\partial \mathcal L}{\partial u^{[\ell](i)}} =\frac{1}{m}\cdot \text{np.sum}\brac{\frac{\partial \mathcal L}{\partial U^{[\ell]}},\text{axis = 1}}} \] For derivation of these formulas one can visit my another post: https://checkerlee.blogspot.com/2019/11/important-formulas-in-backward.html#more

Intutive derivation of Cross Entropy as a "loss" function

In defining "loss" function for classification problems given $p_i=\mathbb P\{\text{$i$ occurs}\}$, $i=1,2,\dots,n$,  from emperical data, we measure the accuracy of estimated data (from our output layer in neuron network) $[q_1,q_2,\dots,q_n]$ by the cross-entropy: \[L=\sum_{i=1}^n p_i\ln q_i.\] Recently I revisit this topic, and understand that this comes very naturally from solving maximum-likelihood estimation problem!

        Let's take an example, consider flipping a coin with getting a head with probability $p$ and tail with $1-p$, then the probability of getting 2 heads out of 6 flipping is \[ L(p) = \binom{6}{2} p^2 (1-p)^4 = 15 p^2(1-p)^4. \] Maximum-likelihood estimation ask the following problem:

The phenomenon of getting 2 heads is most likely to happen under what value of $p$?

In other words, the above question is the same as at what value of $p$ the proability $L(p)$ gets maximized? By simply solving $L'(p)=0$ we get the answer $p=\frac{1}{3}$.

        But in more complex problem we could not have the probability of some phenomenon to occurs based on another probablity with explicit formula. Instead of computing the probability $p$ directly, we try to estimate it such that our observation (the phenomenon from empirical data) is most likely to occur, and such an estimated value $p$ is considered as a good estimation.

        Now the derivation of cross-entropy will be very intuitive: Assume that \[ \text{mutally disjoint }E_i=\{\text{$i$ occurs}\},\quad \mathbb P(E_i) = p_i, \quad i=1,2,3,\dots,n. \] And assume further that $E_i$'s are iid events. Consider events $A_1,\dots,A_N$ are such that $A_i = \cupp_{i=1}^n E_i$ for each $i$ (for example, flipping coins $N$ times), then $p_i = N_i/N$, where $N_i$ is the number of times $i$ occures among $A_1,\dots,A_N$.

        Now we get another estimation $q_i$ of the same event $E_i$ from what ever experiment we can imagine. How good is $[q_1,\dots,q_n]$ as an estimation to the past emperical data $[p_1,\dots,p_n]$? The standard distance in $\R^n$ is certainly not a good choice since an quantity $\epsilon$ from $q_i$ to $p_i$ can mean huge difference from $q_{i'}$ to $p_{i'}$. $[q_1,\dots,q_n]$ is considered as good estimation if the observed phenomenon \[ \{\text{1 appears $N_1$ times}\}, \quad \{\text{2 appears $N_2$ times}\},\quad \dots ,\quad \{\text{n appears $N_n$ times} \} \] is very likely to happen under the estimates $[q_1,\dots,q_n]$, i.e., when \[ L = \prod_{i=1}^N q_i^{N_i}\iff \frac{\ln L}{N}= \sum_{i=1}^N \frac{N_i}{N}\ln q_i = \sum_{i=1}^n p_i\ln q_i. \] is large, and we have derived the cross-entropy at this point.

babel-node template

npm install babel-cli babel-preset-env --save-dev

{
  "presets": [
    "env"
  ]
}

 nodemon src/index.js --exec babel-node

Completely uninstall apache2 to get fresh config

apt-get remove --purge apache2 apache2-data apache2-utils

Powershell command to debug ios in chrome

remotedebug_ios_webkit_adapter --port=9000

Open safari, browse to the page that is going to be inspected. Then in chrome go to chrome://inspect/#devices and choose the device.

Copy all lastly updated files into a single directory if the version control is horribly not done by git:

Instantiate a git repository (don't need to connect to any remote branch, do everything locally). Create a .sh file at the current directory, and paste:

#!/bin/bash

git add .
git status

updatedFiles=$(git status | awk '{print $2}' | grep -P "\..*$")

touch updates/update.txt
git status > updates/update.txt

for file in $updatedFiles
do
    cp --parents "$file" ./updates
    echo "copied $file to ./updates/$file"
done;

read -p "Press enter to exit"

mkdir updates and then run the bash script above. Files will be copied into updates directory, and we can manage it by date.

如果用了 spring-boot-devtools 的話，謹記在開啟 SpringApplication 前:

public class Main {
  public static void main(String[] args) {
    System.setProperty("spring.devtools.restart.enabled", "false");
    SpringApplication.run(Main.class, args);
  }
}

不然不知為甚麼它有 restart 機制，restart 前後的同一個 class 將不視為同一個 class，database transaction 將發生錯誤。

Hilbernate Database Configuration without XML

package com.springboot.mvc;

import java.util.Properties;
import com.springboot.mvc.models.Customer;
import org.hibernate.SessionFactory;
import org.hibernate.boot.registry.StandardServiceRegistryBuilder;
import org.hibernate.cfg.Configuration;
import org.hibernate.cfg.Environment;
import org.hibernate.service.ServiceRegistry;

public class HibernateUtil {
  private static SessionFactory sessionFactory;

  public static SessionFactory getSessionFactory() {
    if (sessionFactory == null) {
      try {
        Configuration configuration = new Configuration();
        // Hibernate settings equivalent to hibernate.cfg.xml's properties
        Properties settings = new Properties();
        settings.put(Environment.DRIVER, "com.mysql.cj.jdbc.Driver");
        settings.put(Environment.URL,
            "jdbc:mysql://192.168.99.100:3306/JDBC_spring_mvc_tutorial?useSSL=false&serverTimezone=UTC");
        settings.put(Environment.USER, "cclee");
        settings.put(Environment.PASS, "ccleedb12345");
        settings.put(Environment.DIALECT, "org.hibernate.dialect.MySQL55Dialect");
        settings.put(Environment.SHOW_SQL, "true");
        settings.put(Environment.CURRENT_SESSION_CONTEXT_CLASS, "thread");
        settings.put(Environment.HBM2DDL_AUTO, "create-drop");

        configuration.setProperties(settings);
        configuration.addAnnotatedClass(Customer.class);
       // we add more and more classes here.
       
       
        ServiceRegistry serviceRegistry = new StandardServiceRegistryBuilder()
            .applySettings(configuration.getProperties()).build();
        sessionFactory = configuration.buildSessionFactory(serviceRegistry);
      } catch (Exception e) {
        e.printStackTrace();
      }
    }
    // Then:
    // Session session = sessionFactory.openSession();
    // Transaction transaction = session.beginTransaction();

    return sessionFactory;
  }
}

Record for my Docker Files

FROM node:10

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install && npm rebuild bcrypt --build-from-source

EXPOSE 3000

CMD ["npm", "start"]

and

version: "3.7"

services:
  db:
    container_name: postgres_screencapdic_db
    image: postgres
    restart: always
    environment:
      POSTGRES_USER: cclee11111
      POSTGRES_PASSWORD: cclee11111
      POSTGRES_DB: screencapdb
    volumes:
      - screencapdb:/var/lib/postgresql/data
    ports:
      - "5432:5432"
  # screencap_api:
  #   build:
  #     context: ./
  #     dockerfile: Dockerfile-screepcap-express
  #   container_name: screencap_api
  #   restart: always
  #   ports:
  #     - "8080:3000"
  #   volumes:
  #     - type: bind
  #       source: ./
  #       target: /usr/src/app
  #     - /usr/src/app/node_modules

volumes:
  screencapdb:

State Pattern

Trying to refactor my old project by using design pattern. It really becomes simple to implement new functionality now! What I need to do is just to create additional state. And my code get less coupled! My class mainVM know nothing about my translation service now!

https://github.com/machingclee/ScreenCapDictionaryNoteApp_refactor/tree/2020-06-23-refactor-translation-by-state-pattern/ScreenCapDictionaryNoteApp/ViewModel/Helpers/TranslationHelper

Bash Script

 for f in *\ *; do mv "$f" "${f// /_}"; done

This change all spaces in file name by a "_".

Use Sequelize Migration with ES6 Syntax

After yarn add sequelize-cli it is clear from the --help command how to generate a migration folder and migration file. The only trouble is to use them with ES6 syntax. From the official document:

https://sequelize.org/master/manual/migrations.html#using-babel

we add

yarn add babel-register

and add a .sequelizerc runtime config with

// .sequelizerc
require("babel-register");

const path = require('path');

module.exports = {
  'config': path.resolve('config', 'config.json'),
  'models-path': path.resolve('models'),
  'seeders-path': path.resolve('seeders'),
  'migrations-path': path.resolve('migrations')
}

We can copy the implementation of altering, creating, deleting table from official documentation:

https://sequelize.org/master/manual/query-interface.html

Official document also says that in migration file we can export async function up and async function down instead of returning a chain of promises (i.e., a promise)! For example it happens that I want to add a column for users to implement mobile push notification, then I need to add a column called push_notification_token, I can do the following in our migration file:

"use strict";
import { modelNames } from "../src/enums/modelNames";
import { Sequelize, DataTypes } from "sequelize";

module.exports = {
  async up(queryInterface, Sequelize) {
    await queryInterface.addColumn(modelNames.USER + "s", "push_notification_token", {
      type: DataTypes.STRING,
      allowNull: true
    });
  },

  async down(queryInterface, Sequelize) {
    await queryInterface.removeColumn(
      modelNames.USER + "s",
      "push_notification_token",
      {}
    );
  }
};

Now if you run the code, we encounter the following error

Loaded configuration file "config\config.js".
Using environment "development".
== 20200615141047-add-push-notification-token-to-users-table: migrating =======

ERROR: regeneratorRuntime is not defined

so we need the transform runtime plugin by babel,

yarn add babel-plugin-transform-runtime

and in our .babelrc add:

{
  "presets": ["env"],
  "plugins": [
    ["transform-runtime", {
      "regenerator": true
    }]
  ]
}

and we are done!