conv neural network - How should we compare two CNNs training time when we train them several time?

Question

I want to train a cnn on different data-sets and compare prediction accuracy and cnn training time. For each data-sets I need to train several time the CNN and take the one with best perfomance. My first question is why do we need to train several time a CNN ? Does the model after one training keep the weight or any information from previous training or every training is independant?

My second question is when we compare training time between CNN, should we also compare the number of training iterations we needed before obtaining the best model?

Below an example where I'm training 100 time a model(cnn) and print the best MSE :

for i in range(100):
   history = model.fit(training_data,label_data,
                    validation_split=0.2,
                    epochs= nb_epoch, verbose = False,
                    callbacks=callbacks)            
   pred= model.predict(test_data)

   if(min_MSE > mean_squared_error(label_test, pred)):
      model.save('XXX.h5')
      min_MSE = mean_squared_error(label_test, pred)
    
print("The best: ",min_MSE)

Thanks in advance for any clarification !!

question from:https://stackoverflow.com/questions/66064676/how-should-we-compare-two-cnns-training-time-when-we-train-them-several-time

Answer 1

For question 1:

Fran?ois Chollet (Author of Keras) mentions here, that calling model.fit multiple times will incrementally train the model.

You may also want to check this: What is the trade-off between batch size and number of iterations to train a neural network?

For question 2:

I believe the number of iterations should not be taken into account as every time the weights are randomly initialized.