Is it ever the case that a system cannot go below a certain loss value? Are there training datasets for which values cannot be satisfactorily assigned?
randomthread
This depends on whether the solution is realizable. The network model could be too constrained to model the input. We could also potentially have two inputs that are the same but give a different output (giving a non-zero lower bound on the loss), but this would be a poor training set. It is also significant to note that we want good testing performance so perfect performance on the training set could mean we are overfitting leading to worse overall performance on test data.
Is it ever the case that a system cannot go below a certain loss value? Are there training datasets for which values cannot be satisfactorily assigned?
This depends on whether the solution is realizable. The network model could be too constrained to model the input. We could also potentially have two inputs that are the same but give a different output (giving a non-zero lower bound on the loss), but this would be a poor training set. It is also significant to note that we want good testing performance so perfect performance on the training set could mean we are overfitting leading to worse overall performance on test data.