clamp cross-entropy loss

the Matlab cross-entropy loss has this form:

loss = -sum(W*(T.*log(Y)))/N;

I would like to "clamp" it so that the log function output is bounded, for example it cannot be less than 100.

See here: https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html

Can we do it?

Answer 1

[David Goodmanson]

Hi Matt,

z = log(Y);
z(z<100) = 100;
loss = -sum(W*(T.*z))/N;

In the link you provided, they talk about a limit of -100 rather than +100. The former appears to make more sense. Lots of possibilities for a smooth differentiable cutoff, here is one, assuming Y>=0

Ylimit = -100;
loss = -sum(W*(T.*log(Y+exp(Ylimit)))/N;

[Matt Fetterman]

That does seem reasonable but we have to remember that this is machine learning loss function and it has to be differentiable. So I am not sure if that answer will work.

[David Goodmanson]

Hi Matt,

see amended answer.

[Matt Fetterman]

probably a smart approach.