differentiable approximation of bandwidth in ksdensity for optimization

I have an optimization loss term ||x* - y||^2 where x* corresponds to a quantile approximated via ksdensity (e.g., x* = ksdensity(xq, 0.9, 'Function', 'icdf')) and y is an optimization variable optimized by fmincon.

By profiling, I figured out that ksdensity internally calculates the bandwidth (smoothing parameter) using the median absolute deviation estimator which I verified below:

n = 100; d = 1;
xq = randn(n,1); % data
sigma = median(abs(xq - median(xq,1,'omitmissing')),1,'omitmissing') / 0.6745;
bw = sigma * (4/((d+2)*n))^(1/(d+4)) 

bw = 0.4088

[~,~,bw_ref] = ksdensity(xq)

bw_ref = 0.4088

The sigma calculation involves abs(...) and median(...) -- both not continously differentiable and analytical gradients can not be derived.

A typical distribution of my xq's is shown in the attached file. Characteristic is that the the values have a positive lower bound and that the majority of the sampled values are at the lower bound and values close to the maximum are barely sampled.

My goal is to come up with a custom bandwidth calculation, appropriate for the shape of my data, which I can differentiate w.r.t. the xq's. Any suggestions what can be done here?

Answer 1

[Matt J]

ChatGPT offers the following:

[SA-W]

Would that be good in your opinion? Instead of 1:1 reproducing the matlab bandwidth with continuous diff equivalents, can you think of alternative solutions one could give a try?

[Matt J]

I think it's worth a try. The ksdensity is itself just an approximation, anyway.

[SA-W]

Makes sense. I am expecting around 1e6 elements in the xq vector. And to implement this softmedian(...)/softabs(...) strategy efficiently, I should first downsample the xq vector. Clearly, some sort of uniform sampling (e.g. downsample function) is quick and easy, but my data is far from being uniformly sampled (most of my samples are at the left extreme as shown in data.pdf).

Can you think of a better strategy to preserve the shape/trend of the histogram?