Discarding a value from a set of values

Hello,

I am trying to get time difference between the consecutive peaks found in a every 1 second in a signal.But sometimes some of the intervals in a total second intervals are very large (comapred to the other intervals found)as shown in the image attached.

I tried median filter but it is not what I need.

I want to discard the intervals that are differing form the other interval found in a second duration.

Can someone suggest a ways(method) to discard the values that are more than the set of all values found in 1 second interval. Thanks.