Roipoly preventing updating of image in loop?

I have a while loop going frame by frame through a video with user input on each frame. I would like to select an ROI from the figure, then update the image and a plotted line to the next frame using "set". I found using imshow and plot again resulted in slow execution at high frame numbers (e.g. 1000), so I opted for "set".

The plotting is working correctly as you step through frames, but once the roipoly functionality is uncommented, the image and lines no longer appear or refresh. Here is a sample script showing my issue:

fakeVideo=rand(600,500,10);

frame=1;
f1=figure;
hold on
hImage=imshow(fakeVideo(:,:,frame));
xlim([0 600]);
ylim([0 500]);
title(num2str(frame))
hold on
randomLine=plot([1 200],randi([1 500],1,2),'LineWidth',3);

endloop=0;
while endloop==0
    
    w = waitforbuttonpress;
    if w==1
        
        key = get(gcf,'currentcharacter');
        switch key
            
             case 100 %100 is "d" key
                %advance to next frame
                frame=frame+1;
                
                %Select ROI
                BW=roipoly(fakeVideo(:,:,frame));
                drawnow
                
                %Display next frame and new line
                figure(f1)
                hold on
                title(num2str(frame))
                set(hImage,'CData',fakeVideo(:,:,frame));
                set(randomLine,'YData',randi([1 500],1,2))
                drawnow
                
              case 113 %113 is "q" key
                %quit the loop
                endloop=1;
                 
        end
    end
end