How to combine audio and video?

it will run.. but not play the video... please tell me what the problem and how to resolve that?

videoFReader = vision.VideoFileReader('video1.avi');
[AUDIO,Fs] = audioread('audio1.wav');
videoFWriter = vision.VideoFileWriter('newvideo.avi','AudioInputPort',true);
videoFrame = step(videoFReader);
step(videoFWriter,videoFrame,AUDIO);
release(videoFReader);
release(videoFWriter);

Answer 1

[Walter Roberson]

You are writing all of the audio for the first frame and only reading and writing one frame.

you need to loop reading one input frame and writing that frame out together with the audio that matches it. For example if your audio output rate were 44100 and your video output were 22 frames per second then you should emit roughly 2000 audio samples for each video frame .

You might want to buffer() your audio to make it easier to write the appropriate amount of audio per frame .

[keshav poojari]

videoFReader = vision.VideoFileReader('video1.avi');
[AUDIO,Fs] = audioread('audio1.wav');
videoFWriter = vision.VideoFileWriter('newvideo.avi','AudioInputPort',true);
while ~isDone(videoFReader)
   videoFrame = step(videoFReader);
   step(videoFWriter,videoFrame,AUDIO);
end
release(videoFReader);
release(videoFWriter);

Now it play full video without audio!!!

help me...

[Walter Roberson]

video_filename = 'video1.avi';
audio_filename = 'audio1.wav';
out_filename = 'newvideo.avi';
info = aviinfo(video_filename);
FR = info.FramesPerSecond;
videoFReader = vision.VideoFileReader(audio_filename);
[AUDIO,Fs] = audioread(filename);
SamplesPerFrame = floor(Fs/FR);
videoFWriter = vision.VideoFileWriter(out_filename, 'AudioInputPort', true, 'FrameRate', FR);
framenum = 0;
while ~isDone(videoFReader)
   videoFrame = step(videoFReader);
   this_audio = AUDIO(framenum*SamplesPerFrame + 1 : min(end, (framenum+1)*SamplesPerFrame), :);
   step(videoFWriter,videoFrame, this_audio);
   framenum = framenum + 1;
end
release(videoFReader);
release(videoFWriter);

Note: with this code it would be common for the last few samples of the audio to not be written. This is due to the fact that the audio sampling rate will not typically be an exact multiple of the video frame rate, but the code allocates a fixed number of samples per video frame. For example if the audio is 22100 samples per second and the video is 24 frames per second, that should be an average of 920.833333333333 audio samples per video frame. The code takes the floor, 920 samples per video frame. You can see that after 100 video frames (about 4 seconds) it would be about 83 audio samples behind. You would need to improve the algorithm for selecting out of AUDIO in order to get around this problem.

... ideally you should also take into account that video might be variable frame rate.

[keshav poojari]

I run above code but error!

Warning: AVIINFO will be removed in a future release. Use VIDEOREADER instead. 
> In aviinfo at 67
  In audvidmix2 at 4 
Error using vision.VideoFileWriter/step
Changing the size on input 2 is not allowed without first calling the release() method.

Error in audvidmix2 (line 14)
   step(videoFWriter,videoFrame, this_audio);

How can i solve this error!!!....

[Walter Roberson]

I suspect that the message is arising out of the final partial frame of audio due to the integer issues I highlighted above.

[Walter Roberson]

I just noticed that I named the wrong file for the video reader.

The below also avoids using aviinfo().

video_filename = 'video1.avi';
audio_filename = 'audio1.wav';
out_filename = 'newvideo.avi';
videoFReader = VideoReader(video_filename);
FR = videoFReader.FrameRate;
[AUDIO,Fs] = audioread(audio_filename);
SamplesPerFrame = floor(Fs/FR);
videoFWriter = vision.VideoFileWriter(out_filename, 'AudioInputPort', true, 'FrameRate', FR);
framenum = 0;
while hasFrame(videoFReader)
   videoFrame = readFrame(videoFReader);
   this_audio = AUDIO(framenum*SamplesPerFrame + 1 : min(end, (framenum+1)*SamplesPerFrame), :);
   step(videoFWriter,videoFrame, this_audio);
   framenum = framenum + 1;
end
delete(videoFReader);
release(videoFWriter);

[keshav poojari]

Sir.. Any other method to do this?

[Walter Roberson]

yes there are other ways. you could use system() to invoke vlc.exe to combine the files. you could find an appropriate java library and call it from MATLAB . you could develop aa complete h.264 package written in MATLAB that used fread and fwrite.

[Walter Roberson]

Corrected code:

video_filename = 'video1.avi';
audio_filename = 'audio1.wav';
out_filename = 'newvideo.avi';
videoFReader = VideoReader(video_filename);
FR = videoFReader.FrameRate;
[AUDIO,Fs] = audioread(audio_filename);
SamplesPerFrame = floor(Fs/FR);
videoFWriter = vision.VideoFileWriter(out_filename, 'AudioInputPort', true, 'FrameRate', FR);
framenum = 0;
while hasFrame(videoFReader)
   videoFrame = readFrame(videoFReader);
   this_audio = AUDIO(framenum*SamplesPerFrame + 1 : min(end, (framenum+1)*SamplesPerFrame), :);
   if size(this_audio,1) < SamplesPerFrame
       this_audio(SamplesPerFrame,:) = 0; %zero pad short frames
   end
   step(videoFWriter,videoFrame, this_audio);
   framenum = framenum + 1;
end
delete(videoFReader);
release(videoFWriter);

However, my experiments are having a bit of difficulty in playing the result on Mac. aviinfo() says there is incorrect CHUNK information. Apple Quicktime did not include the sound when playing it. But VLC had no problem.

I notice that vision.VideoFileReader() reads the audio in 'native' format -- int16 rather than the floating point that audioreader() automatically converts to.

[keshav poojari]

I used above code.. But some Error!! please solve this...

>> audvidmix
Undefined function 'hasFrame' for input arguments of type 'VideoReader'.

Error in audvidmix (line 10)
while hasFrame(videoFReader)

[Walter Roberson]

Which MATLAB release are you using? That function was introduced in R2014b.

When you are not using a fairly current release it is important to state your release somewhere so that people do not waste time suggesting solutions that cannot work for you.

[keshav poojari]

sir,, i using R2013a. In this release any solution???? please

[Walter Roberson]

Barely. Your original comment was that you were using R2010a: it is not possible in that version as there was no support for vision.VideoFileWriter or equivalent before R2012a.

video_filename = 'video1.avi';
audio_filename = 'audio1.wav';
out_filename = 'newvideo.avi';
videoFReader = VideoReader(video_filename);
FR = videoFReader.FrameRate;
[AUDIO,Fs] = audioread(audio_filename);
SamplesPerFrame = floor(Fs/FR);
videoFWriter = vision.VideoFileWriter(out_filename, 'AudioInputPort', true, 'FrameRate', FR);
videoframes = read(videoFReader);    %reads all frames, not recommended in current MATLAB
for framenum = 0 : size(videoframes, 4)-1
   videoFrame = videoframes(:,:,:,framenum+1)
   this_audio = AUDIO(framenum*SamplesPerFrame + 1 : min(end, (framenum+1)*SamplesPerFrame), :);
   if size(this_audio,1) < SamplesPerFrame
       this_audio(SamplesPerFrame,:) = 0; %zero pad short frames
   end
   step(videoFWriter, videoFrame, this_audio);
end
delete(videoFReader);
release(videoFWriter);

[keshav poojari]

THANKS A LOT sir... it worked

[keshav poojari]

When i used above code to combine audio and video, The output video take much Storage space... can i compress this??? How??

[Walter Roberson]

keshav poojari, you might need to post-process the video.

But have a look at the description of the AudioCompressor and VideoCompressor properties that you can set for vision.VideoFileWriter. The choices depend upon what you have installed on your system, so I cannot tell you what the choices are.