Simultaneous Video and Audio Acquisition

I'd like to record, simultaneously, from a USB video-camera (webcam) and a separate USB microphone. The synchronization between the two is important.

What specifically would I need to do this? I've found several posts, but it's not entirely clear.

I found this: http://www.mathworks.com/matlabcentral/newsreader/view_thread/140663 Which is an old post, so I'm not sure if it's still relevant. It links to this page (also old): http://www.mathworks.com/support/solutions/en/data/1-1MNV35/?solution=1-1MNV35, but this seems to apply to recording from a single device. I am unclear if the triggering is between the video card and the sound card, or if it is in reference to a recording device (such as a webcam) that has on-board video and audio, and that device is triggered.

Again, I'm not sure if there is an easier way to do this now that several years have passed since those posts. I'd like to know what type of hardware is needed, whatever the solution.

Answer 1

[Jiro Doke]

I assume you want something much better than issuing a "start" command to both analoginput and videoinput objects like this:

start(ai); start(vid);

I think the Solutions page you indicated is still valid, and it should be valid for different devices as well. You would use Data Acquisition Toolbox for the audio input and Image Acquisition Toolbox for the video input. You would need a video device that offer "hardware triggers" (as described here). I'm not sure if there are any webcams with hardware triggers, though. Also, I'm not sure a USB microphone can be externally triggered. Perhaps using a true data acquisition device where you can monitor a channel for external triggers would be the way to go.

[Valerie]

Thanks for the response! Sorry I'm still kind of grappling with this issue.

Do you know if doing the start command would add a significant latency between the audio and video? And would it be constant or random?

Buying an external data acquisition device may end up being the best solution, with latency issues in mind. Do you know of one that is compatible with Matlab, could receive both audio and video, and wouldn't break the bank?

[Jiro Doke]

It depends on what your acceptable latency is. It certainly is not constant (deterministic) since it's all running in an OS. If you already have a webcam and a microphone (and have Image Acquisition Toolbox and Data Acquisition Toolbox), then you should give it a try and see what the latency is like. I found this on the File Exchange:

http://www.mathworks.com/matlabcentral/fileexchange/18596

I haven't used it, but maybe you can use that to check your current set up.

I'm not familiar with hardware that does both audio and video. I'm sure there are such boards, but I don't know them. For data and image acquisition, I would take a look at these pages:

http://www.mathworks.com/products/daq/supportedio.html

http://www.mathworks.com/products/imaq/supportedio.html

[Walter Roberson]

If you are doing start() for two different devices and they are _not_ NI devices interfaced to with the new session level interface, then *yes* you will get significant latency between the audio and video.

I have not investigated whether NI has any video acquisition devices. I would be surprised if they do _not_... but it could be that their video devices don't support session level interface yet (I haven't checked.)

Answer 2

[Walter Roberson]

The old post is still relevant. The only real improvement since then is that if you use the R2010b or later, and you use National Instruments devices and you use the new Session Based Interface, then synchronization over multiple devices becomes easier. The capabilities have evolved rapidly, so use the newest release you can get.

However, if you are using USB, you are pretty much wasting your time if you are using the more common meanings of "simultaneously". USB cannot deliver data from multiple sources to you and have them arrive at your program in synchronization. USB is a serial bus that uses time slicing to share the bus between multiple devices, a packet from one device, a packet from the next... You can use multiple USB controllers so as to have the possibility of having simultaneous acquisition, but then you have to figure out some way of synchronizing the controllers and the micro-cycles on the USB bus. Which is not going to be fun as the video and audio packets are going to be different lengths and using different internal USB packet protocols.

"Real-time" and USB don't mix. Indeed, the real-time programming (e.g., RealTime Workshop) supports only a very very small number of USB devices.

The version of "simultaneous" that can be supported (in theory) over USB has to do with using a common hardware data acquisition trigger for the start of the data block from each device (so the devices have a common time base), with a block count or time-stamp added at the point of data acquisition, and then the data transferred to MATLAB, not arriving at the same time, with the sources then being matched up by block count or time stamp.

I do not recall seeing any USB web-cams that supported hardware triggers (but I never really looked). Most USB web-cams would suffer from not having a synchronization event (count or time-stamp) in the data stream. That leads to problems if frames are ever lost. Which does happen in the USB protocols for both bulk audio and bulk video modes. The USB bandwidth reservation mechanisms for A&V work on the assumption that if there is any data error (e.g., USB link-level) that cannot be corrected by the data-stream's own error correction, then the packet can be discarded. The alternative USB modes that have "safe delivery" ("reliable mode") (checksums, retries) are, like TCP, not for use with real-time data as you never know how long delivery will take.

If you don't want to deal with packet delivery problems, then you need to have the data acquisition (of the original data) attached to the computer running the data merging. That means either direct data acquisition boards, or a cable to a data acquisition controller. And rules out USB (except perhaps in reliable mode in transfers between the acquisition controller and the computer.)

Now, you said that you wanted to record this simultaneous data. Please note that there is no routine in MATLAB that can produce movies with synchronized audio and video -- not unless there is something in the Video Acquisition Blockset in Simulink. Nothing in MATLAB itself. You can use event logging to produce copies of data with all the relevant time-points and so on, but no video-with-sound production.

[Walter Roberson]

(Sorry, posting got delayed by a "Come down and cook supper _now_" interrupt ;-) )

[Valerie]

Thanks for your extremely detailed response!

This is pretty much what I had feared. I have been recording video/audio with an off-board video camera, and then loading that in for analysis.

Maybe it will be more clear if I give a better explanation of what I'm doing:

I'm collecting audio and video data from an experiment. The computer asks the subject a question (pre-recorded audio) and then plays a pure tone. From there, the audio has been recorded using matlab and a USB microphone. If there is a little bit of random latency in the recording, this is okay, because response times are on the order of tens of seconds (and what I'd really be upset about is an error that increases my type I error, but this merely messes with my type II error). The video is currently recorded by a separate device (along with a second recording of audio), and the start of each trial is extracted using a filter for that pure tone I mentioned. The downside of my this setup is (1) it would be nice to have greater accuracy on the audio, if the current setup is adding a significant amount of noise, (2) recording audio and video from the same computer that is controlling the experiment could ensure that there aren't trials where video/audio are failed to be collected, i.e., missing data, (3) it could avoid the video-extraction procedure I must currently undertake to extract the video for each trial from the second device's continuously recorded video.

Using a National Institute device would be nice, but I don't have over $4,000 to do so, such as the devices described here: <http://zone.ni.com/devzone/cda/tut/p/id/6432>

[Walter Roberson]

I will not have time to think through this more until later.

Have you looked at Psychophysics toolbox? http://psychtoolbox.org

Answer 3

[Daniel Shub]

You really need to specify what you mean by "simultaneous". It would surprise me if you could get below 100 ms latency, which is probably good enough for it to be perceived as simultaneous. If you manage to create a simultaneous recording, how are you going to play it back? Without specialized playback hardware there is no reason to sync the recordings

[Valerie]

I don't actually need to play it back, I need to analyze it. Analyzing video and audio separately is fine for my purpose. I'm not exactly sure why I would want to play it back.

[Daniel Shub]

If you are analyzing them separately, how much time can there be between the audio and video? Is a 1 second offset okay?

[Jiro Doke]

Just throwing out ideas, but if you are not super concerned about having it synced to the micro- or even milli-seconds, how about you record some event that can be detected both visually and audibly, and then you can synchronize the event when you analyze them. For example, after you start recording (audio and video), you can tap on some object that makes a sharp noise, and then you'll have both audio and visual cues recorded.

Again, it really depends on what you mean by "simultaneous". I did biomechanics experiments back in the day, and this technique was good enough for my purpose.

[Daniel Shub]

Valerie say the audio and video analysis will be done separately. I take this to mean that as long as the video and audio can be recorded in parallel, as opposed to serially, it should be fine.

[Jiro Doke]

Yeah. Easiest and cheapest is to have two machines record it simultaneously, and just synchronize later.