Friday, August 31, 2012

Reading and processing video frames with Java


The class VideoSource loads a video file and offers access to the individual frames.

To use this class you need to have JMF installed on your machine. The video format must be supported by your Java/JMF platform. Best is to use no codec at all; "raw" video.

The usage is as follows:

VideoSource vs = new VideoSource("file://c:\test.avi");
vs.initialize();
...
int frameIndex = 12345; // any frame
BufferedImage frame = vs.getFrame(frameIndex);

package popscan; 
import java.awt.Graphics2D
import java.awt.Image
import java.awt.image.BufferedImage
import java.net.URL
import javax.media.Buffer
import javax.media.ControllerEvent
import javax.media.ControllerListener
import javax.media.Manager
import javax.media.Player
import javax.media.PrefetchCompleteEvent
import javax.media.RealizeCompleteEvent
import javax.media.control.FrameGrabbingControl
import javax.media.control.FramePositioningControl
import javax.media.format.VideoFormat
import javax.media.util.BufferToImage
public class VideoSource implements ControllerListener { 
public static final int NOT_READY = 1; 
public static final int READY = 2; 
public static final int ERROR = 3; 
Player _player; 
String _videoFilename; 
FramePositioningControl _framePositioningControl; 
FrameGrabbingControl _frameGrabbingControl; 
private int _state; 
// the filename must contain protocol,  
//for example file://c:\\test.avi 
public VideoSource(String videoFilename) { 
    _videoFilename = videoFilename; 
    _state = NOT_READY
} 
/* 
 * Create Player object and start realizing it 
 */ 
public void initialize() { 
    try { 
        _player = Manager.createPlayer(new URL(_videoFilename))
        _player.addControllerListener(this)
        // realize call will launch a chain of events,  
        // see controllerUpdate() 
        _player.realize()
    } catch (Exception e) { 
        System.out.println("Could not create VideoSource!")
        e.printStackTrace()
        setState(ERROR)
        return
    } 
} 
/* 
 * Returns the current state 
 */ 
public int getState() { 
    return _state; 
} 
/* 
 * Returns the number of frames for current video if  
 * the VideoSource is ready, in any other case returns -1. 
 */ 
public int getFrameCount() { 
    if (getState()!=READY) { 
        return  -1; 
    } 
    return  _framePositioningControl. 
            mapTimeToFrame(_player.getDuration())
} 
/* 
 * Returns the video frame from given index as BufferedImage.  
 * If VideoSource is not ready or index is out of bounds, 
 *  returns null. 
 */ 
public BufferedImage getFrame(int index) { 
    if (getState()!=READY||index<0||index>getFrameCount()) { 
        return null; 
    } 
    _framePositioningControl.seek(index)
    Buffer buffer = _frameGrabbingControl.grabFrame()
    Image img = new BufferToImage((VideoFormat)buffer. 
            getFormat()).createImage(buffer)
    // image creation may also fail! 
    if (img!=null) { 
        BufferedImage bi = new BufferedImage(img.getWidth(null),  
                img.getHeight(null)BufferedImage.TYPE_INT_ARGB)
        Graphics2D g = bi.createGraphics()
        g.drawImage(img, 0,0, null)
        return bi; 
    } 
    return null; 
} 
     
// callback for ControllerListener 
public void controllerUpdate(ControllerEvent event) { 
    if (event instanceof RealizeCompleteEvent) { 
        _player.prefetch()
    } else if (event instanceof PrefetchCompleteEvent) { 
        // get controls 
        _framePositioningControl = (FramePositioningControl)_player. 
          getControl("javax.media.control.FramePositioningControl")
        if (_framePositioningControl==null) { 
            System.out.println("ErrorFramePositioningControl!")
            setState(ERROR)
            return
        } 
        _frameGrabbingControl = (FrameGrabbingControl)_player. 
            getControl("javax.media.control.FrameGrabbingControl")
        if (_frameGrabbingControl==null) { 
            System.out.println("ErrorFrameGrabbingControl!")
            setState(ERROR)
            return
        }             
        setState(READY)
    } 
} 
// for setting the state internally 
private void setState(int nextState) { 
    _state = nextState; 
} 
     
} 

And here is a tester class to demonstrate how the VideoSource works.


It creates an AWT Frame and draws the video frames one by one as fast as possible to the Frame's Graphics surface.

First it draws the frames from the first frame to the last frame and when the loop ends, the frames are drawn from the last frame to first frame. This is to show that you really can access the individual frames.

In this case the frames to be drawn are next to each other, so if you are accessing the frames randomly, you may encounter some performance issues.

package popscan; 
import java.awt.Frame
import java.awt.image.BufferedImage
public class TestFrame { 
public static void main(String[] args) { 
    VideoSource videoSource = new VideoSource(args[0])
    videoSource.initialize()
    while (videoSource.getState()==VideoSource.NOT_READY) { 
        try { Thread.sleep(100)} catch (Exception e) { } 
    } 
    if (videoSource.getState()==VideoSource.ERROR) { 
        System.out.println("Error while initing"+args[0])
        return
    } 
    Frame frame = new Frame()
    frame.setVisible(true)
    int frameCount = videoSource.getFrameCount()
    // forward 
    for (int frameIndex=0;frameIndex<frameCount;frameIndex++) { 
        BufferedImage i = videoSource.getFrame(frameIndex)
        // do what ever you need to do your frame 
        drawFrame(frame,i,frameIndex)
    } 
    // backwards 
    for (int frameIndex=frameCount;frameIndex>=0;frameIndex--) { 
        BufferedImage i = videoSource.getFrame(frameIndex)
        // do what ever you need to do your frame 
        drawFrame(frame,i,frameIndex)
    } 
    System.exit(0)
} 
public static void drawFrame(Frame frame, BufferedImage image,  
        int index) { 
    if (image!=null) { 
        frame.setSize(image.getWidth(), image.getWidth())
        frame.getGraphics().drawImage(image, 0, 0, null)
        System.out.println("Image at index: "+index)
    } else { 
        System.out.println("null image")
    } 
     
} 
} 

7 comments:

Anonymous said...

Good work... Helped me a lot.
Which all video formats does the above code support?

sri said...

hey,
i tried your code for .avi video but got error as:
"
Unable to handle format: MPG4, 440x380, FrameRate=23.9, Length=501600 0 extra bytes
Failed to realize: com.sun.media.PlaybackEngine@173ceeb6
Error: Unable to realize com.sun.media.PlaybackEngine@173ceeb6"
please help with this, what should i do.

Jussi said...

Hi! Looks like your environment does not contain suitable codec for decoding video file with format "MPG4". You could try to convert the contents to "YUV" or "Cinepak" etc format.

Sadly, Java does not support modern video formats natively.

You can check the supported formats from official page http://www.oracle.com/technetwork/java/javase/formats-138492.html

Pratiksha said...

I am trying the tester for the Videosource functionality.
I have installed the jmf.
But, still I am getting the error as -
error: cannot find symbol
VideoSource videoSource = new VideoSource(args[0]);

Do I have to do an import statement-
import popscan.*;

Or do I have to change the classpath??

How will the tester class refer to the Videosource class.

Jussi said...

Hi. If you use the code as is, you need to place them to folder popscan (that is the package).

Otherwise, you need to make sure that the JMF libraries are in your classpath. Please, use the configuration tool that is installed with JMF. The tool should make the necessary checks for the paths.

Neha Alreja said...

Hi, I am getting an error on line
VideoSource videoSource = new VideoSource(args[0]);
I have installed JMF and also added the jar files to my project. I have also taken care of the package name.

Please tell me what else i may be missing.

Thank you.

Neha Alreja said...

Hi, the previous issue was my problem.
Now after running i am getting the following error,
Error: FramePositioningControl!
The framePositionControl object is getting null in initialization.

Please Help.
Thank you.