Draft of aRts Media Interface
=============================

1. General

This is only a first draft of a possible Media Interface in aRts. It is neither
completely implemented yet, nor in a fixed unchangable state. Feel free to
explain your ideas how to improve the interface.


2. Goal

2.1 Current state of aRts

aRts was originally developed to handle data streams representing digital
audio. It was built around the needs of those streams. For some
time it has support for asynchronous next to synchronous streams.
aRts gives you the tools to work with audio data in a very modular, easy to use
and intuitive way. With the right tools (aRts Builder) every kid could stick
audio modules together to control the flow of the audio data.
For some weeks there is very simple support to play video files by using
aRts objects. But it ignores more or less the strength of using aRts modules
to stream the video data.


2.2 Goals of aRts Media Support

The central idea of this Media Support interface is to allow the modular 
handling of any media data (especially video) as it is possible with audio.
It should enable you to choose the right filter modules to stick them together
into a much more complex filter graph that can load, decode, manipulate, 
encode, render or save your video data.

To give an example I will create a small screnario. You want to play a MJPG
compressed video file that you can get from many professional video capture 
cards. Let's say it is stored as an AVI files. You will pick a AVI loader 
module  at first. It will have output streams for MJPEG video and 16 bit 
audio for example. Take another filter for the audio, a audio player, and 
connect it with the AVI loader. To decode the MJPEG video data you have to take
the MJPEG module and connect the input stream with the AVI loader and the 
output stream with a X11 video renderer. Press the start button and enjoy the 
video. You don't want to display the video, but store it as raw RGB images? 
No problem. Replace the renderer with a AVI file writer and you're finished.


3. Problems to solve

Unlike the handling of raw audio data, streaming of video or other media may be
much more problematic to do. There're a several areas that have to be 
addressed to make it fast and flexible.


3.1 Format

There are a lot of different video format out there that use completely 
different compression algorithm, data format, color spaces, frame sizes....
In short the differences between two of those formats may be very huge. 
Moreover there can be a lot of parameters that can differ in one single 
format. For example you can save an Intel Indeo video in different sizes 
for different TV standards.

This results in the demand for a flexible format handling for media streams.
You can't connect every two filters without finding out a common format 
or sticking a conversion filter in between.


3.2 Data amount

An audio data stream typically has a bandwith of around 150 KB/s. If you want
to stream video through your computer memory you will get much higher values 
up to several MB/s! Copying 150 KB per second around doesn't hurt much. 
Doing this with a video streams will break down the frame rate dramatically.

To reduce the need of bandwidth there shouldn't be more copy processes than
absolutely needed. To reach this goal techniques like shared memory segments
may be necessary.


4. API Proposal

4.1 Base classes/interfaces

interface MediaModule : SynthModule {
    /* ask module whether the format is supported
     * returns true if supported
     */ 
    boolean isSupported( format fmt, string strm );

    /* let module propose a format by setting all still unset parameters    
     */
    format proposeFormat( format proposal, string strm );

    /* sets format. it is called mainly by connect
     * return true if successfull
     */
    boolean setFormat( format fmt, string strm );

    /* get set format
     * returns an empty or incomplete format if it isn't set yet 
     */
    format getFormat( string strm );    
};

class Format : public map<string, string>
{
 public:
    bool isSet( string key );
};


4.2 Stream Definition

interface MediaFilter : MediaModule {
	in media stream videoin supporting "MPEG1", "MPEG2";
	out media stream videoout supporting "RAW";
};


4.3 Examples

4.3.1 Connection of modules without passing any format

MPEGLoader loader;
MPEGDecoder decoder;

if ( !connect( loader, decoder ) )        
{ 
	cerr << "can't connect modules" << endl; exit(1); 
}


4.3.2 Connection of modules with changed format

Format format;
format["width"] = "160";
format["height"] = "120";

if ( !connect( decoder, "videoout", renderer, "videoin", format ) )
	...


4.3.3 Elementary connection of modules

Format format;
format["colorspace"] = "YUV442";

if ( decoder.isSupported( "videoout", format ) )
{
        decoder.proposeFormat( "videoout", format );
	connect( decoder, "videoout", renderer, "videoin", format );
}









