MPEG-4 Part 21 4.1Disposition of Comments

4.2M12068 – A generic architecture for GFX media handling

4.2.1Outline of the proposal

1. 
   Application opens a content - e.g. terminal.open (http://server.com/content.mp4)
   
2. 
   Application starts rendering loop - e.g. get screen area, init graphics hardware etc.
   
3. 
   Application retrieves composition data from decoder - e.g. frame=decoder.getVideoFrame()
   
4. 
   Application renders composition data - e.g. texture map video frame onto surfaces
   

The salient features of the proposal are described below:

11. High-level architecture

1. 
   DataSource for protocol handling
   
2. 
   Player for media handling
   
3. 
   Control for processing handling
   
4. 
   Renderer for presentation handling
   

• 
  Conceptually, not different from MPEG-4 or any other multimedia systems
  
• 
  Higher level API than MPEG-4 (OD/ESD)
  
• 
  Does not enforce a specific implementation of the multimedia system
  

12. Interfacing with MPEGlets

13. URI locators are used :// (RFC 2396). URI are more generic

14. OD access is possible using XMT-like URI: od://od_id [ @es_id ]

15. Content descriptors follow MIME syntax (RFC 2045, 2046) - More generic and already available for MPEG types (RFC 3016, 3555, 3003, 3119, 3640). Much easier to interface with non-MPEG types

16. Application-specific DataSources

• 
  In general, it’s a demux that output multiple streams
  
• 
  A player can be created from such a DataSource
  
• 
  Enable synchronization with other players
  
• 
  Enable apps to decode data frames (AU) themselves
  

Use of MMAPI proposed:

• 
  Extend for elementary stream control and composition buffer access
  
• 
  Control for specific ES processing
  
• 
  CB access reusing GFX previous spec
  
• 
  Accessing composition data
  
• 
  ElementaryStream interface provides access to BufferInfo.
  

• 
  We need access to streams’ CB(Composition Buffer)
  
• 
  We need to attach a renderer to a player
  

• 
  100% compatible with MMAPI spec
  
• 
  Use different MPEG specific interfaces/controls
  
• 
  Wraps MPEG-4 Systems (does NOT replace it)
  
• 
  API can also work with non-MPEG-4 systems
  

1. Persistent storage

• 
  MIDP Record Management System is enough
  
• 
  Record = application-specific byte[]
  
• 
  Storage format is terminal vendor specific
  
• 
  In some cases, the store may need to be used on different terminals. A storage format may be defined.
  
• 
  MMAPI and RMS are already in millions of phones, APIs are ‘safe’
  

4.2.2Decisions

18. Systems interaction

• 
  Connect to a media location using a protocol,
  
• 
  Control of the playback of a media (e.g. play, pause, stop a video stream and its associated audio),
  
• 
  Retrieve the output of a stream for composition on the terminal’s output,
  
• 
  Possibly, control the post-processing of a media
  

• 
  DataSource abstracts protocol handling,
  
• 
  Player abstracts content handling,
  
• 
  Control provides a way to interact with the Player’s processing.
  

Figure 1   Conceptual view of the terminal from an application.

Elementary streams (ES) output composition buffers abstracted by a BufferInfo interface that enables access to a Buffer interface that wraps native composition data:

• 
  For a video decoder, the TexInfo interface extends BufferInfo and provides image specific information. Buffer may wrap a byte array of pixels or a video decoder buffer.
  
• 
  For an audio decoder, Buffer wraps PCM data that can be used by an audio mixing engine. An AudioInfo interface extending BufferInfo provides audio specific information.
  
• 
  For other decoders, specific BufferInfos may be defined.
  

19. Contexts

In this specification, the following contexts are discussed (but not restricted to these contexts only):

• 
  Application context or MPEGletContext – enables the application (MPEGlet) to communicates with the application manager within the terminal.
  
• 
  Rendering contexts – to access graphic resources (e.g. OpenGL driver) and audio resources.
  

20. Resource manager

This interface defines the following methods:

• 
  Player createPlayer(String locator)   creates a Player from a locator, which takes the form of a Uniform Resource Identifier i.e. a string of the form :// for example: http://server.com/movie.mp4. To access a media from an object descriptor, the syntax “od://” is used with replaced by the appropriate object descriptor identifier.
  
• 
  Player createPlayer(DataSource source)   creates a Player for a DataSource.
  
• 
  Player createPlayer(InputStream stream, String content_type)   creates a Player to playback a media from an InputStream.
  
• 
  String[] getSupportedProtocols(String content_type)   returns the list of protocols supported for a content type.
  
• 
  String[] getSupportedContentTypes(String protocol)   returns the list of supported content types for the given protocol.
  
• 
  TimeBase getSystemTimeBase()   returns the system time-base, which is a continuously ticking source of time in the system.
  
• 
  RecordStore getRecordStore()   returns RecordStore object.
  
• 
  Renderer createRenderer(String name)   creates a Renderer given its fully qualified class name.
  

21. Media API

• 
  Movie/audio capture is optional
  
• 
  MIDI and tone support are optional
  
• 
  GUIControl and its derivative VideoControl are not supported because they conflict with usage of Renderers
  
• 
  MMAPI’s Manager factory class is not supported and its methods are instead moved to ResourceManager
  

• 
  Elementary stream composition buffer access
  
• 
  Renderer control
  

Controls defined in JSR-135 are reused except those for video rendering that have been replaced by GFX RendererControl to attach a player to a renderer and StreamControl for accessing elementary streams output.

Figure 3 – Predefined controls

22. Persistent storage (record store)

The Mobile Information Device Profile (MIDP) provides the Record Management System (RMS) in the javax.microedition.rms package. A record store consists of a collection of records that will remain persistent across multiple invocations of the MPEGlet. RMS records consist of byte arrays.

An application can store any kind of information including elementary streams of a given content. Stored contents can be played back from the store by creating a Player from an InputStream encapsulating a stored content

4.2.3Issues