The ScreenPlay Graphics Architecture

Key features

• Asynchronous hardware-accelerated rendering and composition on devices on which dedicated graphics acceleration hardware is available. This is achieved through a Hardware Adaptation Layer (HAL).

• The ability to composit a semi-transparent UI buffer over highly dynamic content, such as OpenGL ES games, video and the camera viewfinder.

• The separation of control and data flow. This has advantages when running on non-uniform memory architectures and means that video decoding, UI rendering, and so on can take place and remain in the GPU memory domain.

• A foundation for secure screen content and Digital Rights Management (DRM). Applications no longer have direct access to the screen. Read and write access to the screen is controlled by the Window Server.

• Direct Screen Access (DSA) is supported in order to provide backwards compatibility. However, because in ScreenPlay the screen is no longer controlled by the Screen Driver, the DSA frame buffer is just another buffer that can be allocated dynamically on demand. ScreenPlay provides alternatives to DSA.

Architecture

The following diagram shows the key components in the Symbian Graphics package and some closely related components in other packages.

Figure: Symbian^3 component architecture

The key ScreenPlay components are introduced below under separate subheadings.

Graphics Composition

The composition engine composes content, possibly from several different sources, before it is displayed on the screen. Composition involves the important concepts of scene elements (or layers) and surfaces. Scene elements describe the geometric position, size and orientation of items to be displayed on the screen; whereas surfaces are pixel buffers for holding an image or part of a scene.

The composition engine maintains the stack of scene elements and computes what is visible. For example, it culls invisible areas and maintains a list of dirty rectangles. It blends the pixels if necessary and can perform limited transformations, such as scaling and rotation (in 90° increments). The composition engine is an adaptation component, which means that device creators can adapt or replace it to suit the exact hardware on the device. The composition engine can utilize GPU hardware composition and LCD hardware rotation if they are available.

The composition components are specific to ScreenPlay. For more information, see Graphics Composition.

Surface Manager and Surface Update Server

The Surface Manager component creates and manages graphics composition surfaces. The Surface Manager reference implementation implements surfaces as shared chunks because they must be accessible by user-side processes and the kernel and composition hardware. Surfaces can be multi-buffered and are identified by a 128 bit identifier (called the surface ID). This gets resolved to an actual memory address by calling the Surface Manager Map Surface API. Surfaces can be used by other Symbian components such as the Multimedia Framework (MMF) and by applications such as OpenGL ES games. The Surface Manager is an adaptation component and so can be adapted or replaced to suit the hardware.

The Surface Update component provides a communication mechanism between the composition engine and clients. This is particularly useful for clients (such as video) that produce fast updates and use multi-buffered surfaces.

Window Server

The Window Server has been extended with a render stage framework, which enables the last stage of the Window Server rendering to be customizable through render stage plug-ins. This process, known as "deferred rendering" is achieved by intercepting the output of the Window Server and then deciding how that output should be rendered. For example, the output can be hardware accelerated or it can be sent to a third party graphics engine. The render stage framework enables device creators to integrate different UIs and runtime environments (such as Flash or Silverlight) and to achieve transition effects such as slide, zoom and fade.

Symbian provides more than one render stage solution. The following diagram provides a simplified representation of one possible solution (called solution A in this topic). This solution is full featured. The diagram focuses on the more relevant components and does not attempt to show all components in the complete solution. This solution has a dependency on the S60 middleware layer, in particular on the Hitchcock component (which is in the UI Accelerator package).

Figure: Render stage solution A

Another possible solution (called solution B) is based on the DirectGDI and Graphics Resource components (which are described next), both of which have interface and adaptation layers. This solution is not full featured. Like the previous diagram, this diagram focuses on the more relevant components and does not attempt to show everything.

Figure: Render stage solution B

Both of these render stage solutions mean that existing Window Server applications can take advantage of hardware acceleration if it is available (and therefore run faster) without recompiling the code.

ScreenPlay provides extensions to the Window Server client-side API, which enable mobile devices to respond to events from a number of pointers, including their proximity and pressure. This feature is known as advanced pointers.

A new API, RWsSession::Finish(), has been added to allow Window Server client applications to synchronize with the completion of Window Server rendering. The existing API, RWsSession::Flush(), is redefined to simply flush the client-side command buffer, whereas previously it also provided a guarantee that Window Server had completed the command buffer’s operations. This behavioral change allows legacy clients to benefit from the asynchronous hardware rendering when supported by the render stage plug-in(s) that are in use.

For more information, see Windowing Collection.

DirectGDI

DirectGDI provides a graphics context that can be hardware accelerated and allows an asynchronous interface. DirectGDI has two parts: a generic layer, which provides a client API and an adaptation layer. Device creators can replace the adaptation layer with an implementation that takes advantage of graphics accelerated hardware, if it is available, or a software implementation, if it is not available.

DirectGDI was introduced as a prototype in the development of ScreenPlay. It is deprecated in Symbian^3.

Graphics Resource

The Graphics Resource component provides an abstraction layer for the memory management of pixel and non-pixel data (such as OpenVG command lists). Like DirectGDI, it has a generic part, which provides a client API and an adaptation part, which device creators can adapt to take advantage of graphics hardware when it is available.

The Graphics Resource component was introduced as a prototype in the development of ScreenPlay. It is deprecated in Symbian^3 and will be removed in Symbian^4. However, a new Graphics Resource Interface component is planned for S^4. This new component will provide a similar but reduced API that is optimized for sharing images across processes.

OpenVG, OpenGL ES and EGL

Symbian provides support for OpenVG, OpenGL ES and EGL. The main advantage of ScreenPlay with regard to EGL is that EGL can render into composition surfaces. For application developers this offers the ability to have semi-transparent GDI content on top of EGL content. The EGL client can query whether these new features are supported on the particular device.

For more information, see Khronos API Support.

Screen Driver

In ScreenPlay, the implementation of the Screen Driver has been changed so that DSA content can be passed into the composition engine.