A library to take a continuous media source
[camera, video, gstreamer byte arrays], apply OpenGL
[vertex+fragment shaders], and configure rendering in shape, size and rotation in 3D space the input
and direct to any/multiple
Surfaces on Android. The output can then be streamed using Gstreamer or any
streaming library from an offscreen Surface's callback.
The best explanation to understand how + why this was made is in the library's
readme of the library hosted internally on Tonbo's GitHub.
MediaSource | Attach [Multiple] Output Surfaces | Provide a Filter | Provide a Renderer ... Send to Display ... Callback from Surface (if any) ... Send data from callback to Gstreamer
Each surface is an output destination to where the
mediasource's frames are rendered to. You can provide
custom filters and renderers.
Media Source Types
A Media Source is any data source that writes to a
- Any DataSource writing to
SurfaceTexture. This could be a custom data source created by you which writes to a
SurfaceTexture. To do so you must use
The library provides support for the following output surfaces by default:
SurfaceViewYou can attach/deattach surfaceviews at runtime to increase or decrease the number of surfaces being written to.
MediaMuxerUse this when you would like to record a video
Surfacewith a data callback Use this when you would like data callback from a
Surfacewhich encodes the
Media Source's frames.
Offscreen SurfaceThis is useful for when you want to render to an
offscreen surface, ie, any surface which isn't backed by a view, or MediaCodec. The library allows you to asynchronously read the frames from this surface internally using a
These are OpenGLES shader programs. Each filter has a
Fragment shader. There is
a list of filters provided in the library. We've re-used several filters from Bard Larson's GPUImage library.
Creating your own Filters
- if you'd only like to change the fragment shader, create a class which extends
BaseFragmentShaderwhich provides a
mediaTextureIdwhich is the texture id of the data source's current frame.
- if you'd like to change both the fragment and vertex shader, extend the
- Your fragment shader must also name the mediaTextureId
sTexturein the program if you'd like to use the
This is how the frame is rendered to the
surface. You are provided a
Sprite3d helper class
that allows you to move the video sprite in 3D with simple builder functions.
Creating your own Renderer
- extend the
- ensure you provide your
- Define your rendering defaults in the
- set your viewport and draw your Sprite in
onDraw. This is called every time the