I am investigating if I can process real-time video using SkiaSharp. I want to see if I can replicate (or achieve something similar to) what I do in UWP using Win2D. As the apps that we are working on, I'm hoping that using SkiaSharp I can zero in to an increasingly cross-platform solution.
To illustrate, in UWP using MediaCapture, I can apply effects to video frames in real-time using Win2D by implementing the IBasicVideoEffect interface and do something like:
public void ProcessFrame(ProcessVideoFrameContext context)
{
using (CanvasBitmap inputBitmap =
CanvasBitmap.CreateFromDirect3D11Surface(_canvasDevice, context.InputFrame.Direct3DSurface))
using (CanvasRenderTarget renderTarget =
CanvasRenderTarget.CreateFromDirect3D11Surface(_canvasDevice, context.OutputFrame.Direct3DSurface))
using (CanvasDrawingSession ds = renderTarget.CreateDrawingSession())
{
_invertEffect.Source = inputBitmap;
ds.DrawImage(_invertEffect);
}
}
While this example illustrates an invert effect, I would like to be able to apply other effects / filters already available in SkiaSharp and also draw on frames when desired. The intent is to render the processed frames in the image preview and any videos / photos taken.
Instead of trying to rewrite what I have already got in my Xamarin.UWP implementation, I'm trying to implement this in Xamarin.Android. I have got my Camera2 implementation going using a TextureView and am now trying to figure out how I can leverage the SurfaceTexture associated with this to draw on each frame using SkiaSharp. I am hopeful this is a viable path as I can see SkiaSharp 1.60.0 implements SKGlView as a TextureView in Android.
What I think I need to do is to use GLContext and GRBackendRenderTargetDesc to create an SKSurface, but I have got a bit stuck with this. I couldn't understand how I will grab the GLContext associated with the TextureView and generate the GRBackendRenderTargetDesc.
I would appreciate a confirmation of if the approach I'm attempting makes sense and if it is viable. Also, I'm open to pointers to better or more appropriate ways of achieving this. Also, any pointers, samples or help regarding how to create an SKSurface from the existing TextureView would be most welcome.