Migrating from kernel to gstreamer drivers

I've been asked a number of times why I stopped developing Linux kernel camera drivers and instead bind them to gstreamer and/or write userspace libraries. This article takes my journey from gspca_touptek to gst-plugin-toupcam as a case study.

TLDR:

• Quicker model support
• Community better structured to accept contributions
• Better cross platform support

Linux kernel

Everyone told me the "proper" way to support a camera is to get it upstreamed in the Linux kernel. So let's give it a go!

First, getting a driver PoC required breaking their protocol obfuscation and writing a driver. While I can do binary reverse engineering, there are those much better than me and fortunately I got some help from Ben Byer. With his help I was able to make a utility to break the encryption and develop a PoC userspace application. This process was a big motivation to develop my usbrply USB reverse engineering tool / framework which ended up being my most popular open source project once I cleaned up the documentation.

Anyway, getting it merged was a high touch process involving changing APIs and tweaking to coding preferences. This was especially frustrating in some instances where I copy and pasted existing code that no longer conformed. This is all reasonable given the scale of a project like the Linux kernel, but it discourages smaller contributors that don't do this as their full time job.

I wanted to use a nicer camera but was pretty tied to MU800. In fact I had even already written a driver for an MD series camera and even received some patches for MU300 but wasn't excited about going through the process again to merge them in. Even if I did the work it would take a long time to filter through distribution channels to Ubuntu and such.

There were also practical problems. The color balancing scheme I had (via gspca?) is green gain, red balance, and blue balance. This meant that it was hard to have accurate blue channel control when you had low green gain and other quality issues that didn't allow using the full camera capabilities. In general I really need direct setting (register) control to get good quality images. However, kernel devs are strongly against allowing raw I2C access and instead want proper controls implemented for everything which greatly increases the complexity of the project. Ultimately I use a small patch to hack around direct RGB control, the most critical issue.

gstreamer

But wait, do I actually need a V4L driver for my use case? What are my requirements? Ideally:

• Available through gstreamer on Linux
• Support MU800
• Support future touptek cameras like E3ISPM series
• Plus if it runs on ARM like Raspberry Pi
• Plus if its open source
• Plus if its cross platform

Could I make a gstreamer plugin instead? Well I could make a userspace driver as I did anyway sometimes prototyping the kernel drivers. But touptek also publishes an SDK (C and Python bindings).  I took a look at the SDK and quickly got some images out in Python. Could I pipe that in somehow? It's not open source, but would satisfy everything else.

I took a look at getting started with gsreamer development but ran into a few snags. I can't recall specifics, but they had a quick start tutorial that partially worked but didn't get me far enough to actually get started. I think it was a combination of not finding an example on how to make a video source and issues related to Meson. Ultimately I found someone on Upwork that helped me to get a PoC to captured a single image as a C gstreamer plugin.

Once I had a PoC I was able to iterate for what I really wanted. At this point I was communicating with gstreamer developers for both my plugin development and userspace app. They were super friendly and helped me when I ran into problems. Additionally, while Linux has a monolithic plugin repository, gstreamer has an interesting approach with several classes of plugins loosely like:

• good: good quality code, correct functionality, our preferred license 
• ugly: good quality and correct functionality, but problematic distribution
• bad: fundamental issues (ex: licensing, stability), but its useful enough to be kept

This means they can accept things that might otherwise be rejected by putting it into the right bucket. 

Additionally they are more accepting of out of tree plugins. This significantly reduces my development overhead as:

• Friendlier to use existing system software instead of compiling gstreamer from source
• I can make my own small git repo instead of merging into mainline

I also found other communities maintaining industrial camera gstreamer plugins and had talks to them about the merits of merging it into their repos vs keeping my own. They were open to me merging things in but, at least at the time, it was better to have more direct control to keep development moving quickly.

Finally, gstreamer sometimes retweets gstreamer related things I post. Even though I didn't merge these patches into mainline, its a nice acknowledgement that I'm still part of the community.

Final thoughts

Both approaches have their merits but for me gstreamer has been a clear win. Its sad I've dropped my other kernel patches but the ROI for me to get them merged is too low.

I've also been told its possible to do a userspace-v4l bridge, but haven't looked into this much. My instinct says that direct gstreamer support is better for my use case.