Generating Interactive Particle Systems in OpenCV and SpriteKit

This project is a proof of concept for capturing infrared video from a homebrew camera, sending the frames through OpenCV to generate keypoints and contours, and then forwarding that data into SpriteKit as the source vector for particle systems and other visual highlights. The video analysis and particles are processed in real-time on device.

The links below show the components used for a homebrew IR camera. In order to make the camera, one just cracks open the webcam and slips in a slice of the IR filter. The light helps create contrast that becomes helpful for analysis in OpenCV.

• Webcam (Amazon)

• Lighting (Amazon)

• Filter (B&H Photo)

The video below shows an OpenCV preview of the homebrew camera generating keypoints and contours on OpenCV.

Extracting a single channel of color information as grayscale data can help speed up the video analysis. The code below shows how some low level APIs send the data into OpenCV and then on to an operation queue used by another component in the application.

Converting a CMSampleBufferRef to a single channel of color

The video below shows the extracted keypoints and contours generated in SpriteKit:

Once the data is available, SpriteKit effects, such as particle systems, create nice effects.

This rendering was projected back into the space being captured at an art event on Chelsea Piers.