SentiSight SDK

Object recognition for robotics and computer vision

SentiSight is intended for developers who want to use computer vision-based object recognition in their applications. Through manual or fully automatic object learning it enables searching for learned objects in images from almost any camera, webcam, still picture or live video in an easy, yet versatile, way.

Available as a software development kit that provides for the development of object recognition systems for Microsoft Windows or Linux platforms.

Object learning and recognition processes

SentiSight has two operation modes: learning and recognition. In learning mode, the SentiSight algorithm creates an object model by extracting object features from an image or video. In recognition mode, SentiSight finds and tracks objects with features matching those previously stored in object models.

Object learning process

Object learning
SentiSight learns an object in manual mode Click to zoom
Object learning
SentiSight learns an object in manual mode

In order to recognize an object in an image, the appearance of an object must first be catalogued. In the learning phase, SentiSight algorithms extract specific object features from a video stream or single image and save them into an object model.

In many cases there is information in a video or single image beyond the features of an object you wish SentiSight to learn: a background, other objects in the room, a hand holding the target object. For SentiSight to properly learn an object, information about the exact location of the object within the image should be provided.

SentiSight supports 2 methods of object learning: manual and automatic.

Manual object learning is suitable for most situations. A user performs the following steps for manual object learning in the SentiSight-based application:

  1. Outline an object's shape on an image by marking the object's corner points to build a polygon. The image can be provided from an image file, video file or live video stream.
  2. Select the algorithm to use: blob-based, shape-based, or both.
  3. As an option, additional images of the object may be provided, repeating Step 1 for each image. The algorithm assists the user by estimating an approximate shape for the object if the image is recognized by way of previously catalogued images. Learning the object from different sides and angles results in better recognition quality.
  4. Input the learned object name (ID) into the system.

Automatic object learning is suitable for lightweight, movable objects. This learning procedure is based on detecting an object through the exclusion of a static background and the object's holder (usually a hand). A fixed camera is highly recommended for this process.

A user performs the following steps for automatic object learning in the SentiSight-based application:

  1. Select a background and position the camera.
  2. Select a holder – an object that will be used to hold and move the object to be learned. A user's hand can be the "holder".
  3. The "holder," if it is not a rigid onject, should be presented to the camera in various poses and configurations so SentiSight can learn it.
  4. Select the algorithm to use: blob-based, shape-based, or both.
  5. After the holder has been learned, SentiSight is ready to learn the object itself. Use the holder to rotate and move the object, both closer to, and further from, the camera.
  6. Input the learned object name (ID) into the system.

The automatic method requires the use of live video or separate video / image sets of background, holder and object. Other background elements may be learned together with the object if the object is hardly separable from the background. Too little disparity between object and background, if not learned together, may affect the ability of the algorithm to recognize an object's unique qualities, possibly resulting in the object being misclassified along with other objects having the same background.

Manual object learning should be used for objects that cannot be moved or if there is no way to provide separate media of an object's background and/or holder. Automatic learning requires less user interaction with the system, but it is not as precise as manual learning. Manual learning is suitable, generally, for a wider range of cases.

Object recognition process

Object recognition
SentiSight recognizes the object Click to zoom
Object recognition
SentiSight recognizes the object

Object recognition requires no user interaction apart from providing a video file containing the object or pointing a camera at a scene wherein the learned object is present or will appear. When an object appears in the vision field, SentiSight works at recognizing it. If the object is identified by SentiSight, the object's name (ID) and coordinates are returned.

The SentiSight algorithm creates a model in the object learning stage, using views of the object from different sides, different 3D poses and in variable lighting conditions, improving the system's overall recognition capability.