SentiSight object learning and recognition processes

Object learning process

Object learning
Click to zoom

In order to recognize an object in an image, the appearance of an object must first be memorized. In the learning phase, SentiSight algorithms extract specific object features from a video stream or single image and save them into what is known as an object's model.

In many cases there is more information in a video or single image than just the object you want SentiSight to learn, like a background, other objects in the room or a hand holding the object. Therefore, to learn an object, information about the exact location of the object in the image should be provided.

SentiSight supports 2 methods of object learning: manual and automatic.

Manual object learning is suitable for most situations. A user must perform these steps for manual object learning in the SentiSight 2.1 SDK:

1. Outline object's shape on an image by marking object's corner points to build a polygon. The image can be provided by user from image file, video file or live video stream.
2. Optionally choose more images of the object and repeat Step 1 for each image. The algorithm assists the user by estimating an approximate shape of the object if the object in an image is recognized using data from previous images. Learning the object from different sides and angles results in better recognition quality.
3. Input the learned object name (ID) into the system.

Automatic object learning is suitable for lightweight movable objects. This learning procedure is based on detecting an object by excluding a static background and object's holder (usually a hand) from an image.

A user must perform these steps for automatic object learning in the SentiSight 2.1 SDK:

1. Choose a static background (preferably smooth) and direct the camera to it.
2. Choose a holder – an object that will be used to hold and move the learned object. A user's hand can be the "holder".
3. This "holder" should be presented it to the camera first, in various poses and configurations (if it is not rigid object) so that it can be learned by SentiSight.
4. After the holder has been learned, SentiSight is ready to learn the object itself, by having the holder rotate and move the object closer and further from the camera.
5. Input the learned object name (ID) into the system.

Therefore, the automatic method requires to use live video or to provide separate videos or image sets of background, holder and object. Also, the other background elements could be learned together with the object if the object is hardly separable from the background. This can affect the ability of the algorithm to recognize the unique qualities of the object and may result in the object being misclassified with other objects that have the same background.

Manual object learning should be used for objects that cannot be moved or if there are no way to provide separate media with objects background and/or holder. Thus, automatic learning provides less amount of user interaction with the system, but it is not as precise as manual learning. Also manual learning is suitable for wider range of cases.

Object recognition process

Object recognition
Click to zoom

Object recognition requires no user interaction apart from providing a video file with the object or pointing a camera to the scene where the learned object is presented or will appear. When the object appears in the vision field, SentiSight tries to recognize it. If the object is recognized by SentiSight, object's name (ID) and coordinates are returned.

The SentiSight algorithm creates a model with possible views from different sides, in different 3D poses and in different lighting conditions in object learning stage. This object's model improves recognition capability.

SentiSight's object recognition is comparably fast – around 10 frames per second for a single object model (320 X 240 resolution). However for tasks when an even faster response is needed, the SentiSight 2.1 library has a tracking mode that enables tracking speeds up to 20 frames per second. Tracking is initialized if an object is recognized and located, then tracks the object until it changes somewhat in appearance, at which point tracking is reinitialized by recognition. The tracking feature is sensitive to complex backgrounds, and tracking is more difficult with homogenous objects

Go to SentiSight contents

SentiSight algorithm capabilities and requirements

SentiSight is designed to be as universal as possible and is able to perform fully automatic and manual object learning. Some of the potential applications for SentiSight include security systems, vision systems for robots, machine vision (like parts recognition in production lines), search engines that recognize objects in picture files, road signs recognition, etc.

The SentiSight 2.1 algorithm has these capabilities for advanced visual-based object learning and recognition:

• Accurate object detection. The SentiSight algorithm is able to find out:
  • whether a particular object is presented in a scene;
  • where the object is located in the scene;
  • how many instances of the object are there in the scene.
• Simultaneous multiple object recognition. The SentiSight algorithm provides simultaneous multiple 2D and 3D object detection and recognition
• Object evaluation. The algorithm is also able to estimate the region an object occupies in a scene, providing additional information about the size, orientation and scale of the recognized object.
• Fast image processing. SentiSight can process video streams in real time, so it can be used for real-time applications.
• Rotation and translation tolerance. The algorithm is generally rotation and translation invariant in a plane perpendicular to the camera.
• Free rotation tolerance. The algorithm is invariant for rotations up to 10-15 degrees out of a plane perpendicular to the camera. Different views of an object can be added to a model to handle larger rotations.
• Resolution and scale tolerance. Scale (size in image) difference between object's model and object itself can be up to 2-3 times. Objects should contain enough details, and be large enough to be recognized.
• Occlusions tolerance. The algorithm is robust to occlusions as big as 50% of the objects size.

These conditions may alter algorithms performance:

• Lighting conditions (illumination, shadows and reflectance). Planar objects only have problems with reflectance. 3D objects have problems with varying lighting conditions, but constant lighting conditions do not cause many problems.
• Transparency. Generally transparent objects are difficult to recognize.
• Rigidity. The algorithm can recognize only rigid objects. At least significant part of the object should be rigid.

All performance evaluations were made using a PC with 2.4 GHz Intel Core2 Duo CPU

Go to SentiSight contents

Reliability Tests and Technical Specifications

SentiSight 2.1 was tested with object images from many cameras. At 0.1% False Acceptance Rate (FAR), the recognition rate is from 70% to more than 99% depending on object structural appearance, transparency, etc. For objects with well defined intenal structure, the recognition rate is 98% - 99% at 0.1% FAR.

Go to SentiSight contents