Page 1 of 1

SQLite and memory-mapped IO

PostPosted: Apr 26, 2017 07:40
by transtep
Hi,

I want to use SQLite to store subjects and perform identification task directly (not via NServer).
But I have a performance issue.
Does biometric client enable memory-mapped I/O feature of SQLite when it performs identification task?

Neurotec Biometric 9 SDK Product revision number: 162703

Re: SQLite and memory-mapped IO

PostPosted: Apr 26, 2017 07:56
by MartynasV
Hello,

When you connect NBiometricClient to database, it copies all templates to memory(and holds copy of current templates in memory for all its existence), so data could be accessed immediately for verification/identification tasks.

Could you explain in more details what performance issue do you have?

Re: SQLite and memory-mapped IO

PostPosted: Apr 27, 2017 07:24
by transtep
Hi,

With Biometric 5 SDK, I invoke NMIdentifyStartEx / NMIdentifyNextEx / NMIdentifyEnd functions to perform identification task. I store all templates in a big file and map to memory (mmap). This way is 100% faster then IdentifyOnSQLiteDatabase.

Re: SQLite and memory-mapped IO

PostPosted: Apr 27, 2017 09:20
by MartynasV
transtep wrote:Hi,

With Biometric 5 SDK, I invoke NMIdentifyStartEx / NMIdentifyNextEx / NMIdentifyEnd functions to perform identification task. I store all templates in a big file and map to memory (mmap). This way is 100% faster then IdentifyOnSQLiteDatabase.

Hello transtep,

Using older API, you control how identification is done and where templates are held, so to ensure high performance you have to load templates to memory and keep them there for fast access yourself(as you did). In newer API (NBiometricClient) all this identification process management is moved inside our API, so when NBiometricClient is initialized, all data is copied from database to local memory for fast access and identification is called with one method: "NBiometricClient.Identify(NSubject)".

Re: SQLite and memory-mapped IO

PostPosted: Aug 29, 2019 07:27
by mohammadhatami
hi
if i use some other method for saving template(like file system/or no sql db/or ...)

when i load template by

NBiometricTask enrollTask = biometricClient.CreateTask(NBiometricOperations.Enroll, null);
enrollTask.Subjects.Add(All Subject);
biometricClient.PerformTask(enrollTask);

to initialize load all template in biometric client in ram

if i call "biometricClient.PerformTask(enrollTask);" once--> is use (all template size)*2 in initialize time
so i need 2* Ram

if i divide all temple to m
and call
biometricClient.PerformTask(enrollTask);
in m steps

its very slow

how can speed up this problem?


i use high performance server
for 2,000,000(1 large face,2 large iris,10 large finger)template
load time near 3 hours!
9 min for read all template
2.51 hours for biometricClient.PerformTask(enrollTask);(5/10/50/500 steps is tested)

Re: SQLite and memory-mapped IO

PostPosted: Sep 03, 2019 14:10
by MartynasV
mohammadhatami wrote:hi
if i use some other method for saving template(like file system/or no sql db/or ...)

when i load template by

NBiometricTask enrollTask = biometricClient.CreateTask(NBiometricOperations.Enroll, null);
enrollTask.Subjects.Add(All Subject);
biometricClient.PerformTask(enrollTask);

to initialize load all template in biometric client in ram

if i call "biometricClient.PerformTask(enrollTask);" once--> is use (all template size)*2 in initialize time
so i need 2* Ram

if i divide all temple to m
and call
biometricClient.PerformTask(enrollTask);
in m steps

its very slow

how can speed up this problem?


i use high performance server
for 2,000,000(1 large face,2 large iris,10 large finger)template
load time near 3 hours!
9 min for read all template
2.51 hours for biometricClient.PerformTask(enrollTask);(5/10/50/500 steps is tested)

Hello,

Try connecting SQLite database to NBIometriClient(as in "SDK\Tutorials\Biometrics\CS\IdentifyOnSQLiteDatabaseCS" tutorial) and on initialization templates would be loaded into memory, it should be faster than calling enroll.