SQLite and memory-mapped IO

General information about Neurotechnology and its software

SQLite and memory-mapped IO

Postby transtep » Apr 26, 2017 07:40

Hi,

I want to use SQLite to store subjects and perform identification task directly (not via NServer).
But I have a performance issue.
Does biometric client enable memory-mapped I/O feature of SQLite when it performs identification task?

Neurotec Biometric 9 SDK Product revision number: 162703
transtep
 
Posts: 5
Joined: Sep 01, 2016 09:43

Re: SQLite and memory-mapped IO

Postby MartynasV » Apr 26, 2017 07:56

Hello,

When you connect NBiometricClient to database, it copies all templates to memory(and holds copy of current templates in memory for all its existence), so data could be accessed immediately for verification/identification tasks.

Could you explain in more details what performance issue do you have?
Martynas V
Neurotechnology
MartynasV
Neurotechnology
 
Posts: 531
Joined: Feb 27, 2015 11:24
Location: Vilnius, Lithuania

Re: SQLite and memory-mapped IO

Postby transtep » Apr 27, 2017 07:24

Hi,

With Biometric 5 SDK, I invoke NMIdentifyStartEx / NMIdentifyNextEx / NMIdentifyEnd functions to perform identification task. I store all templates in a big file and map to memory (mmap). This way is 100% faster then IdentifyOnSQLiteDatabase.
transtep
 
Posts: 5
Joined: Sep 01, 2016 09:43

Re: SQLite and memory-mapped IO

Postby MartynasV » Apr 27, 2017 09:20

transtep wrote:Hi,

With Biometric 5 SDK, I invoke NMIdentifyStartEx / NMIdentifyNextEx / NMIdentifyEnd functions to perform identification task. I store all templates in a big file and map to memory (mmap). This way is 100% faster then IdentifyOnSQLiteDatabase.

Hello transtep,

Using older API, you control how identification is done and where templates are held, so to ensure high performance you have to load templates to memory and keep them there for fast access yourself(as you did). In newer API (NBiometricClient) all this identification process management is moved inside our API, so when NBiometricClient is initialized, all data is copied from database to local memory for fast access and identification is called with one method: "NBiometricClient.Identify(NSubject)".
Martynas V
Neurotechnology
MartynasV
Neurotechnology
 
Posts: 531
Joined: Feb 27, 2015 11:24
Location: Vilnius, Lithuania

Re: SQLite and memory-mapped IO

Postby mohammadhatami » Aug 29, 2019 07:27

hi
if i use some other method for saving template(like file system/or no sql db/or ...)

when i load template by

NBiometricTask enrollTask = biometricClient.CreateTask(NBiometricOperations.Enroll, null);
enrollTask.Subjects.Add(All Subject);
biometricClient.PerformTask(enrollTask);

to initialize load all template in biometric client in ram

if i call "biometricClient.PerformTask(enrollTask);" once--> is use (all template size)*2 in initialize time
so i need 2* Ram

if i divide all temple to m
and call
biometricClient.PerformTask(enrollTask);
in m steps

its very slow

how can speed up this problem?


i use high performance server
for 2,000,000(1 large face,2 large iris,10 large finger)template
load time near 3 hours!
9 min for read all template
2.51 hours for biometricClient.PerformTask(enrollTask);(5/10/50/500 steps is tested)
mohammadhatami
 
Posts: 2
Joined: Jan 04, 2017 05:12

Re: SQLite and memory-mapped IO

Postby MartynasV » Sep 03, 2019 14:10

mohammadhatami wrote:hi
if i use some other method for saving template(like file system/or no sql db/or ...)

when i load template by

NBiometricTask enrollTask = biometricClient.CreateTask(NBiometricOperations.Enroll, null);
enrollTask.Subjects.Add(All Subject);
biometricClient.PerformTask(enrollTask);

to initialize load all template in biometric client in ram

if i call "biometricClient.PerformTask(enrollTask);" once--> is use (all template size)*2 in initialize time
so i need 2* Ram

if i divide all temple to m
and call
biometricClient.PerformTask(enrollTask);
in m steps

its very slow

how can speed up this problem?


i use high performance server
for 2,000,000(1 large face,2 large iris,10 large finger)template
load time near 3 hours!
9 min for read all template
2.51 hours for biometricClient.PerformTask(enrollTask);(5/10/50/500 steps is tested)

Hello,

Try connecting SQLite database to NBIometriClient(as in "SDK\Tutorials\Biometrics\CS\IdentifyOnSQLiteDatabaseCS" tutorial) and on initialization templates would be loaded into memory, it should be faster than calling enroll.
Martynas V
Neurotechnology
MartynasV
Neurotechnology
 
Posts: 531
Joined: Feb 27, 2015 11:24
Location: Vilnius, Lithuania


Return to Public

Who is online

Users browsing this forum: No registered users and 1 guest

cron