Intera ya Nyundo yo gushakisha Hybrid muri SQLite

Inyundo intera ni ishingiro ryibanze rihuza ibara ritandukanye hagati yimigozi ibiri, bigatuma bumwe muburyo bwihuse kandi bunoze bwo gushakisha hafi-umuturanyi ushakisha mububiko. Iyo ushyizwe kuri SQLite unyuze mubushakashatsi bwububiko, intera ya Hamming ifungura ibigo-byo mu rwego rwo gushakisha ubushobozi bwo gushakisha nta mutwe wububiko bwihariye bwabigenewe.

Intera ya Nyundo ni iki kandi ni ukubera iki ari ngombwa gushakisha amakuru?

Intera yo ku nyundo ipima umubare wimyanya ibiri ya binary imirongo yuburebure bungana. Kurugero, imirongo ya binary 10101100 na 10001101 ifite intera ya Nyundo ya 2, kuko itandukanye mumyanya ibiri ya biti. Mubushakashatsi bwububiko, ibi bisa nkibyoroshye kubara biba imbaraga zidasanzwe.

Gakondo SQL ishakisha ishingiye kumyitozo ihuye neza cyangwa yuzuye-yuzuye yerekana indangagaciro, irwana no guhuza ibisobanuro - kubona ibisubizo bivuze ikintu kimwe aho gusangira ijambo ryibanze. Intera yo ku nyundo ikemura iki cyuho ikora kuri binary hash code ikomoka kubintu byashizwemo, bituma data base nka SQLite igereranya miriyoni zanditse muri milisegonda ukoresheje ibikorwa bya XOR bito.

Ibipimo byatangijwe na Richard Hamming mu 1950 murwego rwo gukosora amakosa. Nyuma yimyaka icumi, byabaye intangiriro yo gushakisha amakuru, cyane cyane muri sisitemu aho umuvuduko ufite akamaro kuruta gutondeka neza. Iharurwa ryayo O (1) kugereranya (ukoresheje amabwiriza ya popcount ya CPU) ituma ikwiranye bidasanzwe na moteri yububiko kandi yoroheje.

Nigute Gushakisha Hybrid Bihuza Intera yo Kunyundo nibibazo gakondo bya SQLite?

Ishakisha rya Hybrid muri SQLite rihuza ingamba ebyiri zuzuzanya zo gushakisha: gushakisha ijambo ryibanze rito (ukoresheje SQLite yubatswe muri FTS5 yuzuye yo gushakisha inyandiko yuzuye) hamwe nubushakashatsi bwimbitse (ukoresheje intera ya Nyundo kuri binary numero yashizwemo). Ntabwo inzira yonyine yonyine ihagije kubisabwa bigezweho.

Imiyoboro isanzwe ishakisha imiyoboro ikora kuburyo bukurikira:

Kwiyongera kwa SQLite binyuze mumagambo aremereye cyangwa ibikorwa byakusanyirijwe hamwe bituma iyi myubakire igerwaho utimukiye muri sisitemu iremereye. Igisubizo ni moteri yishakisha yonyine ikoresha ahantu hose SQLite ikora - harimo ibikoresho byashizwemo, porogaramu zigendanwa, hamwe n’ibikorwa byoherejwe.

Ubushishozi bwibanze: Ishakisha rya Binary Inyundo kuri 64-bit hashes hafi 30-50x yihuta kuruta cosine isa na float32 yuzuye yuzuye. Kuri porogaramu zisaba sub-10ms zishakisha ubukererwe miriyoni zinyandiko zidafite ibyuma kabuhariwe, Intera yo ku nyundo muri SQLite akenshi nuburyo bwiza bwogukora ibicuruzwa hagati yubusobanuro nibikorwa.

Ni ibihe bintu biranga imikorere yo gushakisha inyundo muri SQLite?

SQLite ni dosiye imwe, idafite seriveri, ikora imbogamizi zidasanzwe n'amahirwe yo gushyira mu bikorwa intera ya Hamming. Hatariho indangagaciro za kavukire kavukire nka HNSW cyangwa IVF (iboneka mububiko bwabigenewe bwabigenewe), SQLite yishingikiriza kumurongo wo gushakisha gushakisha Inyundo - ariko ibi ntibigabanuka kurenza uko byumvikana.

Kubara intera ya 64-biti Inyundo bisaba XOR gusa ikurikirwa na popcount (kubara abaturage, kubara bits bits). CPU igezweho ikora ibi mumabwiriza amwe. Isuzuma ryuzuye rya miriyoni 1 64-bit hashes ryuzuza hafi ya milisegonda 5-20 kumashanyarazi yibicuruzwa, bigatuma SQLite ifatika kuri datasets zigera kuri miriyoni nyinshi zanditse nta mayeri yinyongera yerekana.

Kuri datasets nini, kunoza imikorere biva mubakandida mbere yo kuyungurura: ukoresheje SQLite AHO ingingo zo gukuraho umurongo ukoresheje metadata (urutonde rwamatariki, ibyiciro, ibice byabakoresha) mbere yo gukoresha intera ya Nyundo, kugabanya ingano ya scan ikurikije amategeko yubunini. Aha niho hubatswe ubushakashatsi bwububiko burabagirana - ijambo rito ryijambo ryungurura rikora nkibyihuta mbere yo kuyungurura, kandi intera ya Hamming yongeye gutondekanya abakandida barokotse.

Nigute Washyira mubikorwa Imikorere ya Nyundo muri SQLite?

SQLite ntabwo ikubiyemo imikorere ya kavukire ya Nyundo, ariko C yaguye ya API ituma ibikorwa bya scalar byabigenewe byoroshye kwiyandikisha. Muri Python ukoresheje sqlite3 module, urashobora kwandikisha imikorere ibara intera ya Nyundo hagati yimibare ibiri:

> Bimaze kwiyandikisha, iyi mikorere iboneka mubibazo bya SQL nkibikorwa byose byubatswe, bigushoboza ibibazo nko guhitamo umurongo aho intera ya Nyundo kugeza kubibazo hash kugwa munsi yumuryango, byateganijwe nintera izamuka kugirango igarure imikino yegeranye mbere.

Kubikorwa byoherejwe, gukusanya logique ya popcount nkumugereka wa C ukoresheje SQLite ya sqlite3_create_imikorere API itanga umusaruro mwiza wa 10–100x kuruta gusobanurwa na Python, kuzana SQLite ya Hamming mububiko bwihariye bwa vectori kubikorwa byinshi bifatika.

Ni ryari Abashoramari Bagomba Guhitamo SQLite Inyundo Ishakisha Ububiko Bweguriwe Ububiko?

Guhitamo hagati ya SQLite ishingiye kubushakashatsi bwa Hamming hamwe nububiko bwihariye bwa vector nka Pinecone, Weaviate, cyangwa pgvector biterwa nubunini, imikorere igoye, hamwe nimbogamizi zoherejwe. SQLite Inyundo ishakisha nuguhitamo kwiza mugihe ubworoherane, ubwikorezi, hamwe nigiciro cyinshi - nicyo kibazo kubenshi mubisabwa mubucuruzi.

Ububiko bwihariye bwabigenewe butangiza ibikorwa byingenzi: ibikorwa remezo bitandukanye, ubukererwe bwurusobe, guhuza ibikorwa, hamwe nigiciro kinini mubipimo. Kuri porogaramu zitanga ibihumbi mirongo kugeza kuri miriyoni zanditse, ishakisha rya SQLite Hamming ritanga igereranya ryabakoresha-bareba hamwe nibikorwa remezo bya zeru. Ihuza urutonde rwishakisha hamwe namakuru yawe yo gusaba, ikuraho icyiciro cyose cya sisitemu zagabanijwe uburyo bwo gutsindwa.

Ibibazo bikunze kubazwa

Gushakisha intera ya Hamming birahagije kugirango ushakishe umusaruro?

Inyundo intera kuri binary-ingano yububiko iragurisha umubare muto wo kwibuka neza kubwinyungu nini yunguka. Mubimenyerezo, ingano ya binary isanzwe igumana 90-95% yubwiza bwibutsa bwa float32 cosine ishakisha. Kubikorwa byinshi byo gushakisha ubucuruzi - kuvumbura ibicuruzwa, kugarura inyandiko, gushingira ubumenyi bwibanze kubakiriya - uku gucuruza biremewe rwose, kandi abakoresha ntibashobora kubona itandukaniro mubisubizo byubuziranenge.

SQLite irashobora gukemura icyarimwe gusoma no kwandika mugihe cyo gushakisha Inyundo?

SQLite ishyigikira ibyasomwe icyarimwe binyuze muburyo bwa WAL (Andika-Imbere yo Kwinjira), bituma abasomyi benshi babaza icyarimwe nta guhagarika. Andika guhuza ni ntarengwa - SQLite ikurikirana yandika - ariko ibi ntibikunze kuba icyuho cyo gushakisha-akazi karemereye aho kwandika ari gake ugereranije no gusoma. Kubisoma-byibanda cyane kubushakashatsi, uburyo bwa SQLite bwa WAL burahagije rwose.

Ni gute ingano ya binarike igira ingaruka kububiko ugereranije nu kureremba kureremba?

Kuzigama kubika biratangaje. Ubusanzwe 768-yuzuye ya float32 gushiramo bisaba 3,072 bytes (3 KB) kuri buri nyandiko. 128-bit binary hash yo gushiramo kimwe bisaba bytes 16 gusa - kugabanuka 192x. Kuri dataset yinyandiko ya miriyoni 1, ibi bivuze itandukaniro riri hagati ya 3 GB na 16 MB yo gushyiramo ububiko, bigatuma ubushakashatsi bushingiye kuri Hamming bushoboka mubidukikije bitarimo kwibuka aho ububiko bwuzuye bureremba bwaba budakwiye.

---

Kubaka ibicuruzwa byubwenge, bishakishwa nuburyo bwubushobozi butandukanya ubucuruzi butera imbere nibihagarara. Mewayz niyo yose-imwe-imwe yubucuruzi OS yizewe nabakoresha barenga 138.000, itanga module 207 ihuriweho - kuva CRM hamwe nisesengura kugeza gucunga ibikubiyemo ndetse no hanze yacyo - guhera kumadorari 19 / ukwezi. Hagarika kudoda hamwe ibikoresho byaciwe hanyuma utangire kubaka kumurongo wagenewe igipimo.

Tangira urugendo rwawe rwa Mewayz uyumunsi kuri app.mewayz.com hanyuma wibonere icyo sisitemu yubucuruzi ihuriweho nukuri ishobora gukorera ikipe yawe.