Hamming Distance for Hybrid Search in SQLite

Hamming distance ke metric ea motheo ea ho ts'oana e balang li-bits tse fapaneng lipakeng tsa likhoele tse peli tsa binary, e leng se etsang hore e be o mong oa mekhoa e potlakileng le e sebetsang hantle bakeng sa ho batla moahelani o haufi haholo ho database. Ha e sebelisoa ho SQLite ka meaho e nyalisitsoeng ea patlo, Hamming sebaka se notlolla bokhoni ba patlo ea boemo ba khoebo ntle le ho feta ho li-database tse inehetseng tsa vector.

Hamming Distance ke Eng mme Hobaneng ho le Bohlokoa Bakeng sa Patlisiso ea Bobolokelo?

Bohole ba Hamming bo lekanya palo ea maemo ao likhoele tse peli tsa binary tsa bolelele bo lekanang li fapaneng. Mohlala, likhoele tsa binary 10101100 le 10001101 li na le sebaka sa Hamming sa 2, hobane li fapana ka maemo a mabeli hantle. Maemong a patlo ea polokelongtshedimosetso, palo ena e bonahalang e le bonolo e ba matla a sa tlwaelehang.

Patlisiso ya SQL ya tlwaelo e itshetlehile hodima ho nyallana hantle kapa tlhakisetso ya mongolo o felletseng, e nang le bothata ba ho tshwana ha semantic — ho fumana sephetho se bolelang ntho e le nngwe ho ena le ho arolelana mantswe a tshwanang. Hamming distance borokho ba sekheo sena ka ho sebetsa ho binary hash codes tse nkiloeng ho tse kenyellelitsoeng litaba, e lumellang datha e kang SQLite ho bapisa lirekoto tse limilione ka milliseconds ho sebelisa ts'ebetso ea XOR bitwise.

Metric e hlahisitsoe ke Richard Hamming ka 1950 molemong oa ho lokisa liphoso. Mashome a lilemo hamorao, e ile ea e-ba setsi sa ho khutlisa tlhahisoleseling, haholo lits'ebetsong moo lebelo le leng bohlokoa ho feta ho nepahala ho phethahetseng. Khokahano ea eona ea O(1) papisong ka 'ngoe (ho sebelisa litaelo tsa CPU popcount) e etsa hore e tšoanelehe ka mokhoa o ikhethileng bakeng sa lienjineri tse kentsoeng le tse bobebe tsa polokelo.

Ke Joang Hybrid Search e Kopanyang Hamming Distance le Traditional SQLite Queries?

Patlo ea Hybrid ho SQLite e kopanya maano a mabeli a tlatsetso a ho khutlisa: patlo ea mantsoe a bohlokoa a fokolang (ho sebelisa SQLite's build-in FTS5 full-text search extension) le patlo e teteaneng ea ho tšoana (ho sebelisa sebaka sa Hamming ho li-binary quantized embeddings). Ha ho mokhoa o le mong o lekaneng litlhoko tsa sejoale-joale tsa ho batla.

Tsela e tloaelehileng ea ho batla e sebetsa ka tsela e latelang:

1. Moetso oa ho kenya: Tokomane e 'ngoe le e 'ngoe kapa rekoto e fetoleloa ho vector ea boemo bo holimo ba ho phaphamala ho sebelisoa mofuta oa puo kapa ts'ebetso ea khouto.
2. Binary quantization: The float vector e hatelloa hore e be compact binary hash (mohlala, 64 or 128 bits) ho sebelisoa mekhoa e kang SimHash kapa random projection, e fokotsang litlhoko tsa polokelo haholo.
3. Hamming index storage: The binary hash e bolokoa joalo ka kholomo ea INTEGER kapa BLOB ho SQLite, e nolofalletsang ts'ebetso e potlakileng ka nako ea potso.
4. Liphetho tsa nako ea ho botsa: Ha mosebelisi a fana ka potso, SQLite e bala sebaka sa Hamming ka mokhoa o ikhethileng o sebelisang XOR le popcount, e khutlisetsa bakhethoa ba hlophiloe ka ho tšoana.
5. Score fusion: Liphetho tse tsoang ho Hamming-based semantic search le FTS5 keyword search li kopanngoa ho sebelisoa Reciprocal Rank Fusion (RRF) kapa lintlha tse boima ho hlahisa lenane la ho qetela la maemo.

Katoloso ea SQLite ka likatoloso tse ka jarolloang kapa mesebetsi e kopaneng e etsa hore moaho ona o fihlellehe ntle le ho falla ho sistimi e boima ea polokelo. Sephetho ke mochini oa ho batla o ikemetseng o sebetsang kae kapa kae moo SQLite e sebetsang teng - ho kenyeletsoa lisebelisoa tse kentsoeng, lisebelisoa tsa mohala, le lisebelisoa tse ling.

Key Insight: Patlo ea Binary Hamming ho 64-bit hashes e batla e le 30–50x kapele ho feta cosine ho tšoana ho li-vector tse felletseng tsa float32 tsa boholo bo lekanang. Bakeng sa lits'ebetso tse hlokang "sub-10ms search latency" ho limilione tsa lirekoto ntle le lisebelisoa tse ikhethang, Hamming distance ho SQLite hangata ke eona tsela e nepahetseng ea ho rarolla mathata pakeng tsa ho nepahala le ts'ebetso.

Litšobotsi life tsa tšebetso ea Hamming Search ho SQLite?

SQLite ke faele e le 'ngoe, database e se nang seva, e hlahisang litšitiso le menyetla e ikhethang ea ho kenya tšebetsong lipatlisiso tsa sebaka sa Hamming. Ntle le meaho ea tlhaho ea li-vector indexing joalo ka HNSW kapa IVF (e fumanoang mabenkeleng a ikhethileng a li-vector), SQLite e tšetlehile ho sekeneng sa mohala bakeng sa patlo ea Hamming - empa sena ha se na moeli ho feta kamoo se utloahala.

Palo ea bohole ba 64-bit Hamming e hloka feela XOR e lateloang ke palo ea batho ba bangata (palo ea baahi, lipalo tse behiloeng). Li-CPU tsa sejoale-joale li etsa sena ka taelo e le 'ngoe. Tlhahlobo e felletseng ea li-hashes tse limilione tse 1 tsa 64-bit e fella ka li-milliseconds tse ka bang 5–20 ho hardware ea thepa, e leng ho etsang hore SQLite e sebetse bakeng sa li-dataset tse fihlang ho lirekoto tse limilione tse 'maloa ntle le maqheka a mang a indexing.

Bakeng sa li-datasets tse kholoanyane, ntlafatso ea ts'ebetso e tsoa ho mokhethoa pele ho sefa: ho sebelisa SQLite's WHERE clauses ho tlosa mela ka metadata (mekhahlelo ea matsatsi, lihlopha, likarolo tsa basebelisi) pele u sebelisa sebaka sa Hamming, ho fokotsa boholo ba scan ninete ka litaelo tsa boholo. Mona ke moo meralo e nyalisitsoeng ea patlo e khanyang e le kannete - "sparse keyword filter" e sebetsa e le sefe se potlakileng, 'me sebaka sa Hamming se beha maemo a batho ba setseng.

U Kenya Mosebetsi oa Hamming Distance Joang ho SQLite?

SQLite ha e kenyelletse ts'ebetso ea sebaka sa Hamming, empa C extension API e etsa hore mesebetsi e tloaelehileng ea scalar e be bonolo ho ingolisa. Ho Python o sebelisa sqlite3 mojule, o ka ngolisa tšebetso e lekanyang sebaka sa Hamming pakeng tsa lipalo tse peli:

Ts'ebetso e amohela lintlha tse peli tse felletseng tse emelang li-hashes tsa binary, e kopanya XOR ea bona, ebe e bala likotoana tse behiloeng ka ho sebelisa Python's bin().count('1') kapa mokhoa o potlakileng oa ho qhekella. Hang ha e se e ngolisitsoe, ts'ebetso ena e ba teng ka lipotso tsa SQL joalo ka ts'ebetso efe kapa efe e hahelletsoeng, e nolofalletsang lipotso tse kang ho khetha mela moo sebaka sa Hamming ho ea ho hash ea potso se oelang ka tlase ho moeli, se laetsoeng ke ho nyoloha ho ea ho fumana lipapali tse haufi pele.

Bakeng sa lisebelisoa tsa tlhahiso, ho hlophisa logic ea popcount joalo ka katoloso ea C ho sebelisa SQLite's sqlite3_create_function API e fana ka ts'ebetso e betere ka makhetlo a 10–100 ho feta Python e tolokiloeng, e tlisang patlisiso ea SQLite's Hamming hore e fihlelle li-database tse khethehileng tsa vector bakeng sa mesebetsi e mengata e sebetsang.

Likhoebo li Lokela ho Khetha Neng SQLite Hamming Batlisisa Holim'a Li-database tse Nehetsoeng tsa Vector?

Khetho lipakeng tsa patlo e thehiloeng ho SQLite ea Hamming le li-database tse inehetseng tsa vector joalo ka Pinecone, Weaviate, kapa pgvector e ipapisitse le sekala, ho rarahana ha ts'ebetso, le litšitiso tsa phepelo. Patlisiso ea SQLite Hamming ke khetho e nepahetseng ha bonolo, bonolo, le litšenyehelo li bohlokoa haholo - e leng boemo bo boholo ba lits'ebetso tsa khoebo.

Li-database tsa vector tse inehetseng li hlahisa ts'ebetso e kholo ea ts'ebetso: meaho e arohaneng, latency ea marang-rang, ho rarahana ha khokahano, le litšenyehelo tse ngata ka tekanyo. Bakeng sa lits'ebetso tse sebeletsang mashome a likete ho isa ho limilione tse tlase tsa lirekoto, patlo ea SQLite Hamming e fana ka bohlokoa bo bapisoang ba basebelisi le lisebelisoa tse ling tsa zero. E kopanya index ea hau ea ho batla le data ea ts'ebeliso ea hau, 'me e felise mofuta o felletseng oa mekhoa e sa sebetseng ea sistimi e ajoang.

Lipotso Tse Botsoang Hangata

Na patlisiso ea sebaka sa Hamming e nepahetse ka ho lekana bakeng sa ts'ebeliso ea tlhahiso?

Sebaka se otlolohileng ho tse kenngoeng ka mokhoa oa binary-quantized e rekisa palo e nyane ea ho hopola hantle bakeng sa phaello e kholo ea lebelo. Ha e le hantle, palo ea binary hangata e boloka 90-95% ea boleng ba ho hopola ha lipatlisiso tse tšoanang tsa float32 cosine. Bakeng sa lits'ebetso tse ngata tsa patlo ea khoebo - ho sibolla lihlahisoa, ho khutlisa litokomane, tsebo ea tšehetso ea bareki - khoebo ena e amohelehile, 'me basebelisi ba ke ke ba bona phapang ea boleng ba sephetho.

Na SQLite e ka khona ho bala le ho ngola ka nako e le 'ngoe nakong ea lipatlisiso tsa Hamming?

SQLite e ts'ehetsa ho bala ka nako e le ngoe ka mokhoa oa eona oa WAL (Write-Ahead Logging), e lumellang babali ba bangata ho botsa ka nako e le 'ngoe ntle le ho thibela. Ngola concurrency e na le moeli - SQLite e ngola seriali - empa sena ha se hangata e leng bothata bakeng sa mosebetsi o boima oa ho batla moo ho ngola ho sa baloeng khafetsa. Bakeng sa ts'ebeliso e matla ea ho batla e nyalisitsoeng, mokhoa oa WAL oa SQLite o lekane ka botlalo.

Ke joang binary quantization e amang litlhoko tsa polokelo ha e bapisoa le li-vectors tse phaphametseng?

Poloko ea polokelo e ntle haholo. Ho kenyeletsoa ha 768-dimensional float32 ho hloka li-byte tse 3,072 (3 KB) rekoto ka 'ngoe. 128-bit binary hash ea ho kengoa ho tšoanang e hloka li-byte tse 16 feela - phokotso ea 192x. Bakeng sa pokello ea lintlha tsa lirekoto tse limilione tse 1, sena se bolela phapang lipakeng tsa 3 GB le 16 MB ea polokelo e kentsoeng, ho etsa hore patlo e thehiloeng Hamming e khonehe libakeng tse nang le mohopolo o thata moo polokelo e felletseng e ke keng ea sebetsa.

Ho aha lihlahisoa tse bohlale, tse batlisisoang hantle ke mofuta oa bokhoni bo arohanyang likhoebo tse ntseng li hola ho tse emeng. Mewayzke OS ea khoebo ea bohle-in-one e tšeptjoang ke basebelisi ba fetang 138,000, e fanang ka li-module tsa 207 tse kopantsoeng - ho tloha ho CRM le li-analytics ho ea ho tsamaiso ea litaba le ho feta - ho qala ka $ 19 / khoeli feela. Emisa ho kopanya lisebelisoa tse khaotsoeng 'me u qale ho haha sethaleng se etselitsoeng sekala.

Qala leeto la hao la Mewayz kajeno ho app.mewayz.com 'me u utloe seo tsamaiso ea khoebo e kopaneng e ka se etsetsang sehlopha sa hau.