Nisa Hamma don Neman Haɓaka a cikin SQLite

Nisan hamma shine ma'aunin kamanni na tushe wanda ke ƙidayar ragi daban-daban tsakanin igiyoyin binary biyu, yana mai da shi ɗaya daga cikin mafi sauri kuma mafi inganci hanyoyin don neman kusanci kusa da makwabci a cikin bayanan bayanai. Lokacin da aka yi amfani da shi zuwa SQLite ta hanyar gine-ginen bincike na gauraye, nisan Hamming yana buɗe damar binciken mahimmin darajar kasuwanci ba tare da keɓancewar bayanan bayanan vector ba.

Mene ne Hamming Distance kuma Me yasa Yayi Mahimmanci ga Binciken Database?

Nisan Hamming yana auna adadin matsayi inda igiyoyin binary guda biyu masu tsayi daidai suka bambanta. Misali, igiyoyin binary 10101100 da 10001101 suna da tazarar Hamming na 2, saboda sun bambanta a daidai matsayi biyu. A cikin mahallin binciken bayanai, wannan lissafin da ake ganin yana da sauƙi ya zama mai ƙarfi sosai.

Binciken SQL na al'ada ya dogara da ainihin ma'auni ko cikakkun bayanai na rubutu, wanda ke gwagwarmaya tare da kamancen ma'anar ma'anar - gano sakamakon da ma'ana abu ɗaya ne maimakon raba kalmomi iri ɗaya. Tsayar da nisa yana cike wannan gibin ta hanyar aiki akan lambobin hash na binary da aka samo daga abubuwan da aka haɗa, ba da damar bayanai kamar SQLite don kwatanta miliyoyin bayanai a cikin millise seconds ta amfani da ayyukan XOR na bitwise.

Richard Hamming ne ya gabatar da ma'aunin a cikin 1950 a cikin mahallin lambobi masu gyara kuskure. Shekaru da yawa bayan haka, ya zama tsakiya ga maido da bayanai, musamman a cikin tsarin da saurin ya fi dacewa. Ƙididdigar ta O(1) a kowace kwatancen (ta amfani da umarnin popcount CPU) ya sa ta dace ta musamman don injunan bayanai masu nauyi da nauyi.

Ta Yaya Neman Haɗaɗɗen Haɗa Haɗin Nisa tare da Tambayoyin SQLite na Gargajiya?

Binciken Haɓakawa a cikin SQLite yana haɗa dabarun dawo da madaidaitan guda biyu: Binciken mahimmin kalmomi (ta amfani da ginanniyar ginanniyar binciken SQLite a cikin FTS5) da kuma binciken kamanni mai yawa (amfani da nisan Hamming akan abubuwan ƙididdigewa na binary). Babu wata hanya kaɗai da ta isa ga buƙatun neman zamani.

Wani bututun bincike na yau da kullun yana aiki kamar haka:

1. Ƙirƙirar ƙira: Kowace takarda ko rikodin ana jujjuya su zuwa babban madaidaicin ma'aunin iyo ta amfani da ƙirar harshe ko aikin ɓoyewa.
2. Kididdigar binary: Ana matse vector mai iyo a cikin ƙaramin hash na binary (misali, 64 ko 128 bits) ta amfani da dabaru kamar SimHash ko tsinkayar bazuwar, yana rage buƙatun ajiya sosai.
3. Hamming index ma'ajiyar: Ana adana hash ɗin binary azaman ginshiƙin INTEGER ko BLOB a cikin SQLite, yana ba da damar ayyukan bitwise cikin sauri a lokacin tambaya.
4. Maki-maki-lokaci: Lokacin da mai amfani ya ƙaddamar da tambaya, SQLite yana ƙididdige nisa ta hanyar aikin sikeli na al'ada ta amfani da XOR da popcount, maido da ƴan takarar da aka jera su ta ɗan kamanni.
5. Fusion Score: Sakamako daga bincike na tushen Hamming da binciken kalmomin FTS5 an haɗa su ta amfani da Reciprocal Rank Fusion (RRF) ko maki mai nauyi don samar da jeri na ƙarshe.

Ƙarfafawar SQLite ta hanyar haɓakawa mai ɗorewa ko ayyukan da aka haɗa a ciki yana sa wannan gine-ginen ya sami nasara ba tare da ƙaura zuwa tsarin bayanai mafi nauyi ba. Sakamakon shine injin bincike mai ƙunshe da kansa wanda ke gudana a duk inda SQLite ke gudana - gami da na'urorin da aka haɗa, aikace-aikacen hannu, da tura baki.

Maɓalli Maɓalli: Binciken Binary Hamming akan hashes 64-bit yana da sauri kusan 30–50x fiye da kamancen cosine akan cikakken float32 vectors na daidai girma. Don aikace-aikacen da ke buƙatar ƙaramar bincike na sub-10ms a cikin miliyoyin bayanai ba tare da na'ura na musamman ba, Hamming nesa a cikin SQLite galibi shine mafi kyawun cinikin injiniya tsakanin daidaito da aiki.

Mene ne Halayen Ayyuka na Binciken Hamming a cikin SQLite?

SQLite fayil ne guda ɗaya, ma'ajin bayanai mara sabar, wanda ke haifar da maƙasudi na musamman da dama don aiwatar da binciken nesa na Hamming. Ba tare da sifofi na asali ba kamar HNSW ko IVF (ana samun su a cikin shagunan da aka keɓe), SQLite ya dogara da sikanin linzamin kwamfuta don binciken Hamming - amma wannan baya iyakancewa fiye da yadda ake ji.

Ƙididdigar nisa na 64-bit Hamming yana buƙatar XOR kawai wanda ke biye da yawan jama'a (ƙidaya yawan jama'a, ƙidayar saiti). CPUs na zamani suna aiwatar da wannan a cikin umarni ɗaya. Cikakken sikanin linzamin kwamfuta na miliyan 1 hashes 64-bit yana ƙarewa a cikin kusan mil 5-20 millisecond akan kayan masarufi, yana yin SQLite mai amfani don tsara bayanai har zuwa rikodin miliyan da yawa ba tare da ƙarin dabaru ba.

Don manyan bayanai, ingantattun ayyuka sun fito ne daga ɗan takara kafin tacewa: ta amfani da SQLite's WHERE jumla don kawar da layuka ta metadata (jerin kwanan wata, nau'ikan, sassan mai amfani) kafin amfani da nisan Hamming, rage ingantaccen girman sikirin ta umarni na girma. Wannan shi ne inda gine-ginen bincike na matasan ke haskakawa da gaske - matattarar maƙasudin mahimmin kalmomi suna aiki azaman mai saurin tacewa, kuma nisan Hamming ya sake sanya ƴan takarar da suka tsira.

Ta Yaya Kuke Aiwatar da Aikin Nisa a SQLite?

SQLite bai haɗa da aikin nesa na Hamming ba, amma API ɗin tsawo na C yana sa ayyukan sikeli na al'ada kai tsaye don yin rajista. A cikin Python ta amfani da tsarin sqlite3, zaku iya yin rijistar aikin da ke ƙididdige nisa tsakanin lamba biyu:

Aikin yana karɓar gardama guda biyu masu wakiltar hashes na binary, suna ƙididdige XOR ɗin su, sannan a ƙidaya saiti ta hanyar amfani da Python's bin().count('1') ko kuma hanyar sarrafa bit ɗin sauri. Da zarar an yi rajista, wannan aikin yana samuwa a cikin tambayoyin SQL kamar kowane aikin da aka gina a ciki, yana ba da damar tambayoyi kamar zabar layuka inda nisan Hamming zuwa zaren tambaya ya faɗi ƙasa da mashigin, ana ba da umarni ta hanyar hawan nesa don dawo da mafi kusancin ashana farko.

Don tura kayan aiki, harhada ma'anar popcount a matsayin tsawo na C ta amfani da SQLite's sqlite3_create_function API yana samar da 10-100x mafi kyawun aiki fiye da fassarar Python, yana kawo binciken SQLite's Hamming a cikin isar da mahimman bayanai na vector don yawancin ayyuka masu amfani.

Yaushe Ya Kamata Kasuwancin Zaba SQLite Hamming Neman Akan Sadadden Bayanan Bayanai na Vector?

Zaɓin tsakanin binciken Hamming na tushen SQLite da keɓaɓɓun bayanan bayanai kamar Pinecone, Weaviate, ko pgvector ya dogara da ma'auni, rikitarwar aiki, da ƙuntatawa na turawa. Binciken SQLite Hamming shine zaɓin da ya dace lokacin da sauƙi, ɗaukar nauyi, da al'amuran farashi mafi yawa - wanda shine yanayin mafi yawan aikace-aikacen kasuwanci.

Kaddamar da bayanan bayanan vector suna gabatar da babban aikin aiki: keɓancewar kayan aikin, jinkirin hanyar sadarwa, haɗaɗɗiyar aiki tare, da ƙima mai ƙima a sikeli. Don aikace-aikacen da ke ba da dubun-dubatar dubbai zuwa ƙananan miliyoyin bayanan, binciken SQLite Hamming yana ba da dacewa mai kama da mai amfani tare da ƙarin abubuwan more rayuwa. Yana haɗa ma'aunin bincikenku tare da bayanan aikace-aikacenku, yana kawar da duka nau'ikan hanyoyin gazawar tsarin rarrabawa.

Tambayoyin da ake yawan yi

Shin binciken nisan Hamming daidai ne don aikace-aikacen neman samarwa?

Nisa mai nisa akan abubuwan ƙididdigewa na binary suna cinikin ɗan ƙaramin adadin tunowa don ɗimbin ribar gudu. A aikace, ƙididdigewa na binary yawanci yana riƙe 90-95% na ƙimar tunawa na cikakken binciken kamanni na float32 cosine. Don yawancin aikace-aikacen neman kasuwanci - gano samfur, dawo da daftarin aiki, tushen ilimin goyon bayan abokin ciniki - wannan cinikin gaba ɗaya karɓuwa ne, kuma masu amfani ba za su iya fahimtar bambancin ingancin sakamako ba.

Shin SQLite na iya sarrafa karantawa da rubutu lokaci guda yayin tambayoyin neman Hamming?

SQLite yana goyan bayan karantawa lokaci guda ta hanyar WAL (Rubuta-gaba da Logging), yana bawa masu karatu da yawa damar yin tambaya lokaci guda ba tare da toshewa ba. Rubutun ƙididdiga yana da iyaka - SQLite serializes ya rubuta - amma wannan ba kasafai ba ne ƙugiya don ayyukan bincike-nauyin aiki inda rubuce-rubuce ba su da yawa game da karantawa. Don aikace-aikacen bincike na haɗaɗɗun karatu, yanayin WAL na SQLite ya wadatar gaba ɗaya.

Ta yaya ƙididdigewa na binary ke shafar buƙatun ajiya idan aka kwatanta da na'urori masu iyo?

Ajiye ajiya yana da ban mamaki. Ainihin 768-dimensional float32 sakawa yana buƙatar 3,072 bytes (3 KB) kowane rikodin. Hash ɗin binary 128-bit na haɗawa iri ɗaya yana buƙatar kawai 16 bytes - raguwa 192x. Don bayanan bayanan miliyan 1, wannan yana nufin bambanci tsakanin 3 GB da 16 MB na ma'ajiyar sakawa, yin bincike na tushen Hamming mai yuwuwa a cikin wuraren da ke da ƙayyadaddun ƙwaƙwalwar ajiya inda cikakken ma'ajin iyo ba zai yi tasiri ba.

---

Gina samfuran wayo, da za a iya nema shine ainihin irin ƙarfin da ke raba kasuwancin haɓaka da waɗanda suka tsaya cik. Mewayzshine duk-in-daya kasuwanci OS dogara a kan 138,000 masu amfani, bayar da 207 hadedde kayayyaki - daga CRM da nazari zuwa abun ciki management da kuma bayan - farawa a kawai $19 / watan. Dakatar da dinke kayan aikin da aka katse tare kuma fara gini akan dandalin da aka ƙera don sikeli.

Fara tafiyar Mewayz yau a app.mewayz.com kuma ku fuskanci abin da haƙiƙanin tsarin aiki na kasuwanci zai iya yi wa ƙungiyar ku.