"Kuenderera mberi batching hakungogadzirisi kubuda - kunogadzirisa mhando yehupfumi yeAI inference. Nekuchengetedza maGPU akabatikana pane iteration granularity pane kukumbira granularity, vashandisi vanowana 5-10 × kushandiswa kwepamusoro kunoshanda kubva kune yakafanana hardware, inova ndiyo imwe huru lever inowanikwa kuderedza per-token kushumira mutengo mu2p02."

Kutumirwa Kwenyika Kwechokwadi Kunoyera Sei Kubudirira Kwekuita?

Benchmark mhedzisiro kubva kuAnyscale, pamwe chete neyakazvimiririra kudhirowa kumhuri dzakawanda dzemodhi muna 2024, zvinogara zvichiratidza kuenderera mberi kwekuburitsa pakati pe23× ne36 × yepamusoro yekubuda kana ichienzaniswa ne naïve static batching pasi pechokwadi chetraffic maitiro. Zvawawana zvinonyanya kutaurwa kana kureba kwechikumbiro kwakakwira - chaizvo mamiriro ezvinhu anotaridza magadzirirwo ehurukuro yeAI apo mibvunzo yemushandisi inotangira kubva kumashoko matatu-matatu kusvika kune akawanda-mapeji mapepa ekutumira.

Latency inotaura imwe nyaya isina kujeka. Nguva-kusvika-yekutanga-chiratidzo inovandudza zvinoshamisa nekuti sisitimu haichamiriri kuti yakazara static batch iungane isati yatanga kufanozadza. Inter-token latency inoramba yakagadzikana pasi pemutoro uri pakati nepakati asi inodzikisira zvakanaka pasi pekuzara kwete kudonha, nekuti mugadziri anoramba achifambira mberi pane ese anoteedzana anoteedzana kunyangwe mutsetse wakura wakadzika. Kune mabhizinesi ari kuvaka chaiyo-nguva AI maficha, iyi yakanaka yekudzikisira curve inonyanya kukosha mune zvekutengesa pane peak throughput manhamba.

Mabhizinesi Angashandisa Sei Masevhisi Anoramba Achiita Mabatiro Anopfuura AI Inference?

Muono wekuvaka kuseri kwe batching inoenderera - torazve zviwanikwa pa granularity yakanakisa uye wozvigashira nekukurumidza pane kumirira kuti chikamu chakaomarara chebasa kuti chipedze - ndiwo musimboti wechero system inodzora mabasa akasiyana. Bhizinesi masisitimu anoshanda akatarisana nedambudziko rakafanana: mabasa ehurefu hwakasiyana-siyana achikwikwidza kugovaniswa kugadzirisa kugona mukati meCRM workflows, kushambadzira otomatiki, analytics mapaipi, uye e-commerce mashandiro.

Mewayz inoshandisa huzivi uhwu pa207-module bhizinesi OS, ichifambisa zvine simba mabasa ekushanda papuratifomu yakabatanidzwa inoshandiswa nemabhizinesi 138,000 pasi rese. Panzvimbo pekumanikidza zvikwata kuti zvimirire kutenderera kwebatch, mitsetse yemvumo yakatevedzana, kana maturusi ekushandisa, Mewayz inogadzira zviitiko zvebhizinesi nguva nenguva - kupa zvakapedzwa zvinobuda nekukasira mumamodule ekudzika nenzira iyo inoenderera batching scheduler inodyisa yakasunungura GPU slots kudzokera kumutsara wekukumbira. Mhedzisiro yacho inoyereka yekuvandudza mashandisirwo ebhizinesi, kwete mabhenji chete.

Mibvunzo Inowanzo bvunzwa

Kuramba vachibatanidza kwakafanana here neshanduko yeBatching muTensorFlow Serving?

Kwete. TensorFlow Serving's dynamic batching inounganidza zvikumbiro kuita mabhechi ehukuru hwakasiyana zvichienderana nenguva windows uye kudzika kwemutsetse, asi ichiri kugadzirisa batch yega yega kubva pakutanga kusvika pakupedzisira. Kuenderera mberi batching kunoshanda padanho rekugadzira tokeni, zvichibvumira kuumbwa kwebatch kuchinja kwese kwemberi. Musiyano wegranularity ndosaka kuenderera mberi batching ichiwana yakakwira zvakanyanya kuburitsa kune autoregressive chizvarwa mitoro chaiyo.

Kuenderera mberi batching kunoda shanduko yemhando yezvivakwa here?

Standard transformer architecture inoda kusandurwa. Kuenderera mberi batching kunoitwa zvachose pane yekushandira layer kuburikidza neshanduko kune inference scheduler, memory maneja, uye kutarisisa kernel. Nekudaro, mamwe magadzirirwo - kunyanya PagedAttention - inoda tsika yeCUDA kernels inotsiva yakajairwa kutarisisa kuita, ndosaka kugadzirwa-giredhi inoenderera mberi batching masisitimu sevLLM uye TensorRT-LLM isiri yekudonhedza-in inotsiva yemasevha-yechinangwa-sevha.

Ndezvipi zvimhingamipinyi zvehardware zvinotadzisa kuenderera mberi kwe batching?

GPU HBM bandwidth uye huwandu hweVRAM huwandu ndizvo zvinonetsa. Yakakura KV cache inoda imwe ndangariro, ichidzikamisa concurrency yakanyanya. Yakakwira-bandwidth yekubatanidza (NVLink, Infiniband) inove yakakosha kune akawanda-GPU deployments uko KV cache inofanirwa kugoverwa pamidziyo yese. Munzvimbo dzakadzorwa nendangariro, hukasha hwehuwandu hweKV cache values (kubva paFP16 kusvika INT8 kana INT4) inodzoreredza huwandu nemutengo wekuderedzwa kudiki kwechokwadi kunogamuchirika kune dzakawanda zvekutengesa zvikumbiro.

Kunyangwe iwe uri kuvaka maAI-powered maficha kana kuronga mashandiro akaomarara ebhizinesi pasangano rako rese, musimboti uripo wakafanana: bvisa nguva yekusaita basa, dzosera simba nguva dzose, uye gadzirisa basa rakawanda nezviwanikwa zvauinazvo. Mewayz anoisa musimboti iwoyo mukuita mumamodule 207 akabatanidzwa - kubva kuCRM uye e-commerce kuenda kuanalytics nekubatana kwechikwata - kutanga pamadhora gumi nepfumbamwe pamwedzi.

Wagadzirira kuita bhizinesi rako zvizere? Tanga kuedza kwako mahara paapp.mewayz.com uone kuti mabhizimisi 138,000 ari kushanda zvakachenjera neMewayz.