Hacker News

Ferret-UI Lite: Ebyokuyiga okuva mu kuzimba ba agenti ba GUI abatono ku kyuma

Ebiteeso

14 min read Via machinelearning.apple.com

Mewayz Team

Editorial Team

Hacker News

Okulinnya kwa Ba agenti ba GUI ku kyuma: Ensalo Empya mu Nkolagana y’Omuntu ne Kompyuta

Okumala emyaka mingi, enkola esinga obunene ey’enkolagana ya pulogulaamu esigadde nga tekyukakyuka mu bukakanyavu: omuntu asoma screen, atambuza cursor, anyiga button, era alinda okuddibwamu. Loop eno — perceive, decide, act — ebadde etegeeza kompyuta okuva graphical desktop eyasooka lwe yalabika mu myaka gya 1970. Naye enkyukakyuka esirifu egenda mu maaso. Abanoonyereza ne bayinginiya bazimba ebikozesebwa ebitonotono, ebikola obulungi ebya AI ebisobola okutegeera, okukubaganya ebirowoozo ku, n’okukola munda mu nkolagana z’abakozesa ez’ebifaananyi zonna ku kyuma, awatali kweraliikirira kwa kusirika, ssente, oba okweraliikirira okw’ekyama okw’okuteebereza okwesigamiziddwa ku kire. Eby’okuyiga ebiva mu pulojekiti zino biddamu okukola engeri gye tulowooza ku pulogulaamu ez’amagezi, okukola mu ngeri ey’obwengula, n’ebiseera eby’omu maaso eby’ebikozesebwa mu bizinensi.

Okukola compact GUI agents — models nga Apple's Ferret-UI ne lighter counterparts — eraga ekintu ekizito: teweetaaga massive language model okutegeera screen. Okwetaaga enzimba entuufu, data entuufu ey’okutendekebwa, n’okwewaayo okutaliimu kisa eri okukola obulungi emirimu egy’enjawulo. Enkola zino bwe zikula, zitandise okukyusa engeri bizinensi gye zikwataganamu n’emikutu gyazo egya pulogulaamu, ne ziggulawo ebisoboka edda ebyali bya ssaayansi byokka.

Lwaki Models Ezitazitowa ze Zimenyawo Entuufu

Waliwo omuze mu mboozi ya AI okwenkanya obusobozi n’ekipimo. Ebika ebinene, endowooza bw’egamba, bikolwa ebigezi. Naye ku ba agenti ba GUI — enkola ezirina okutegeera ensengeka z’omutendera gwa pikseli, okusengejja ebintu ebikwatagana, n’okukola emirimu egy’emitendera mingi mu nkola enzibu — okubala paramita embisi tekikulu nnyo okusinga obutuufu bw’ekifo n’obutuufu bw’oku ttaka. Omuze gwa paramita obuwumbi 7 ogusobola okwesigika okukuba ku bbaatuuni entuufu mu nkolagana y’essimu gusinga omuze ogw’obuwumbi 70 ogwa paramita generalist alowooza ku bifo bya elementi.

Okunoonyereza mu bikolwa ebitono ebya GUI ku kyuma kulaga bulijjo nti okulongoosa okugendereddwamu ku data eyeetongodde ku UI kuvaamu okulongoosa okw’amaanyi okusinga okukubiriza kwokka enkola ennene ey’omusingi. Models abatendekeddwa ku screenshots eziriko annotated, element hierarchies, ne interaction traces bayiga grammar ey’okulaba ey’enjawulo mu musingi okusinga abo abatendekeddwa ku biwandiiko bya yintaneeti n’ebifaananyi eby’obutonde. Bakulaakulanya okutegeera ku affordances — ebiyinza okukubwa, swiped, scrolled, oba typed — nti generalist models simply tezibula.

Ebiva mu nkola bikulu nnyo. Omuze ogukola ku ssimu ya ssimu ekola ku busimu gusobola okuyamba abakozesa mu kiseera ekituufu, okuyiga okuva mu nkola z’enkolagana y’omu kitundu, n’okukola mu mbeera ezitaliimu yintaneeti. Ku mbeera z’ebitongole nga data y’ebyensimbi enzibu, ebiwandiiko bya HR, oba amawulire agakwata ku kasitoma gabeera munda mu nkolagana za pulogulaamu, okuteebereza ku kyuma si kirungi-okubeera nabyo — kyetaagisa okugoberera.

Emisomo gy’Ebizimbe Egikyusa Mu butuufu

Okuzimba agenti wa GUI asobola ku mutendera omutono kyetaagisa okusalawo mu by’okuzimba okwawukana ennyo ku dizayini y’ekyokulabirako ky’olulimi lw’okwolesebwa olwa bulijjo. Eby’okuyiga ebiwerako bivuddeyo obutakyukakyuka mu ttiimu z’abanoonyereza ezikola ku kizibu kino.

Ekisooka, okukwasaganya okukiikirira ensonga nnyingi nnyo. Ba agenti ba GUI abaasooka baalwana kubanga basikira ensonga z’ekifo okuva mu bikolwa ebyatendekebwa okunnyonnyola ebifo okusinga okukolagana nabyo. Model egamba nti "waliwo button ya bbulu mu kitundu ekya wansi ku ddyo ku screen" terina mugaso mu automation. Omuze oguzzaayo enkwatagana eza bulijjo n’obutuufu bwa sub-pixel — era nga gukikola mu ngeri eyesigika mu kusalawo kwa screen ez’enjawulo, ensengeka za DPI, n’emiramwa gya OS — kya mugaso mu butuufu. Okukyuka okuva ku bifulumizibwa mu kifo ebinnyonnyola okudda ku bikolebwa kyali kyetaagisa okuddamu okulowooza ku ngeri emitwe gy’oku ttaka gye gitendekebwamu n’okwekenneenya.

Ekyokubiri, encoding emanyi ensengeka y'ebifo etereeza nnyo omulimu. Enkolagana z’enkola ez’omulembe si bifaananyi bipapajjo — biba bisenge bya biyumba eby’ebintu, enkalala, modals, n’ebintu ebikwatagana. Ebikozesebwa ebisobola okutuuka ku muti gw’okutuuka oba okulaba ensengeka y’ebintu ku mabbali g’ekifaananyi ekifulumiziddwa bikola bulungi nnyo ku mirimu egy’okutambulirako egy’amaanyi okusinga egyo egikola okuva ku ppikisi zokka. Eno y’ensonga lwaki ba agenti ba GUI ku kyuma batera okukozesa APIs z’okutuuka ku musingi nga akabonero akakwatagana mu kiseera kyombi eky’okutendekebwa n’okuteebereza.

Ekyokusatu, okuvunda kw’emirimu kulina okuzimbibwa mu nsengeka y’ebifulumizibwa mu muze. Mu kifo ky’okukola enteekateeka y’ebikolwa emu ey’ekintu kimu, ba agenti ba GUI abakola bafulumya ensengeka z’emirimu emitonotono egy’ensengeka n’ebifo eby’okukebera eby’olwatu. Kino kibasobozesa okuddamu okuva mu nsobi wakati mu mulimu — obusobozi obwetaagisa mu nkola za bizinensi entuufu nga okunyiga obubi kuyinza okuleeta enkyukakyuka mu mbeera ezitagenderere.

Ekizibu kya Data: Lwaki Okutendeka Ba Agent ba GUI Kizibu Ekyenjawulo

Ebikozesebwa mu nnimi biganyulwa mu nkola ya yintaneeti mu bukulu etaliiko kkomo ey’ebiwandiiko ebiwandiikiddwa abantu. Abakola ebifaananyi (vision models) basobola okutendekebwa ku buwumbi n’obuwumbi bw’ebifaananyi ebiwandiikiddwako obubonero. Ba agenti ba GUI tebalina kyabugagga kyenkanankana. Enkolagana z’enkola eza kaseera buseera, za bwannannyini, era za njawulo nnyo — olutimbe lw’okusasula mu nkola emu eya SaaS tegabana kumpi kintu kyonna mu kulaba ne dashiboodi ya CRM mu ndala, ne bwe kiba nti byombi bikola emirimu egy’enjawulo.

Ttiimu z’okunoonyereza ezisinga okukola obulungi kino zikoze ku nsonga eno nga ziyita mu kukola data mu ngeri ey’ekikugu (synthetic data generation) ku mutendera. Nga bakozesa ebikozesebwa n’enkola z’okugezesa ez’obwengula, okukwata emitendera gy’enkolagana, n’okuzigatta n’ennyonnyola z’emirimu gy’olulimi olw’obutonde, abanoonyereza basobola okukola obukadde n’obukadde bw’ebyokulabirako bya UI ebiriko obugambo obutonotono. Okusoomoozebwa kwe kulaba ng’okubikka: pulogulaamu za bizinensi ekwata ku buli kimu okuva ku ERP z’ebitongole ezirina data ya tabular enzito okutuuka ku bikozesebwa ebisooka ku ssimu nga birina okutambulira okwesigamiziddwa ku bubonero, era model etendekeddwa ku domain emu eyinza okulemererwa mu ngeri ey’akatyabaga mu ndala.

"Ba agenti ba GUI abasinga obusobozi si be batendekebwa ku data ezisinga obungi — be batendekebwa ku data ezisinga diverse. Obuzibu bw'enkolagana bukola ku bugazi bwa domain, so si kubala screen."

nga bwe kiri

Okutegeera kuno kunyigirizza ttiimu okutuuka ku cross-application generalization benchmarks ezeekenneenya enkola ya agent mu software yonna etaalabwako emabegako. Agent wa GUI afuna obubonero obutuukiridde ku kugabanya kwe okw’okutendekebwa naye n’alemererwa ku nkola empya si yeetegefu kukola. Omutindo gwa zaabu kwe kumaliriza emirimu egya zero-shot — obusobozi okutambulira mu nkolagana gy’otomanyi nga okozesa okulagira kw’olulimi olw’obutonde kwokka n’okutunuulira embeera y’olutimbe eriwo kati.

Okwekuuma, Okusirika, n'enkizo ku kyuma mu mbeera za bizinensi

Ensonga ya bizinensi ya ba agenti ba GUI ku kyuma esukka obusobozi obulongoofu. Ebirungi bisatu ebikwatagana bifuula okuteebereza okw’ekitundu okuwaliriza okuteekebwa mu nkola kw’ebitongole:

💡 DID YOU KNOW?

Mewayz replaces 8+ business tools in one platform

CRM · Invoicing · HR · Projects · Booking · eCommerce · POS · Analytics. Free forever plan available.

Start Free →
  • Obufuzi bwa data: Ebifaananyi eby’oku ssirini bya pulogulaamu za bizinensi biyinza okubaamu ebikwata ku bakasitoma ebikulu, ebiwandiiko by’ebyensimbi, oba ebikwata ku bakozi ssekinnoomu. Okusindika ebifaananyi bino ku API y’ekire kireeta okubikkulwa kw’okulungamya wansi w’enkola nga GDPR, HIPAA, ne SOC 2. Okukola ku kyuma kukuuma data ezirabika ezikwatagana munda mu bbanga ly’obukuumi.
  • Okusirika kw’okuddamu: Ekitongole kya GUI ekyetaaga okugenda n’okudda ku nkomerero y’okuteebereza ekire tekisobola kukola ku sipiidi y’enkolagana y’abantu. Ebikozesebwa ku kyuma biddamu mu makumi ga milisekondi, okusobozesa enkola z’emirimu ez’ekirungo ekikulu ennyo eziwulira nga za nzaaliranwa okusinga ez’ebyuma.
  • Obusobozi obutali ku mutimbagano: Abakozi mu nnimiro, abakola ku by’obulamu, n’abaddukanya eby’okutambuza ebintu batera okukola mu mbeera ezirina omukutu oguteesigika. Omuyambi wa AI eyeetaaga yintaneeti okukola si kintu kya bizinensi ekyesigika — buvunaanyizibwa.
  • Okuteebereza omuwendo: Ekire inference costs scale n'enkozesa. Ku muyambi wa agenti ayinza okukola ku bikumi n’ebikumi by’ebifaananyi eby’oku ssirini buli lutuula lw’omukozesa, emiwendo gya buli kabonero gifuuka egy’ebyenfuna ebikugira ku mutendera. Fixed hardware amortization esinga kuteebereza ku CFOs modeling AI infrastructure costs.

Ebirungi bino bivuga ebbidde ly'okuteeka ssente mu edge AI accelerators okubuna hardware stack. Apple’s Neural Engine, Qualcomm’s Hexagon, ne Google’s Tensor chips zonna zirongooseddwa okusobola okukola matrix eziwanirira enkola z’olulimi olw’okulaba. Enkola ya hardware ya ba agenti ba GUI ku kyuma ekula mangu, era ensengekera za software zigoberera.

Kino Kitegeeza ku Nkola za Sofutiweya za Bizinensi Enzibu

Ebikwata ku nkola za bizinensi eza modulo binene nnyo. Lowooza ku nkola y’emirimu gya kkampuni ekula ng’ekozesa OS ya bizinensi enzijuvu ekwata ku CRM, invoice, payroll, HR, fleet management, n’okwekenneenya — 207 distinct functional modules, mu platform nga Mewayz. Ku mukozi omupya ayingirira, oba maneja atatera kuyingira mu modulo ezimu, okutambulira mu nkolagana z’atamanyi kizibu kya ddala. Ebisale by’okutendekebwa bya ddala. Tikiti z’obuyambi za bbeeyi. Ensobi mu nkola y’emirimu mu nsasula oba invoice zirina ebivaamu wansi ebisukka wala okunyiga okukyamu omulundi gumu.

Omukungu wa GUI asobola ku kyuma akyusa kalkulasi eno yonna. Mu kifo ky’omukozesa omupya okuyiga wa w’ayinza okusanga enkola y’emirimu gy’okukkiriza okuwummula oba engeri y’okusengeka ekifaananyi kya invoice ekiddirira, bannyonnyola ekigendererwa kyabwe mu lulimi olutegeerekeka era agenti n’atambulira mu nkola ku lwabwe. Kino si screen-scraping automation — bwe buyambi obwa nnamaddala, obumanyi embeera obutuukagana n’embeera y’enkolagana, ekwata emisango gy’empenda, era esaba okunnyonnyolwa ng’omulimu tegutegeerekeka.

Enzimba ya Mewayz eya modular naddala etuukira bulungi ku nkola eno. Olw’okuba buli modulo erina olulimi lw’okukola dizayini olukwatagana n’obuwanvu bw’emirimu obutegeerekese obulungi, agenti wa GUI atendekeddwa ku nkolagana ya Mewayz asobola okukola ebikiikirira ebinywevu, ebikyusibwakyusibwa eby’engeri z’enkolagana eza bulijjo — okukakasa okubuuka, okukkiriza okusasula, okulongoosa payipu za CRM — n’okubikozesa mu ngeri eyesigika mu bugazi bw’omukutu mu bujjuvu. Abakozesa 138,000 ku mukutu guno bonna awamu bakiikirira enjawulo ennene ennyo ey’enkola z’emirimu, embeera z’okukozesa, n’emisono gy’enkolagana, nga kino kye kika kyennyini eky’akabonero k’okutendekebwa okw’enjawulo akafulumya ba agenti abasobola, abasobola okugatta.

Okukola Sofutiweya nga tulina ebirowoozo bya Agent-Readiness

Ekimu ku by’okuyiga ebikulu ebiva mu kunoonyereza ku ba agenti ba GUI kwe kuba nti software ezikoleddwa abakozesa abantu ne software ezikoleddwa abakozesa agent si kintu kimu. Interfaces optimized for visual aesthetics — gradients, animations, overlapping layers, custom rendered components — zitera okuba enzibu eri ba agenti okusengejja okusinga ezo ezikoleddwa nga zitunuulidde okutuukako. Okukwatagana kuno wakati wa dizayini esooka okutuukirirwa n’okukola dizayini eyeetegefu agenti y’emu ku nkulaakulana ezisinga okunyumira mu mulimu guno.

Ttiimu za pulogulaamu ezilowooza mu maaso zitandise okuyingiza "agent legibility" mu nkola zaabwe ez'okukola dizayini. Kino kitegeeza:

  1. Okukakasa nti ebintu ebikwatagana birina ebimanyisa eby’enjawulo, ebinywevu ebituukirirwa nga biyita mu muti gw’okutuuka
  2. Okukuuma obusobozi bw’okulaba obutakyukakyuka mu mbeera z’enkolagana okusinga okwesigama ku nkyukakyuka z’embeera ezisinziira ku bifaananyi ebirina obulamu
  3. Okuwa emboozi ezikakasa ezitegekeddwa ku bikolwa ebivaamu ebinene — okukkiriza, okusazaamu, okuwaayo ssente — ebiwa ba agenti ebifo eby’obutonde eby’okukebera
  4. Okubikkula enkolagana enzito ezigenderera emirimu ezisobozesa ba agenti okutambulira butereevu mu mbeera z’enkolagana ezikwatagana awatali kutambula mu mutendera
  5. Okuwandiika metadata y’enkolagana eyinza okukozesebwa okukola data y’okutendekebwa okw’obutonde (synthetic training data) okusobola okulongoosa obulungi ekitongole ekikwata ku domain

Emikutu egiteeka ssente mu bifo bino eby’ebizimbe ennaku zino gizimba enkizo ey’amaanyi mu kuvuganya. Nga ba agenti ba GUI bava ku prototypes z’okunoonyereza okudda ku bikozesebwa mu kukola mu myaka ebiri oba esatu egijja, software esomebwa agent ejja kuwa obumanyirivu bwa agent obulungi ennyo okusinga software ekwata obuyambi bwa AI nga afterthought bolted onto an existing interface paradigm.

Ekkubo eriri mu maaso: Okuva ku Bayambi okutuuka ku Ba agenti abakola emirimu egy’obwetwaze

Entambula y’okunoonyereza ku agenti wa GUI ku kyuma esonga ku biseera eby’omu maaso ng’ensalo wakati w’okukola kw’omuntu n’okutuukiriza mu ngeri ey’obwengula efuuka ya mazima. Ba agenti ba leero basobola okwesigika okumaliriza emirimu gumu, egyategeerekese obulungi — okugenda ku screen eyeetongodde, okujjuza foomu, okuggya omuwendo okuva ku dashboard. Ba agenti b’enkya bajja kuddukanya enkola z’emirimu ez’entuula nnyingi, ez’okukozesa eziwera ezikwata essaawa oba ennaku ez’emirimu gya bizinensi.

Enkyukakyuka eno okuva ku muyambi okudda ku agenti eyeetongodde yeetaaga enkulaakulana si mu busobozi bw’ekyokulabirako bwokka wabula mu enkola z’okwesiga, okukakasa, n’okulondoola abantu. Bizinensi zijja kwetaaga emitendera gy’okubala ebitabo ku bikolwa bya ba agenti, okukakasa okuddamu okukola emirimu egiddirira, n’amakubo amategeerekeka ag’okulinnyisa omutindo ku mbeera ezitategeerekeka. Okusoomoozebwa kwa yinginiya kukwata nnyo ku nsengeka y’enfuga nga bwe kukwata ku nkola y’ekyokulabirako.

Emikutu nga Mewayz, egyalondoola edda emirimu gy’abakozesa mu nkolagana ya CRM, okukkirizibwa kw’omusaala, n’okukakasa okubuuka, giri mu mbeera nnungi okugaziya enkola eno ey’okubala ebitabo okubikka ku bikolwa ebitandikiddwawo ba agenti. Enkola ya data eyetaagisa okugoberera n’okufuga ba agenti okusinga y’emu — era ebibiina ebitadde ssente mu kimu bijja kusanga ekirala nga kigonjoolwa nnyo. Ebiseera eby’omu maaso ebya software ya bizinensi si bantu abakozesa software oba AI okudda mu kifo ky’abantu. Ye loopu ey’okukolagana nga ba agenti abali ku kyuma bakwata omulimu gw’ebyuma ogw’okutambulira mu nsengekera ate ng’abantu bawa okusalawo, okulondoola, n’obulagirizi obw’obukodyo. Eby’okuyiga ebiyigibwa leero mu kunoonyereza kwa compact GUI agent bizimba omusingi gw’ebiseera ebyo eby’omu maaso.

Ebibuuzo Ebitera Okubuuzibwa

Ferret-UI Lite kye ki era eyawukana etya ku bikozesebwa eby’ennono ebya GUI automation?

Ferret-UI Lite ye nkola ya AI entono, eri ku kyuma eyakolebwa okutegeera n’okukwatagana n’enkolagana z’abakozesa ez’ebifaananyi mu ngeri eyeetongodde, awatali kwesigama ku kuyungibwa kw’ekire. Okwawukana ku bikozesebwa eby’ennono eby’otoma ebigoberera amateeka amakakali, agawandiikiddwa, Ferret-UI Lite ekozesa ensonga ezirabika okutegeera embeera ya screen mu ngeri ey’amaanyi. Kino kigifuula ekyukakyuka ennyo mu nkola n'ensengeka ez'enjawulo, okusobozesa enneeyisa entuufu eringa agenti butereevu ku kyuma nga tewali kiseera kitono.

Lwaki okuddukanya ba agenti ba GUI ku kyuma kikulu ku by'ekyama n'enkola?

Okuteebereza ku kyuma kukuuma data ya screen enkulu — omuli ebigambo by'okuyingira, ebiwandiiko by'omuntu, n'enkola y'emirimu gya bizinensi — yonna mu kitundu, okumalawo obulabe obw'ekyama obukwatagana n'okutambuza ebifaananyi ku screen ku seeva ez'ewala. Era eggyawo network latency okuva mu buli cycle y’enkolagana. Ku mikutu gya bizinensi nga Mewayz, OS ya bizinensi eya modulo 207 esangibwa ku app.mewayz.com okuva ku $19/mo, ba agenti ku kyuma okukkakkana nga basobola okukola otomatiki enkola z’emirimu ez’emitendera mingi egy’enjawulo awatali kulaga mirimu gya munda bweru.

Biki ebisinga okusoomoozebwa mu by'ekikugu mu kuzimba ebikolwa ebitono, ebikola obulungi ebya GUI agent models?

Okusoomoozebwa okukulu kwe kugeraageranya obunene bw’ekyokulabirako n’obusobozi bw’okutegeera. Okutegeera GUI kwetaaga okukubaganya ebirowoozo mu kifo, okutegeera ebiwandiiko, n’okuteebereza mu mbeera omulundi gumu — emirimu egitera okwetaaga ebikozesebwa ebinene. Abanoonyereza balina okunyigiriza n’amaanyi ebizimbe awatali kusaddaaka butuufu ku ssirini enzito, ezirimu amawulire amangi. Ebizibu ebirala mulimu okukwata enjawulo ennene ennyo ey’okulaba ey’enkolagana ez’omulembe n’okutendekebwa ku datasets ezikiikirira ezikwata ku apps z’abakozesa, dashboards z’ebitongole, ne suites z’okukola.

Ba agenti ba GUI ku kyuma bayinza batya okukyusa engeri bizinensi gye ziddukanyaamu enkola y'emirimu gya pulogulaamu?

Ba agenti ba GUI ku kyuma bayinza okukola nga abakozi abatalabika, nga batambulira mu pulogulaamu nga beetongodde okumaliriza emirimu egy'okuddiŋŋana nga okuyingiza data, okukola lipoota, oba okulongoosa okuyita mu mikutu. Ku bizinensi ezikozesa emikutu gya byonna mu kimu nga Mewayz — nga egaba modulo 207 ezigatta ku app.mewayz.com ku $19/mo — ba agenti nga bano bayinza okuyunga ebikolwa mu modulo zonna awatali kuyingirira bantu, okukendeeza ennyo ku nsimbi ezisaasaanyizibwa mu mirimu n’okusobozesa ttiimu okussa essira ku kusalawo okw’omuwendo omungi okusinga okutambulira mu nkolagana mu ngalo.

Try Mewayz Free

All-in-one platform for CRM, invoicing, projects, HR & more. No credit card required.

Start managing your business smarter today

Join 30,000+ businesses. Free forever plan · No credit card required.

Ready to put this into practice?

Join 30,000+ businesses using Mewayz. Free forever plan — no credit card required.

Start Free Trial →

Ready to take action?

Start your free Mewayz trial today

All-in-one business platform. No credit card required.

Start Free →

14-day free trial · No credit card · Cancel anytime