I-AWS Aurora PostgreSQL ingeza imibuzo eqondile yamachibi edatha e-Iceberg naseParquet

Ikhaya » SQL » I-PostgreSQL » I-AWS Aurora PostgreSQL ingeza imibuzo eqondile yamachibi edatha e-Iceberg naseParquet

I-Amazon Web Services ivule indlela entsha yamaqoqo e -Aurora PostgreSQL adinga ukuxuba amarekhodi ezinhlelo zokusebenza zansuku zonke namafayela ahlezi echibini ledatha. Isizindalwazi esiphethwe manje sesingafunda amatafula e-Apache Iceberg namafayela e-Apache Parquet ngaphandle kokuwakopisha kuqala ku-Aurora, ngakho-ke isitatimende esisodwa se-SQL singamboza kokubili imigqa yokusebenza ebukhoma kanye nolwazi olugciniwe. Lesi sinyathelo sihloselwe amaqembu asekhathele ukwakha nokunakekela izingane, ukuguqula nokulayisha imisebenzi ukuze nje aphendule imibuzo ehlanganisa idatha yamanje neyomlando.

Esikhundleni sokuphatha ichibi ledatha njengezwe elihlukile, le nsizakalo ivumela amaklayenti e-PostgreSQL akhona, amathuluzi kanye nama-endpoints ukufinyelela okuqukethwe kwe-lakehouse ngamathebula angaphandle. Lokho kusho ukuthi akukho kopi ephindwe kabili yedatha yomlando okufanele ihlale ngaphakathi kwedatha yokusebenza, futhi abathuthukisi bezinhlelo zokusebenza bangaqhubeka nokusebenzisa i-syntax abayazi kakade. I-AWS ibeka lokhu njengendlela yokunciphisa ukulungiswa kwamapayipi, ukunciphisa ukwanda kwengqalasizinda nokwenza kube lula ukucebisa ukuthengiselana ngomongo omdala.

I-athikili ehlobene:

Izitayela Eziyinhloko Zedathabheyisi Ezakha Amapulatifomu Edatha Yanamuhla

Lokho okushintsha ikhono elisha le-Aurora PostgreSQL

Lesi sici akuyona umkhiqizo ohlukile wokuhlaziya amakhasimende okumele ame kwenye indawo. Sivulwa ngesandiso se-aurora_analytics kanye nendima ye-AWS Identity and Access Management enikeza iqembu imvume yokufinyelela i-Amazon S3 kanye ne-AWS Glue Data Catalog. Uma leyo ndima kanye nesandiso sezikhona, abaphathi bakha amathebula angaphandle akhomba kudatha ye-Iceberg noma ye-Parquet bese bewabuza njengamathebula ajwayelekile e-PostgreSQL.

Ingxenye ephawulekayo yokwakhiwa kwesakhiwo ukuthi i-DuckDB ifakwe ngqo ku-Aurora . Injini yokuhlaziya evulekile iphatha ukuskena phezu kwamafayela e-lakehouse kuyilapho i-Aurora iqhubeka nokunikeza imithwalo yemisebenzi yokusebenza. Ukucutshungulwa kwemibuzo kuhlala ngaphakathi kwendawo yedathabheyisi, okugwema ukunikeza umsebenzi eqenjini elihlukile futhi kusiza ukugcina ukubambezeleka kanye nenethiwekhi kulawulwa.

Imithombo esekelwayo ifaka phakathi amatafula e-Iceberg abhaliswe ku-AWS Glue Data Catalog, izinto zeParquet kanye ne-Iceberg ku-Amazon S3, kanye nedatha egcinwe ku-S3 Tables. Amakhathalogi angaphandle alandela isikhombimsebenzisi se-Iceberg REST Catalog angafinyelelwa futhi ngokusebenzisa i-federation ne-Glue, ngakho-ke amaqembu akudingeki alahle ukutshalwa kwezimali kukhathalogi asevele enakho. Umbuzo ungahlanganisa imigqa evela ku-Aurora namathebula e-Iceberg asakazeke kukhathalogi eziningi, okunikeza izinhlelo zokusebenza umbono ohlangene ngaphandle kokuhambisa amafayela.

Izinguqulo, ukusetha kanye nezinsizakalo ezisekelwayo

I-Aurora PostgreSQL 17 eqala kunguqulo 17.11 kanye ne-Aurora PostgreSQL 18 eqala kunguqulo 18.6 yizo ezikhishwayo ezisekelwayo. Ngemva kokudala noma ukukhetha i-cluster, umphathi unamathisela indima ye-IAM ngesici se- AuroraAnalytics bese evula isandiso. Ukusetha kungenziwa nge-console ye-Amazon RDS noma iklayenti le-PostgreSQL njenge-psql.

Izincazelo zethebula langaphandle zingadalwa ngayinye ngayinye, kodwa izindawo ezinkulu zingasebenzisa i-IMPORT FOREIGN SCHEMA ukukhiqiza amathebula amaningi angaphandle ngesinyathelo esisodwa kusuka kudathabheyisi ye-AWS Glue Data Catalog. Kumafayela e-Parquet, i-Aurora ingafunda i-schema kusuka ku-metadata yefayela, okususa isidingo sokupela yonke ikholomu ngesandla. Leyo mininingwane ibalulekile kumaqembu anamasethi edatha abanzi noma ashintsha njalo.

Imibuzo efundwayo ingase isebenze kusibonelo sombhali noma kukhophi yokufunda, ngakho-ke ukuskena kokuhlaziya kungasuswa emsebenzini oyinhloko wokusebenza. Imiyalo ebhala idatha ku-Aurora, njenge-CREATE TABLE AS SELECT, INSERT INTO … SELECT noma MERGE INTO, isebenza kusibonelo sombhali . Lokhu kunikeza amaqembu indlela yokwenza idatha ye-hot lakehouse ibe yithebula lendabuko lapho edinga izimpendulo ezisheshayo njalo.

Ukuze babone ukuthi umbuzo wenzani, abaphathi bangashayela i-aurora_analytics_stat_statements(). Umsebenzi ubika amamethrikhi afana nemigqa eskeniwe, amabhayithi afundwayo kusuka ku -Amazon S3 kanye nama-cache hits . Idatha esetshenziswa njalo igcinwa ku-instance ye-Aurora, ngakho ukufinyelela okuphindaphindiwe ezintweni ezifanayo ze-lakehouse kungabuya ngokushesha ngaphandle kokudonsa amabhayithi afanayo ngokuphindaphindiwe kusuka kusitoreji sezinto.

Ukusebenza kahle, ukugcinwa kwesikhashana kanye nezindlela zokubambezeleka okuphansi

I-Aurora isebenzisa i-predicate pushdown kanye ne-column pruning , okusho ukuthi imigqa namakholomu afanele kuphela acelwa echibini ledatha. Njengoba isethi yedatha eyisisekelo ikhula, lokho kuhlunga kusiza ukuvimbela imibuzo ukuthi ingaguquki ibe ukuskena okugcwele. Imiphumela egciniwe kanye nezibalo zemibuzo kunikeza opharetha indlela esebenzayo yokulungisa imithwalo yemisebenzi esikhundleni sokuqagela ukuthi isikhathi sichithwa kuphi.

Kuzinhlelo zokusebenza ezidinga ukubambezeleka kwe-single-digit-millisecond, i-AWS ikhomba ukwenzeka njengento yokuphuma. Amarekhodi echibi ledatha akhethiwe angakopishwa kuthebula le -Aurora PostgreSQL lendabuko ngemiyalo ye-SQL ejwayelekile, ngemva kwalokho abuzwa njenganoma yiliphi elinye ithebula lendawo. Lokhu kugcina indlela esheshayo yedatha eshisayo ngenkathi kusashiya ichibi ledatha eligcwele litholakala ukuze lihlaziywe kabanzi.

Le nkampani ibala amadeshibhodi esikhathi sangempela, ukuthengiselana okucebile ngomongo womlando, kanye nama-ejenti e-AI adinga ukucabanga ngamarekhodi amanje nagciniwe njengezimo ezivamile zokusetshenziswa. Intambo evamile umthwalo womsebenzi ongeke ulinde iqembu lasebusuku noma umsebenzi we-ETL ophambene. Ngokuvumela uhlelo lokusebenza ukuthi lucele isizindalwazi esisodwa solwazi olusha noludala, i-AWS inethemba lokususa umthombo ophindaphindayo wobunzima bepayipi.

Ukutholakala namanani entengo

Ikhono liyatholakala kuzo zonke izifunda ze-AWS zezentengiselwano kanye nezifunda ze-AWS GovCloud (US), ngokusho kwe-AWS. Akukho zindleko ezihlukile zesici. Amakhasimende asakhokhela i-Aurora computing eyengeziwe esetshenziswa yimibuzo kanye nezindleko zesicelo se-Amazon S3 ezihlobene nokufunda amafayela e-data lake. Leyo modeli yamanani igcina iphuzu lokungena liphansi, yize ukusetshenziswa okukhulu kokuhlaziya kusazobonakala ebhilini.

I-AWS ihlele ukwethulwa njengengxenye yomzamo obanzi wokuletha ukusebenza kwe-DuckDB ezinsizeni zayo eziphethwe. Ithimba le-DuckLabs, eligcina iphrojekthi ye-DuckDB, lijoyine i-Amazon, futhi inkampani ithi ukuthuthukiswa kwesikhathi esizayo kwenjini yomthombo ovulekile kungangena ku-Aurora nakwezinye izinsiza ze-AWS. Lokho kuxhumana kusikisela ukuthi ukuhlanganiswa kuhloswe ukuthuthuka kunokuba kuhlale kuyisici esisodwa.

Kumaqembu edathabheyisi, ukukhanga okusebenzayo kujwayelekile: izingxenye ezimbalwa ezihambayo, idatha ephindaphindwayo encane kanye nendlela emfushane ukusuka embuzweni kuya empendulweni. Ukushintshana ukuthi izilawuli zokufinyelela, izimvume ze-IAM kanye nokucushwa kwekhathalogi kusadingeka kuphathwe ngokucophelela. Iqoqo elingabona kokubili imigqa yokukhiqiza namafayela e-lakehouse liba yindawo enamandla yombuzo owodwa , ngakho-ke ukubusa kuhlala kuyingxenye yomsebenzi.

Umphumela osebenzayo uwukuxhumana okuqinile phakathi kwe-Aurora PostgreSQL kanye ne-data lake. Ukuthengiselana kwamanje namafayela omlando kungahlangana kumbuzo owodwa we-SQL, i-DuckDB iphatha ukuskena kokuhlaziya, futhi ukwenziwa kwezinto kunikeza umzila osheshayo uma kudingeka. Amaqembu asevele etshale imali kumathuluzi e-PostgreSQL athola enye inketho yokuhlaziya - imithwalo yemisebenzi esindayo, kuyilapho lawo anokuphepha okuqinile noma ukulawula izindleko azofuna ukubuyekeza izimvume, ukuziphatha kwe-caching kanye nokusetshenziswa kwe-S3 ngaphambi kokuyikhipha kabanzi.