AWS

Saving into S3 Temp folder

When saving test data to parquet exporters, practice saving into /temp folder: s3://prime-data-lake/production/hubo/vdh/temp/ (if you just save anywhere, it might cause running failures if that path is called as /* somewhere in the pipelines). Files saved in temp/ folder will be automatically deleted after 30 days.

Do not use Development folders S3 (hard to change to production)

You might notice that some projects have saved data into /development s3 folders, before going into the /production folder.

However, we now save the data directly in /production folder to avoid extra work into moving all paths from development to production.