March 14, 2023 
4 min readThere's no denying that we have already ushered in the era of big data. An enormous amount of information is generated every second. While decision-makers can gain invaluable insights from this...
February 9, 2023 3 min readAs a data engineer, you probably have heard about Hadoop. It is one of the most popular frameworks for distributed processing of large data sets. It is less costly and more secure than other...
January 30, 2023 4 min readAs a data engineer, you probably have heard of the data integration methodology called ETL (Extract-Transform-Load). It has been around for a while, and many data engineers have used this methodology...
December 5, 2022 4 min readFew would disagree that the word "lazy" has a negative connotation. We usually describe someone who is not willing to work hard as lazy. However, not all laziness is undesirable, and sometimes we...
November 29, 2022 3 min readIf you have been in the world of big data long enough, you probably have heard about Parquet files. You might even have used it while thinking to yourself: "why can't we just use CSV files?" Today, I...
November 21, 2022 4 min readHi! My name is Ziyu Chen. I am a full-stack engineer at Colorkrew. I love learning and writing about data engineering. Today, I would like to discuss transformations and actions in Spark. Of course,...
