Diary of a DataGirl

A Hopeful Restart

Apr 10 2025

Well… it turns out that I was caught up in learning on the job as well as the Covid-19 pandemic, and neglected to actually write what I wanted to document when I set out to create this blog. I ended up keeping it around because it was useful for testing web stuff, but so many years later, I’m less than a month away from another life milestone… perhaps it is time to revisit this one.

During my time at Vend, it was acquired by Lightspeed, a Canadian company. The timing was rather unfortunate as I missed out on the vesting of my Vend options, though to be fair I did not place much stock in it to begin with (heheh).

What I did learn at Vend and then Lightspeed was managing a data lake with PostgreSQL, a data warehouse with Redshift, a data pipeline with Python and Prefect, and then migrating from Redshift to BigQuery. Being the key data engineer to get Prefect into production with AWS was perhaps the highlight of my time at Vend/Lightspeed: quite painful, but rewarding. As a bonus, I contributed code fixes and bug reports to Prefect such that they roped me into their Club 42 community ambassador programme.

Eventually, I moved on to Marketplacer, an Australian company that had setup a branch in New Zealand. Over there I was introduced to Snowflake and dbt. I also was involved in more attempts to Terraform our way into managing data infrastructure, something I first experienced with AWS and Prefect, though at Vend I had more support from the SRE team. Oh, and we used the AWS integration with Airflow instead of Prefect, so I felt like such a traitor, lol.

I think that I’ll continue future posts by elaborating on these learnings and adventures, as well as updating y’all on where I am now!

Photo by Nadir Čajić

A New Beginning

May 30 2019

Back in 2016, I was feeling bored developing and maintaining websites for a number of disparate clients. Actually, we hardly did any new development; it was more maintenance, dealing with security issues from code written by other people, and providing random IT support on the side because clients expected it as part of the package.

After observing that my local doctor in Singapore was still using paper records, I was inspired to look into health tech, and ended up applying for a master’s degree course in health informatics, in New Zealand. Come 2019, I have this shiny new degree, but responses from district health boards and companies in the health tech space in Auckland was not forthcoming (or just slow, since I did get invited for interviews… after I accepted the job), and I ended up highlighting the “informatics” part of my degree to land myself a data engineer job at a point of sale software company.

So now, I decided to start this blog with the idea of recording and reflecting on my journey into this area. My postgraduate study gave me some overview into data science, and my background as a software engineer gives me the software skills, yet data engineering kind of lies at an intersection between the two, so it will be interesting to see where this leads me. The HIVERY tagline of “Data has a better idea” (photo by Franki Chamaki on Unsplash) is not literally true, of course: data does not have ideas of its own, but in a sense I am hoping that it was a better idea to go into data engineering than to stick with more mainstream software engineering.