-
Dask
If you’ve ever tried to do any serious data analytics at some point you’ve probably hit some sort of hardware limitation. Usually it’s running out of memory, since most analytics tools tend to try and load all your data at once. Either that, or your analysis requires more CPU resources than Intel can manufacture in an entire year. …
Read the rest
-
Geographic Functions in BigQuery
This is a little post to demo a couple of the dead-handy geographic functions that are available in Google Cloud’s BigQuery. I was working on a quick little proof-of-concept project recently where the team and I had to track a bunch of telemetry data on a map in Tableau. The data itself came from a handful of different sources, updated every minute. …
Read the rest
-
GCP SSL Certs
Working at any large enterprise means you’ll more than likely be stuck behind a proxy server that restricts your access to the internet. It’s also likely to intercept your web requests and perform a man-in-the-middle (MITM) inspection of your encrypted data. There’s no privacy on corporate networks! …
Read the rest
-
Load-Testing with Locust on Google Cloud
Everyone’s obsessed with scale. …
Read the rest
-
Practical Multi-threading in Rust
Ages ago I wrote a rudimentary word counter in Rust, and then demoed how to spawn threads in Rust here. In this post I’m going to mash the two together so that my word counter can process multiple files concurrently instead of in serial. …
Read the rest
-
Rusty Threads
Fearless Concurrency Another thing Rust boasts is “fearless concurrency”, another concept that’s hard to implement correctly. I’ll only very briefly touch on this. …
Read the rest
-
Adventures in Rust
I think having “Adventures” in the title is probably pretty misleading. “Mildly interesting nuggets, if you’re into this sort of thing” is probably more accurate. …
Read the rest
-
Getting Rusty
I like systems software development. I’m probably not very good at it, but I still like playing around with the nuts and bolts of systems. Most people think of software as just the applications they use, but there’s a ton of other magic happening beneath them. …
Read the rest
-
Cracking Window Functions
Window functions - what are they and how would you use them? A question is often fired at data engineering interviewees, by interviewers more interested in hearing crap regurgitated from the back-pages of a manual, rather than how you’d solve a particular problem. …
Read the rest
-
Data Bashing
I’ve spent a long time tinkering around with data, moving it around, mangling it and generally trying to make it a bit more useful. There are plenty of fancy and expensive tools out there like IBM InfoSphere, Informatica, Talend, and Ab Initio you can use to do this sort of stuff. So far, so boring. …
Read the rest