The Self-Improving Developer  ←   → 
You’ve taught yourself variables, classes, functions, objects. What next?
Chapter 7

Data Mining

As a developer, there are hundreds of skills you can add to your belt. Front end, back end, ML, big data, databases, micro-controllers, CI, and so on. All of these are interesting and each of them brings exciting new career opportunities, or at least fun diversions.

Today, I’m here to say that good old data mining, despite being one of the less talked about skills, is one of the most powerful and widely applicable disciplines. Data mining can help you with anything from getting the best deal on your car to proving your worth to your employer to finding a once-in-a-lifetime business opportunity.

I’m using the term “data mining” loosely here. For me, it encompasses everything from getting data, to analyzing it, to summarizing everything into some useful nugget of information.

Examples

Most recently, I used the API approach to get data from YouTube, to find out how many people viewed the YouTube videos which I scripted or in which I appeared. This was a fun little exercise with a lot of value for me (making YouTube videos is part of my job, and viewership is one of the signals to track).

In April, I combined several datasets to learn which time it makes most sense to have a live stream if your audience is distributed around the world the way developers are. (That is to say, not uniformly, but in clusters around US Coasts, Brazil, EU, India, China, etc.) I tweeted the result, if you’re interested.

A graph of timezones

Not too long ago, also this year, I had a need for a bespoke tool that would give me writing inspiration for sword & sorcery prose. I had a bunch of highlights in Kindle. I paid a person to extract and lightly categorize those highlights, then put it all into a SQLite database, and wrote a tiny desktop app to access it.

Screen Shot 2020 08 26 at 11 11 18 PM

Now this app sits in my MacBook dock alongside such software as VS Code or Chrome or Evernote, and it gets more use than some of the other ones.

A few years ago I was buying our family’s first car. Me being me, I went all analytical on the task of choosing the right one, weighing such things as Total Cost of Ownership, TÜV reliability index, gas mileage, or average depreciation curve. I am not a car lover, so I had almost zero emotion in the game, and just wanted not to waste money. (As they say, car purchase can be one of the worst financial decisions a person can make.) All this was a substantial amount of work, but I enjoyed most of it, I learned a lot, and I’m convinced our family saved well above $5K while still ending up with a good, reliable car.

Summary

Data mining can help with anything from hobby projects to startup business plans. It spans many disciplines, so it’s seldom boring, and can be actually quite fun. If you’re thinking about learning a new skill, you should seriously consider data mining.

The best way to learn data mining is to

  1. Learn the absolute basics, and then …
  2. … start a project and learn the rest by doing.

Have fun!