A pitfall of A/B testing models served behind a cache
Mar 12, 2023
A few years ago, I was working with Jeff Gee on a system that served models behind a cache. Jeff was trying to launch a new model iteration that showed solid improvements offline but was weak in our A/B tests. After many investigations, he figured out it was due to our cache behaving differently between our control and our treatments. Since I have seen this issue again recently, I thought I would share why serving a model behind a cache can be tricky, and what can be done to mitigate cache related biases.
...
➦
Balancing Focused Work and Meetings
Mar 17, 2022
My colleague Rose Tan recently published an excellent article on How to Run Effective Meetings (she is one of the most disciplined person I know when it comes time to cancel unneeded meetings, you should subscribe to her newsletter!) and dared me to publish an old internal article I had on the topic. So here we are!
Long periods of uninterrupted work are needed to solve complex problems and meetings can be in the way.
...
➦
Notes on Who gets what and why by Alvin E. Roth
Mar 11, 2021
I have the privilege of interacting daily with many smart colleagues having expertise in fields including machine learning, infrastructure, data science, product design, and with economists specialized in marketplace design. I am at ease discussing all of these but marketplace designs where I often feel like a toddler asking adults for explanations and searching words to efficiently express ideas. I thus asked my colleague Giorgio Martini for a book to get started, and he pointed me to Who Gets What and Why by Alvin E.
...
➦
My thoughts on Amusing Ourselves to Death by Neil Postman
Dec 11, 2020
A popular quote that we use in Software Engineering goes like this:
If everything you have is a hammer, everything looks like a nail.
This is known as the The law of the instrument and means that a limited toolset can get the job done but there will be a price to pay in quality or productivity. Looking back at my own experience, it reminded me when I took two hours to create a Java program that would have taken 30 seconds to write as a sed command.
...
➦
Bandaid: A band name generator
May 09, 2020
I have owned an electric bass for the last twenty five years. I managed to play it in the low 10s of hours and at some point could riff noises resembling the Pink Panther theme song. I decided it was time to get a band name matching my talent. Given that I am an engineer, I obviously automated the whole process.
This post explains how I trained a deep neural network to create new band names on demand.
...
➦
My favorite parts of Effective Python by Brett Stalkin
Feb 23, 2020
I programmed in Python on and off for the last 15 years but haven’t looked systematically at the language in the least 10 years. It was time to sharpen my saw so I picked up Effective Python by Brett Stalkin. The book is organized into 90 items show casing Python best practices with their rationale and code examples. This post covers my favorite items, the ones I intend to integrate in my use of Python.
...
➦
Building a gazetteer of music bands using Wikidata
Aug 24, 2019
A few weeks ago I started a hobby project for fun and learning. My task this week is to compile a gazetteer of music bands and artists. I wanted to play with Wikidata for a long time and it was the perfect opportunity.
Getting familiar with Wikidata Wikidata is the Wikipedia of data. Contributors are either robots or human updating a database of facts. The best way to grasp how Wikipedia and Wikidata compares is to look at a concrete example, the entries for the band Arcade Fire: Wikidata, Wikipedia.
...
➦
Crawling all submissions from a subreddit
Jul 29, 2019
I am starting a new hobby project called word of mouth to recommend music groups based on suggestions from the r/ifyoulikeblank subreddit community. One of the first problem to solve is to download the submissions of the subreddit. This post goes over one possible option.
There are three options when it come to crawling reddit:
Use reddit rest api directly or through a python client like praw. The main downside of this approach is that we are limited to the top 100 submissions for each or our query.
...
➦
Building a tweet corpus
Nov 10, 2012
I wanted to play around with some tweets, but I quickly discovered that getting a hand on a corpus is not that easy because of Twitter terms of service. It is up to every one to create their own corpus.
Luckily, Twitter has an API to sample tweets randomly. I created a small application over it that can be used following these steps:
Register a twitter application on https://dev.twitter.com/terms/api-terms. Application name is not important, you only want to get its credentials.
...
➦
Forer Effect
Mar 03, 2012
I recently learned about Forer effect or how people can take general statements and make them their own. In a classic experiment , Forer asked his students to fill a personality test. A week later, he gave his analysis back to each student and made them rate its accuracy on a scale of 0 (poor) to 5 (perfect). The analysis were so targeted that only one out of 39 students rated the results lower than 4.
...
➦
