Goodhart's Law and Hypothesis Testing
I don't know if there's a shorthand way to say "If I only look at cartons of eggs sold, there's no way to tell if shoppers aren't statically assigned their price sensitivity at birth."
Every day I wake up and think "this is it. Today I'm going to be my best self and do my work and stay focused the whole day through" and then nothing gets done. The doctors say I have ADHD, the wife says I have an emotional regulation problem, and I say I just need to focus. Anyway here's my blogpost.
Here are a few facts.
Inflation is high. Specifically, egg prices are through the fucking roof.
I do most of the grocery shopping and cooking. I am very price-sensitive, and will change my eating behaviors based on cost. This, despite having grown up in an upper-middle class household, never knowing poverty, and having a fairly high salary. My wife is close to the opposite in every one of the above mentioned, which has always been strange to me.
Goodhart's Law really matters, and there's a lot of rationalist handwringing about incentive alignment, one that I most recently read here:
I got 3 pages into what I was doing (reading The Big Nerd Ranch Guide's Kotlin Programming 2nd Edition, page 220), where we have "Now that Taernyl has a list of favorite items (and not a list of lists of favorite items), he is equipped to determine the item of the day. Use the random function to bein Taernyl's new marketing campaign.
Do you see where I'm going with this?
The thing I am suddenly wondering is "is there a standardized way of describing 'there is no delta in these metrics between world A and world B' where world A is the real world and world B is a contrived model"?
For example, Taernyl might choose an item from his menu at random, mark it 50% off, and if every customer is completely price insensitive, he changes exactly none of their behaviors and just take in less money at the end of the day. This is currently how the program works, because each customer is assigned one or two favorite foods at random, and orders one item from the menu at random. There is no increased tendency to order your own favorite food, and no price-based decisionmaking (they'll try to order stuff they can't pay for and get nothing). He can’t even manage to get rid of fish that are about to turn if he gives it away for free!
What if humans are like that too? Then there's no reason to ever have sales, you can't change people's minds. And, you would do well to flood the market in millions of tiny variations of your product, because every time you add a new menu item to the local tavern, you increase your market share (remember, it's randomly assigned!).
Ok, but we know this isn't the real world case, because I am price sensitive and my wife is not. So maybe each character is generated with a "price sensitivity" that is set at the beginning of the script, and I just was born with a high value and her with a low value.
What is the current, if any, standardized way to say "given our sales data, we basically aren't even wrong if we just assume our customers are born with a given amount of price sensitivity." Especially if you look at the aggregate. Maybe we need 3 pallets of sliced white bread every week, even on weeks when bagels go on sale, and the waste delta is basically 5-7 loaves, which is basically within random variation. On the aggregate, maybe the consumerbase is almost not price-sensitive at all!
Of course, it's all within context. I'm looking for the shortest, and most standardized way of saying "this way that obviously isn't true is a good enough model, if we only really care about metrics A B and C". I know some basic statistics and I assume there’s a few different, subtly different, ways of doing the math to say “World A and B are not different enough to be not just random chance”, but I guess it would depend on the exact wording of the exact situation, and there might be a few different tests tests that could be done. I’m not an expert on that though, so I don’t know it right now.
And, for examples of things, my mind immediately goes to bad stuff. "It's not that I'm _-ist, it's that every _ I've met has been substandard on the metrics I care about". Which, I mean, is a pretty compelling argument, honestly. Death to normies, is what I have been advocating. Who needs "good" interpersonal skills, when you can just go all-in on autism?
Anyway, I tried to be price sensitive about the price of eggs, but it didn't really work. Bone-in, skin-on chicken leg quarters are $0.99/lb and I've been doing that a lot. I prefer getting a bed of chopped vegetables and some seasonings mixed up on the bottom of a dutch oven, then put the legs on top and stick it in the oven for about an hour at 350, then put the lid on and let it sit hot for a few more hours so it's super soft and the meat falls off the bone.
Back to work.
The next blogpost, I've been putting off for a while because I want it to be good. The short version is