If you're reading this
If you're reading this, you probably saw a comment on one of the Substacks I'm subscribed to and clicked, thinking that there was something interesting here.
Well, unfortunately, not really.
Here’s an interesting thought that came to me a few nights ago when my cat woke me up at 3:30 AM and I couldn’t get back to sleep: How easy would it be to scrape all the comments from all the Substacks that I subscribe to, and get a big pile of greppable .txt files on my hard drive?
For me, it seems like it could take a week, but that’s because I’m not good at programming. But someone who knew what they were doing? Maybe an hour, right?
I’ve seen a bunch of comments on other substacks that are way deeper than the usual fare. Rumor has it that Freddie deBoer could be often found in the Slate Star Codex comments section back in the old blogging days, just as one example.
One deBoer commenter was saying (in response to a post about the general lack of direction amongst the American people, to the point that people are embarrassed to do the 2.5 kids and a white picket fence thing? something. This is part of the reason I need to get a searchable database or pile) that Freddie ABSOLUTELY needs to read In Over Our Heads: The Mental Demands of Modern Life by Robert Kegan because it explains that most people are at the level of doing it because society says so and not at the level of doing it for a real reason. Another commenter, maybe on deBoer, or maybe on Astral Codex Ten, was an actual professor who wrote a paper that joined the author in complaining about the general lack of stats literacy in the humanities. https://arxiv.org/abs/2101.05647 It feels jarring to see professors out in the wild sometimes, but now I just want to see where else that guy pops out in the comments sections. Would be as easy as grep username ./*.txt
, right?
Anyway, if you’re reading this and are good at programming, let me know.