How LLMs Actually Make Better Programmers (and further pedagogical quibbles)

premise

After I saw that Github Copilot was able to do my DSA coursework with little guidance (though, I never did use it to cheat) I for sure thought that LLMs would ruin kids ability to learn how to code and that it would handicap their learning of not only coding but the meta-cognitive skills around learning coding (ie, “learning how to learn”).

In the past few weeks my view has switched entirely. I write this article to help people understand the otherside.

argument

I have been extensively using LLMs to code maximally, as detailed in this article, but only hit my peak productivity a few weeks ago when I fixed my prompts and tooling (and o1 came out). This period feels very reminiscent of when I was a “script kiddie” (or skids, as the young people call them) back in the day.[1] I would copy code from random places and basically pray that my hodgepodge would compile and work, if it didn’t I’d ask on stack overflow or IRC or discord (often amassing mass downvotes and bad rep). This period was a necessary and integral part of my (and I am sure, many others) success and learning.

The only difference is that now “script kiddies” can accomplish much more using their traditional, age old, heuristics, and hit harder problems faster.[2] This is good for two reasons. First, it raises the floor of coding, more people can create more better things. Second, it accelerates the traditional learning process by forcing people to solve harder problems. The issue increasingly becomes one of curiosity and motivation to solve these harder problems. Curiosity is one of the strongest tenets of hackers (and people in generally) and will filter out people who probably should not be programmers in the first place, while decreasing the barrier to entry for even more people who do not fit the traditional mold.[3] Even back in the day, many did not graduate from “script kiddie”.

What I mean by this specifically is that people generally associate programming with “smart” or “nerdy” people and are less likely to try it themselves, especially given system disadvantage. With LLMs more of the people who lack access or systemic advantages or who are societally prodded away from coding can more easily learn. I would have to guess if I was not born in the bay area to two technically inclined parents as an Indian male that I would probably not be a coder (which is an insight I tend to go back and forth on, but I generally consider to be true).[4]

At the end of the day, when you encounter an error or limitation in an LLM, you have to be able to fix it.[5] The only way to fix it is to learn something. The only difference is you learn more things faster by tackling harder problems and if you don’t understand something you have access to a 24/7 intelligent entity. The democratization of intelligence is a wonderful thing, analogous to the internet and stuff like stack overflow, except now the latency has reduced.

calculator (internet?) analogy

This analogy has been beaten to death, so feel free to skip it.

People said the calculator would ruin learning math. It didn’t, and, in the same way we have calculator and non-calculator tests[6], we are going to have AI and non AI tests. I still find the balance important and think that to “formally” learn things on a concrete level, you probably need to forgo LLMs.

I think the internet is also partly analogous just due to the mission of democratizing information. Now, we are democratizing intelligence. I don’t really know how it was with the internet, but I am sure some some white beard GNU wizard who was coding 30+ years before my birth made the argument that kids who did not start on mainframes, punchcards, and manuals/books have it easy and they don’t know how to be a “real programmer”. (Absolutely no hate to white beard GNU wizards, they are wizards for a reason).

[1] The key difference being that if the code doesn’t integrate properly I generally know immediately why and can start integrating or redoing my prompts, making it much more productive.

[2] They also save the time of experienced users by having commoditized access to intelligence, so their questions don’t clog up discords, forums, IRCs, etc.

[3] A similar problem exists in math, where people generally think back to their Calc 1 class and say “yeah I sucked at math” when either 1. their teacher was bad 2. they didn’t even try because people have been telling them they are bad for their whole life or 3. they didn’t even try because they think they hated math. In order to reach the good math (proof based math, which basically requires a whole different host of skills), the people who are good and would like the good math are effectively screened out. Math is literally the most switched out of majors because of this. (probably).

[4] Personally, I did not know I could write or analyze text at the average level until high school, I just always thought I was a STEM guy who was bad at English before that…

[5] If you don’t run into issues like this, or you can always fix them by prompting, I think you found AGI…

[6] I also think non-calculator tests are a symptom of a bigger problem (at least in my experience). I want to preface this by saying the way I learn things is atypical. If I can’t use a calculator because it impedes my understanding of the underlying math, why can you not just teach us something that presupposes understanding of the underlying math?

For example, instead of having me manually multiply matrices without a calculator to prove I understand matrix operations, just teach me linear transformations and ask me to solve problems that require actual understanding–like explaining why matrix multiplication isn’t commutative using geometric intuition, or determining if a transformation is invertible by analyzing its properties. Someone who truly understands matrices isn’t going to be hindered by using a calculator for the arithmetic. Maybe it would be helpful to know how to manually multiply matrices, but testing that has to be one of the worst ways to test for understanding. I frequently partake in rebellion against this in my CS classes by spending time memorizing every algorithm I would need to know and spitting out a super terse hard to grade version on the test. Though I think my anger has befallen the poor TA instead of the relevant systemic forces…

I also often used tricks of playing with the calculator and noting down the invariants when solving those types of problems, which is basically learning on the fly while taking the test. I can also do the same thing manually if I remember a rule but not the proof. This is super fun (because you are actually doing math the way god intended) and minimized the time spent studying (instead relegating to “uhh, hopefully I can figure it out on the test”).

I also think this ties back to my math discussion in footnote 3 and how current math pedagogy is screening and filtering the wrong people.