Rendered at 06:40:39 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
JohnQPulp 6 hours ago [-]
All things in moderation, I suppose. 800k+ lines is a lot of unit test code, even for a big monorepo, and deleting that much could be justified if there are still a solid core of better quality unit tests left over. Now, if they really meant all unit tests, that's a bit more suspect, but maybe they still have a solid core of integration and other higher order tests, which could make the decision still somewhat sensible, on a cost-benefit basis.
> Online I’ve seen instances where developers are proclaiming that all of their tests will be end-to-end. So we’re trading tests that are cheap with tests that are slower, need a running system and fail intermittently. An end-to-end test belongs at the top of the pyramid and when you move everything there, it gives you less coverage at a higher cost and topples over.
E2E tests can definitely be fragile, but I've definitely also seen cases of unit tests being fragile too: they try to go way beyond the scope of the "unit", and end up failing inconsistently for all sorts of benign reasons (e.g., leaking state between tests), wasting time. When you get into those kind of situations, just declaring unit test bankruptcy is a little tempting. But there's got to be some balance.
tadfisher 11 hours ago [-]
If LLMs are good enough to write code, they are good enough to write tests. In fact, they can be quite creative at finding test scenarios which exercise the code in unexpected ways.
The usefulness of testing is basically orthogonal to the advent of language models. This question boils down to whether you think the agents are "cheating" through changing the invariants encoded by the tests to something that no longer matches your project's requirements. To this, I would suggest that you read the code they produce, tests included, and not sign off on stupid changes to cheat on tests. This is not a common thing I've caught agents doing, FWIW.
10000truths 10 hours ago [-]
Another option is to gate success behind a bunch of black-box tests that the agent has permission to execute but not read or write. You'd likely have to write those black-box tests yourself, though.
seanmcdirmid 10 hours ago [-]
I'm implementing a similar process right now (calling it cleanroom for now):
Given a specification as a source of truth:
1. the lib agent can write lib code, but cannot look at test code.
2. the test agent can write test code, but cannot look at lib code.
3. the QA auditor can run tests and can look at lib and test code, but cannot modify either. It determines blame and then delivers feedback to lib or test agent in a way that doesn't reveal details about the code they are not allowed to look at.
You spend more tokens using this process (since multiple agents have to read the spec and similar files), BUT you get better results. The project is here:
> Online I’ve seen instances where developers are proclaiming that all of their tests will be end-to-end. So we’re trading tests that are cheap with tests that are slower, need a running system and fail intermittently. An end-to-end test belongs at the top of the pyramid and when you move everything there, it gives you less coverage at a higher cost and topples over.
E2E tests can definitely be fragile, but I've definitely also seen cases of unit tests being fragile too: they try to go way beyond the scope of the "unit", and end up failing inconsistently for all sorts of benign reasons (e.g., leaking state between tests), wasting time. When you get into those kind of situations, just declaring unit test bankruptcy is a little tempting. But there's got to be some balance.
The usefulness of testing is basically orthogonal to the advent of language models. This question boils down to whether you think the agents are "cheating" through changing the invariants encoded by the tests to something that no longer matches your project's requirements. To this, I would suggest that you read the code they produce, tests included, and not sign off on stupid changes to cheat on tests. This is not a common thing I've caught agents doing, FWIW.
Given a specification as a source of truth:
1. the lib agent can write lib code, but cannot look at test code.
2. the test agent can write test code, but cannot look at lib code.
3. the QA auditor can run tests and can look at lib and test code, but cannot modify either. It determines blame and then delivers feedback to lib or test agent in a way that doesn't reveal details about the code they are not allowed to look at.
You spend more tokens using this process (since multiple agents have to read the spec and similar files), BUT you get better results. The project is here:
https://github.com/mcdirmid/cleanroom
I haven't formally released anything yet. I'm still benchmarking to show how and why it is better (and what the cost is).
I would rather generate them and then read them.