I think I misunderstood the quote. I thought he was ruling out any attempt to describe QM using conventional localized mechanics, hence necessitating the acceptance of randomness in order to preserve what we know to be true (based on every experiment to date) from Relativity.
Bell's own view was that he was ruling out locality,
period. Proofs of Bell inequalities don't explicitly assume "counterfactual definiteness"[sup]1[/sup], and some versions explicitly don't assume determinism. The bound follows completely from locality and the so-called "free choice" hypothesis[sup]2[/sup]. Bell inequalities still apply to the more general class of fundamentally
stochastic locally causal theories.
Historically, Bell's theorem originally appeared as a response to the 1935 EPR paper, whose authors argued that QM couldn't be considered a "complete" theory, basing their case on an example of two spatially separated particles entangled in position/momentum. For anyone who isn't already familiar with the EPR argument, it goes something like:
- QM presents us with a dichotomy: there are sets of complementary observables, such as position and momentum, which are not simultaneously well-defined. We must accept either that
- the wavefunction/state description of QM is incomplete, or
- if we want to retain QM, we have to accept that eg. position and momentum do not have simultaneous reality.
- Assume QM is a complete theory, so we reject (1a) and consequently we're forced to accept (1b).
- QM allows for the existence of pairs of particles in entangled states. Suppose particles A and B are spatially separated and entangled in position/momentum. Then:
- If I measure A's position, then B is projected onto a particular position state. It's position is completely determined before any measurement is made and therefore B has a "real" (ie. exact, predetermined) position.
- If instead I measure A's momentum, then B is projected onto a momentum state and, by the same argument, has a "real" momentum.
- We come to the conclusion that B must have both a real position and a real momentum.
- This contradicts point 2. Reductio ad absurdum. We're forced to accept that (1a) is true.
The authors ended their article noting only a single caveat: step 4 only follows from step 3 because they were assuming that whether
B's position and momentum are "real" should be independent of which measurement I choose to make on
A a long distance away. You escape the conclusion if you allow the choice of measurement on
A to instantaneously toggle which of
B's position or momentum is predetermined, which violates locality.
What often seems to be misunderstood is that Bell actually sympathised with Einstein on this point: basically, his response amounted to something like "yes, unfortunately QM is non-local, but sorry, it doesn't look like there's much we can do about that." It wasn't originally a question of whether "classical" physical principles could be restored by "hidden variables".
There's a good exposition of Bell's views, which makes for reasonably light reading, available
here on arXiv. The EPR paper is also available
here.
*****
[sup]1[/sup]By this I mean that Bell's theorem isn't based on any assumptions about whether eg. particles can have a simultaneously well-defined position and momentum. Giving up on that and for example accepting Heisenberg uncertainty doesn't help you restore locality. Bell inequalities only rely on the assumption that the results of measurements are specific and "real".
[sup]2[/sup]This is an assumption, consistent with experience and arguably necessary for applying the scientific method at all, that arbitrarily strong correlations between systems don't exist in nature, or at least that it's possible to "insulate" against them when necessary in practice. Otherwise you could claim that nature is local and that it's only some detail of the initial conditions of the universe that caused it to evolve in such a way that every Bell experiment came out "wrong". You could use that explanation against
any experimental result you didn't like.