O'Brien knows he has broken Winston when Winston screams for O'Brien to torture Julia instead.
This occurs when Winston has been taken to Room 101 and is confronted by the rats. With the rats about to eat his face, Winston cracks and begs O'Brien to give Julia to the rats instead. This act of betrayal tells O'Brien that he has broken Winston at last.
