He says to her 'You've got to' because he knows that she will not need him.
She thinks she is a 'freak' because she isn't one of 'them'.
She calls herself a 'half-caste' then tells him about her mother crying in the pub because 'we could sing better songs than these'.
As she shows when she tells him about having 'already done' Blake's poetry' - she is moving on, leaving him behind.
As Frank deteriorates - drinking more and falling over in the lecture - Rita becomes more remote.