Do You Know How Conservatives Became Known As The Stupid Party?

In his autobiography, J. S. Mill discusses his time in parliament. He had a long paper trail by then. And those on the other side of the aisle brought up some passages:

"[The Tory leaders] gained nothing by drawing attention to this passage, which up to that time had not excited any notice, but the soubriquet of 'the stupid party' stuck to them for a considerable time afterwards." – John Stuart Mill, Autobiography, Collected Works, Volume 1, p. 277

The editor provides a clarification:

"Mill, who was replying directly to Sir John Pakington, said: 'What I stated was. that the Conservative party was, by the law of its constitution, necessarily the stupidest party. Now. I do not retract this assertion. but I did not mean that Conservatives are generally stupid: I meant, that stupid persons are generally Conservative.'"

This parliamentary interchange was about a footnote in which Mill noted the existence of negative partisanship:

"This blunder of Mr. Disraeli (from which, greatly to his credit, Sir John Pakington took an opportunity, soon after, of separating himself …) is a speaking instance, among many, how little the Conservative leaders understand Conservative principles. Without presuming to require from political parties such an amount of virtue and discernment as that they should comprehend, and know when to apply, the principles of their opponents, we may yet say that it would be a great improvement if each party understood and acted upon its own. Well would it be for England if Conservatives voted consistently for everything conservative, and Liberals for everything liberal. We should not then have to wait long for things which, like the present and many other great measures, are eminently both the one and the other. The Conservatives, as being by the law of their existence the stupidest party, have much the greatest sins of this description to answer for: and it is a melancholy truth, that if any measure were proposed, on any subject, truly, largely, and far-sightedly conservative, even if Liberals were willing to vote for it, the great bulk of the Conservative party would rush blindly in and prevent it from being carried." – John Stuart Mill, Considerations on Representative Governement, Collected Works, Volume 19, p. 452 (footnote)

I think something of that interchange in the British parliament is fairly well-known. Have you heard this story before?