The United States ought to do away with the Electoral College, according to former attorney general Eric Holder, calling it “a defect in our democracy.”

“We have had in the last five elections two presidents who were not — did not win the popular vote. The presidency is the one office in this country that represents all of the people,” said Holder, who served as the nation’s chief law enforcement officer through the first several years of the Obama administration. “You have a district representative, congressman, you have a person who represents your state or two people represent your state, you have senators and you vote for them in a direct way. Seems to me we ought to have a direct election of the president of the United States.”

Holder said how he sees removing the Electoral College would put more voters up for grabs, and the candidates would have to work harder to court them.

“If we did that, you would have Republicans who would be campaigning in California. The Republican would campaign in New York. The Democrat would campaign in Texas,” he said. “That means in some swing states, smaller states, would not get as much attention as they do now but I think the vast majority of people in this country would get a great deal more attention, attention that they deserve. But I don’t want that to take away from the focus … on the problem of gerrymandering. That is what I have devoted most of my post-government life towards, eliminating this whole problem of gerrymandering.”

Gerrymandering involves creating congressional districts tailored to be most easily won by a specific candidate or political party, rather than creating them to keep geographic locations or populations together.