Justice Anthony Kennedy occupied the the U.S. Supreme Court’s ideological center for 12 years. With Kennedy’s retirement, who might become the new “center” or “swing” vote?

Kennedy was the decisive vote in a few, but major victories for the court’s liberal wing each term—although not the term that just ended. He became the lone center figure after the retirement of Justice Sandra Day O’Connor in 2006, but he never swung to the left as often as she did. Kennedy despised the “swing” term.

“The cases swing. I don’t,” Kennedy said in remarks in 2015 at Harvard Law School.

O’Connor herself stepped into the swing role after the retirement in 1987 of the court’s center justice, Lewis Powell Jr. Powell and Justice Potter Stewart were the “swing” voters in the 1970s. And even earlier there was Justice Stanley Forman Reed.

With each of those retirements and new appointments, the question arose: Will the center hold?

If President Donald Trump nominates and the Senate confirms a staunch-right new justice, much as Justice Neil Gorsuch has shown himself to be, the most likely member of the court to take on a limited center role is Chief Justice John Roberts Jr.

“This will make John Roberts the median justice,” said Erwin Chemerinsky, dean of the University of Berkeley School of Law. “What we saw this term—conservatives winning everything—is the future for years to come.”

Elizabeth Wydra, president of the liberal Constitutional Accountability Center, cautioned that a “swing” vote after the retirement of Kennedy is “not in the universe of ‘moderate’ or ‘centrist.’”

Wydra described Roberts as “a deep red conservative who no one would ever mistake for ‘moderate’—but a justice who nonetheless occasionally surprises observers with a vote with the four more liberal justices.”

It would be better to view Roberts as the likely “median” justice, said Ilya Shapiro of the Cato Institute.

“That doesn’t mean that every case where Kennedy joined the liberals will now come out the other way—the chief justice will work hard to gain consensus and may occasionally move over to write narrow opinions—but on many issues that will indeed be the case,” Shapiro said. “I can’t predict where Roberts will move left because these are pragmatic not principled considerations, but it would likely be more in challenges to legislation like Obamacare than in culture-war issues like affirmative action.”

The one area where Roberts has stood firm in his conservative bona fides is race, whether in the context of affirmative action, voting rights or school desegregation.

But Roberts has a keen interest in the structure of the Constitution—who has the authority to do what. It is those structural issues where he may move left. He led the court’s liberal wing to uphold the individual mandate in the Affordable Care Act under the tax and spend clause after finding it did not pass muster under the commerce clause.

President Trump, pointing to Roberts’s votes on health care, said a few years ago that the justice “turned out to be a nightmare for conservatives.”

Roberts has a strong interest in the privacy rights of individuals in the digital age and in the speech rights of individuals, even when speech is hateful. In both areas, he left his conservative colleagues, including Justice Samuel Alito Jr., alone in dissents.

Another factor that may motivate Roberts to shift more to the left in certain areas is an institutional concern that the Supreme Court not always appear to be ideologically divided—and in citizens’ minds, a partisan institution. A hard-right Trump justice could push Roberts to move left, even if ever so slightly.

“It partly turns on who the nominee is and how they get through the Senate,” said Jamil Jaffer, founder of the National Security Institute at George Mason University Antonin Scalia Law School Law and a former clerk to Gorsuch. “Those features—who the nominee is and how they think about their role and the law—have an impact on the court’s dynamics. In fact, it might take a few terms on the court to really know the impact of a new member.”

Jaffer added, “You can look at some features of recent cases. You do see Roberts at times, talking about the role of the court and the court as an institution. That’s not unusual for a chief justice but it also can point to factors that may weigh in his decisionmaking. That’s just an interesting feature of his jurisprudence.”

For some time now, the high court has had a core group of four justices whose opinions show a reach for common ground: Roberts, Kennedy, Stephen Breyer and Elena Kagan. Without Kennedy, the search will be harder.