The effort to oust the Democratic governor began before the year-long pandemic upended life in California, driven by conservative activists.