During the COVID-19 pandemic earlier this year, Shira Shafir was trying to make vaccine appointments for some colleagues at UCLA.

Unfortunately, the first few times she used MyTurn, part of California’s technological response to the pandemic, the site proved difficult even for her, an adjunct associate professor at UCLA focused on epidemiology.

“I would get halfway through the process and then the website would glitch out,” Shafir said. Other times, appointments would appear and disappear.  “It’s a remarkable position to be simultaneously scientist and citizen,” she said. As a health scientist — one who was running training sessions for state workers enlisted as contact tracers — she knew the constraints the state was working under. As a citizen, she wanted to get an appointment for her 88-year-old mentor as quickly as possible.

Since the bumpy early months, though, Shafir said the site has improved dramatically. “One thing that is particularly impressive is how quickly they learned and adapted and went to fix those glitches, or those frustrations.”

Expectations for California’s digital response to the pandemic were high, given the talent in the state, said Dr. John Swartzberg, professor emeritus at UC Berkeley’s School of Public Health. The state didn’t meet those high expectations, he said, but it did pretty well given the technological capacity that state and county public health departments had at the beginning of the pandemic.

“The state of California and, frankly, every other public health department in the country started this pandemic with having been underfunded for decades, which translated to inadequate trained personnel, inadequate facilities, and, very importantly, inadequate digital tools,” said Swartzberg.

California built many new tech tools in response to the pandemic, often on a tight timeline. But the tools, such as MyTurn, the state’s vaccine portal, worked less well for older Californians, as well as those who lacked consistent internet access, or didn’t speak English fluently. Still, despite those and other complaints, experts that CalMatters spoke to thought the state did an overall positive job given the circumstances.

Here’s a look at the four of California’s tech responses to the COVID-19 pandemic.

Tool: COVID19.ca.gov, the state’s COVID-19 website

Cost: Staff time and $450,000 for translation into seven languages

In March 2020, staff at the Office of Digital Innovation created, in the space of four days, a website meant to serve as Californians’ primary source of information for the pandemic, according to J.P. Petrucione, the agency’s deputy director of communications.

The site now includes data on vaccine uptake by demographic groups, food and housing resources, guidelines for large events and more. As of late October it had more than 66.3 million unique visitors, Petrucione said, and users have averaged about three pageviews per person. About 40% of link clicks on the site went to MyTurn, another state tool. While staff built the site, the agency contracted with Avantpage to translate the site into seven languages. So far, that contract has cost $451,250, according to Petrucione.

Several public health experts and advocates said they found the site to be helpful in their own work and visited it frequently, but were less sure how widely used it was by citizens.

Swartzberg, at Berkeley, said he found the site useful and timely. Shafir said she’s been using the site on a daily basis. “I thought they started out very strong and very usable,” she said, and the case and vaccine data available has become even more robust. In trainings, she’s used the data dashboard to underscore to new contact tracers why asking about a person’s age or other health conditions is important.

Directing attorney Denny Chan at Justice in Aging, a national nonprofit fighting senior poverty, said the site was helpful for tracking infection and vaccination rates for older Californians, and for staying up-to-date on state policy changes. “It’s helpful for me as an advocate who’s helping others and helping inform our network about all these changes.” But, he said, “I don’t believe there were a ton of older adults themselves who are on this website getting information.”

Many older adults get information through radio and TV, but he credits the state with doing outreach through those mediums as well.

Tool: MyTurn

Cost: At least $50 million

The state’s vaccine appointment site may have been the most difficult to build, had the most troubled rollout and been the most important tech tool to Californians during the pandemic.

The site, conceived in January 2021 and launched the following month, was more than just a place to book appointments. As CalMatters reported in April, it also tracks vaccine orders and distribution, collects reams of vaccination data and helps organize volunteers statewide. And it can transmit data to the state’s immunization registry. In April, the state Public Health Department told CalMatters the site cost $50 million to build, including an $18 million contract with Accenture.

But it was the vaccine appointment-booking function of the site that most Californians interacted with — and which caused the most headaches.

Older Californians, who were among the first to qualify for the vaccine, experienced the site when it was at its least polished. “Because of the ways in which MyTurn and other appointment scheduling systems weren’t quite ready yet, there were a number of older adults that (said) ‘I’m just going to wait,’” said Chan.

Andrea Rivera, senior legislative advocate at California Pan-Ethnic Health Network, said the portal asked for insurance information, which caused confusion. Even though people could get the vaccine without health insurance, that question dissuaded uninsured people from making an appointment with MyTurn.

Some pages timed out, forcing people to start over, frustrating Californians who needed more time to fill out each page, said Andrew Imparato, executive director of Disability Rights California.

Soon after its launch, some people reported that they made appointments at the site, but due to glitches they were erroneous and were cancelled without notification.

Even health professionals like Shafir struggled to use MyTurn at the outset.

But, she said, now she thinks MyTurn works “spectacularly.” These days, she said, you can find appointments by vaccine type, location, age and more. Now “MyTurn has the functionality that we want it to have to really serve the population in California,” Shafir said.

Tool: Digital vaccine records

Cost: Staff time

Worried about what happens if you lose your federal vaccine card? California built an app for that. Conceived of in April 2021 and built by the Office of Enterprise Technology, this portal for Californians to get a digital record of their vaccination went live in June. The project didn’t involve any contracts with outside firms.

The vision, said Rick Klau, California’s chief technology innovation officer, was never to require people to sign up; it’s an optional tool. “Our goal was simply to ensure that residents had access to a copy if they lost or damaged their original (federal) card,” he said. In the first couple of weeks, a million people downloaded the QR code version of their vaccine record. As of mid-October, that number had grown to 6.5 million, according to Klau.

Some found that if they made a minor mistake entering their vaccine information, or if their vaccine administrator made a small error — if, say, they swapped the month and day in the vaccine date — the tool wouldn’t be able to locate their vaccine record until the information was corrected.

Klau said for people who need to correct information in their vaccine record, a virtual assistant allows them to upload their federal card, and there’s a phone number they can call for help.

Some may not be eager for a digital record, said Rivera, the advocate at California Pan-Ethnic Health Network. Undocumented immigrants, for example, may have less trust in state agencies and might be apprehensive about where their data will go.

Some privacy protections do exist. The resulting QR code can only be scanned by certain devices (if you try to scan it with your own smartphone, it won’t work), and signing up for the record does not grant someone automatic access. Also, the digital record only includes the information on your physical card; it does not include your Social Security number or citizenship status, for example. More information can be found on the digital vaccine record site and on the SMART Health Card site

The state Technology Department released its source code publicly, and three other states are in the process of building their own version based on California’s code, according to Klau.

Tool: CA Notify

Cost: $2 million

Apple and Google both created digital contact tracing tools in 2020 that use bluetooth signals to gauge when users’ phones are near each other; if people using one of the tools tests positive, they can choose to share that information. If they were near people for a certain period of time who also have the system enabled, those close contacts will receive a notification that they may have been exposed to the virus. Apple users can enable the digital contact tracing via a setting on their phone; Android users must download an app.

In September and November of 2020, the California Department of Technology oversaw a pilot program that rolled the apps out at a handful of University of California campuses. UC San Diego was the first campus to pilot the apps, and received a $2 million contract from the state Health Department that the department said went to provide ongoing support for the system, including managing a call center, web development and analytics. On Dec. 10, the system was rolled out statewide.

As of late October 2021, roughly 13.3 million Californians — out of the state’s 39 million — had activated or downloaded the twin products, according to the department. For privacy reasons, it isn’t able to track how many people have since chosen to disable the tool or delete the Android app, officials said.

Between the statewide launch in December 2020 and mid-October 2021, roughly 102,000 people who tested positive used the system to alert others, and approximately 309,000 people received alerts that they may have been exposed to the virus, according to the health department. Longhurst said modeling shows that the adoption of CA Notify in California may have prevented tens of thousands of infections.

Now that officials have relaxed some social distancing guidelines, the apps may prove even more valuable. During the lockdown, the average number of people getting notified per positive test reported was less than two, said Longhurst. Post-lockdown, it’s more than five.

“With COVID becoming sort of endemic, and you know, question mark about whether we’re going to have a big winter surge coming in December and January, now really is the time to turn this on,” Longhurst said.

Grace Gedye is a reporter with CalMatters.