AI Reviews & Inventory | City of San José

AI governance for San José

The City of San José is leveraging the benefits of AI to improve the delivery of services to residents. As the City increasingly uses AI tools, it is more important than ever to ensure that those AI systems are effective and trustworthy. See the City's AI Handbook. Watch the City discuss their implementation of AI accountability.

What is "AI", what is an algorithm?

AI stands for "Artificial Intelligence", or the idea of having a machine do something you would expect a human to do. AI is built by people creating "algorithms", which combine math and logical rules to handle tasks like translating languages (e.g., Google translate) and understanding your voice (e.g., Siri on Apple phones). By reviewing these algorithms, the Digital Privacy Office (DPO) ensures that the City's AI-powered technology acquisitions perform accurately, minimize bias, and are reliable.¹

San José AI Principles

The City follows eight guiding principles in its approach to AI.

Effectiveness: Systems are reliable, meet their objectives, and deliver precise and dependable outcomes for the utility and contexts in which they are deployed;
Transparency: The purpose and use of systems is proactively communicated and disclosed to the public. A system, its data sources, operational model, and policies that govern its use are understandable and documented;
Equity: Systems deliberately support equitable outcomes for everyone. Bias in systems is effectively managed with the intention of reducing harm for anyone impacted by the system’s use;
Accountability: Roles and responsibilities govern the deployment and maintenance of systems, and human oversight ensures adherence to relevant laws and regulations;
Human-Centered Design: Systems are developed and deployed with a human-centered approach that evaluates AI powered services for their impact on the public;
Privacy: Privacy is preserved in all AI systems by safeguarding personally identifiable information (PII) and sensitive data from unauthorized access, disclosure, and manipulation in accordance with the City Council’s Digital Privacy Policy;
Security & Safety: Systems maintain confidentiality, integrity, and availability through safeguards in accordance with the City’s Information and Systems Security Policy. The integrity of information into and out of the City is maintained in light of fake AI-generated content. Implementation of systems is reliable and safe, minimizing risks to individuals, society, and the environment; and
Workforce Empowerment: Staff are empowered to use AI in their roles through education, training, and collaborations that promote participation and opportunity.

AI Review Framework

When a City department wishes to procure an AI tool, the DPO follows the review process below to assess the benefits and risks of the AI system.

230905 AI Review Framework Diagram

Working with Vendors

When the City wishes to procure an AI system to improve service delivery, the DPO and procuring department work with vendors to ensure that the system is effective and trustworthy. As part of an effort to strengthen transparency, the DPO is piloting a process to work with vendors to complete a Vendor AI FactSheet that contains basic facts about the AI system, such as the data used to build the system and what conditions it performs well under. The Vendor AI FactSheet enables the DPO to better understand the technical details of the AI system and ultimately assess the risks and benefits it presents.

You can find the Vendor AI FactSheet template online. The Vendor AI FactSheet template is heavily inspired by the IBM Research AI FactSheets 360 project.

As the DPO continues to review AI systems, completed Vendor AI FactSheets and additional documentation will be added to this webpage in the section below.

AI Inventory

An AI Inventory is an important transparency mechanism that enables the City of San José to meaningfully communicate the AI systems that it uses to its residents. In January 2023, the DPO began maintaining an AI Inventory in an effort to increase transparency of the technological tools used by the City.

Below is a working inventory of the AI systems currently in use by the City of San José. For each catalogued system, you can find a basic description of its function and additional technical details collected in the Vendor AI FactSheet.

The DPO will update the AI Inventory with additional entries as more AI reviews are completed.

Google AutoML Translation

City staff use the Google AutoML Translation tool to translate customer messages in the San José 311 non-emergency helpline (SJ311) service. SJ311 can be accessed via phone, online, and through the SJ311 mobile application. City staff can customize the system to their specific needs by training the system on sentence pairs in English, Vietnamese, and Spanish.

DPO Review - Google AutoML

Review summary: Given the easy access to update the system, addressable set of consequences, and understandable accuracy metrics, this AI system is approved for usage in the City. The City should work to improve its Vietnamese to English translation.

Full Review by Digital Privacy Office

Using the City's Google AutoML translation system, residents can view the San José 311 website and file service requests in Vietnamese or Spanish. The City will receive the service request in English and respond in English. The response will then be translated into the resident's preferred language.

The translation system is built on Google's base translation model and further trained by City-provided data relevant to City service requests. Consequences for poor performance are limited to loss of services, and inaccurate translations can be fixed by the City manually updating the model.

Additionally, the system is easy to update and monitor for accuracy through the vendor's cloud platform. Based on standard evaluation metrics of translation AI (the BLEU metric), the system performs well in English to Spanish, Spanish to English, and English to Vietnamese. However, the system performs relatively poorly (though still usable) when translating from Vietnamese to English. The City should explore improving its Vietnamese to English translation database for better translations.

Given the easy access to update the system, addressable set of consequences, and understandable accuracy metrics, this AI system is approved for usage in the City. The City should work to improve its Vietnamese to English translation.

Vendor AI FactSheet - Google AutoML

A PDF version of the Vendor AI FactSheet is available here.

AI FactSheet
Vendor name	Google
Model name	AutoML Translation
Overview	The system is based on the standard Auto ML Google translation model. City staff can customize the system to their specific needs by training the system on sentence pairs in English, Vietnamese, and Spanish. The tool is used to translate customer messages in the San José 311 non-emergency helpline (SJ311) service. SJ311 can be accessed via phone, online, and through the SJ311 mobile application.
Purpose	The system is used to translate customer messages to and from English, Vietnamese, and Spanish in the SJ311 service’s chat function.
Intended Domain	Natural language processing
Training Data	The base model is trained on millions of examples of sentence pairs for 133 languages. The training data for the customized SJ311 model are sentence pairs for each language combination (English-Vietnamese, English-Spanish). In addition to basic language, the sentences feature vocabulary that is highly relevant for common SJ311 reporting areas (abandoned vehicles, illegal dumping, potholes, etc.).
Model Information	Auto ML Translation enables clients to perform supervised learning, which involves training a computer to recognize patterns from translated sentence pairs.¹ Using supervised learning, clients can train a custom model to translate domain-specific content they care about (i.e., San Jose city services).²
Inputs and Outputs	Input: A text sentence in English, Vietnamese, or Spanish. Output: Depending on the language translation needed, a text sentence in English, Vietnamese, or Spanish.
Performance Metrics	BLEU score (Vietnamese to English): 34.13 BLEU score (English to Vietnamese): 74.37 BLEU score (Spanish to English): 67.38 BLEU score (English to Spanish): 57.7 A rough guideline of BLEU score interpretation is below:
	BLEU Score	Interpretation
	< 10	Almost useless
	10 - 19	Hard to get the gist
	20 - 29	The gist is clear, but has significant grammatical errors
	30 - 39	Understandable to good translations
	40 - 49	High quality translations
	50 - 59	Very high quality, adequate, and fluent translations
	> 60	Quality often better than human
Optimal Conditions	No specific documentation provided. We can infer that the model will perform optimally in conditions similar to the training data it was provided. In this case, the model is the standard google translate model with additional training data on communications relevant to San José 311, or City services.
Poor Conditions	No specific documentation required. We can infer that the model will perform poorly in conditions not similar to the training data it was provided. In this case, the model is the standard google translate model with additional training data on communications relevant to San José 311, or City services. For example, the model may not perform well in translating conversations pertaining to legal matters unrelated to City services.
Bias	There appear to be a varying quality of performance across languages and the direction of translation (i.e., Vietnamese to English vs. English to Vietnamese). According to Google, trying to compare BLEU scores across different corpora and languages is strongly discouraged. Even comparing BLEU scores for the same corpus but with different numbers of reference translations can be highly misleading. The out-of-the-box Google Translate service has been shown to suffer from gender bias by changing the gender of translations when they do not fit with common stereotypes.³ Google has an Inclusive ML guide for clients as they use Auto ML systems for their customized applications that includes best practices to promote fairness in ML outcomes.⁴
Test Data	The testing data are sentence pairs for each language combination (English-Vietnamese, English-Spanish). In addition to basic language, the sentences feature vocabulary that is highly relevant for common SJ311 reporting areas (abandoned vehicle, illegal dumping, potholes, etc.).

Algorithmic Impact Assessment Questionnaire
Accuracy
Under what conditions/circumstances has the system been tested?	Testing data includes common vocabulary and vocabulary particularly relevant to typical SJ311 inquiries.
Have the vendors or an independent party conducted and published a validation report (including the methodology and results) that audits for accuracy and discriminatory/disparate impact? If yes, can the City review the study?	There are compliance reports from third parties, but no specific report published on the translation models.
Will the model be learning from the information it gets in the field during deployment?	Model can be trained by users with additional training data to customize for specific application. For example, San José adds data to improve performance on translations specific to City services. The base model is regularly updated by vendor.
Equity
What quality control is in place to test and monitor for potential biases in the AI system (e.g., non-representative training data, overfitting, hard-coded rules)?	Google’s Ethical AI team reviews new AI tools before they launch, with an eye towards possible bias that may be present in the system.⁵ Reviewers may recommend technical evaluations (e.g., checking for unfair bias in ML models). If necessary, they consult with trusted external advisors (e.g., human rights experts). Adjustments are made after reviewers offer mitigation strategies.
How can the City and its partners flag issues related to bias, discrimination, or poor performance of the AI system?	City can open a support case and Google Support will take the feedback into account.
Explainability
What performance metrics were selected to judge the model’s effectiveness? What is it optimizing for, and under what constraints?	BLEU metric (specific to Translation) Google’s confidence score (relevant to any Auto ML prediction)
How are the outcomes of the AI system explained to subject matter experts, users, impacted individuals, or others?	Each translated sentence is presented with a confidence score from 0-100% that shows the model’s “confidence” in that translation. While the number might not be a perfect interpretation of the BLEU score, it provides individuals with the gist of how good a translation likely was.

¹https://cloud.google.com/translate/automl/docs

²https://cloud.google.com/translate/automl/docs/beginners-guide

³ https://algorithmwatch.org/en/google-translate-gender-bias/

⁴https://cloud.google.com/inclusive-ml

⁵ https://ai.google/responsibilities/review-process/

AIA Form - Google AutoML

The AIA Form is completed by City staff during the AI Review process. City staff responses to the AIA Form for Google AutoML Translation can be found below.

A PDF version of the AIA form is available here.

Project Objective

Question	Response
Please describe the project use case, the current process, and the desired outcome.	When residents creates a service request on the SJ 311 platform (https://311.sanjoseca.gov), if the residents preferred language is set either to Spanish or Vietnamese, the residents received email communication in their preferred language. The call center views this service request in English. While the call center responds in English, the resident receives the email translated in the preferred language. This is achieved using Google AutoML. The SJ 311 website supports in both Vietnamese and Spanish. The website is translated using a free Google widget.
Owning Department	Information Technology
Why does your department choose automation as an approach to this problem? What other approaches to solving this problem were considered (if any) and what led to choosing automation?	57% of the residents in San Jose speak a language other than English at home. To cater to this growing population, SJ 311 web can be viewed in both Spanish and Vietnamese. San Jose 311 has only been available in English since its 2017 launch. October 2020 we introduced the Spanish and Vietnamese language preferences. The website can be translation into these two languages based on the residents choice. The City had reservations about using free tools(Oracle provided out of the box translation) based on past experiences: Default translation was removed after residents complained that some translations were offensive. That is when City adopted the Google free translate widget and the AutoML solution.

Question

Response

Please describe the project use case, the current process, and the desired outcome.

When residents creates a service request on the SJ 311 platform (https://311.sanjoseca.gov), if the residents preferred language is set either to Spanish or Vietnamese, the residents received email communication in their preferred language. The call center views this service request in English. While the call center responds in English, the resident receives the email translated in the preferred language. This is achieved using Google AutoML.

The SJ 311 website supports in both Vietnamese and Spanish. The website is translated using a free Google widget.

Owning Department

Information Technology

Why does your department choose automation as an approach to this problem? What other approaches to solving this problem were considered (if any) and what led to choosing automation?

57% of the residents in San Jose speak a language other than English at home. To cater to this growing population, SJ 311 web can be viewed in both Spanish and Vietnamese.

San Jose 311 has only been available in English since its 2017 launch. October 2020 we introduced the Spanish and Vietnamese language preferences. The website can be translation into these two languages based on the residents choice. The City had reservations about using free tools(Oracle provided out of the box translation) based on past experiences: Default translation was removed after residents complained that some translations were offensive. That is when City adopted the Google free translate widget and the AutoML solution.

Vendor Details

Question	Response
Will the algorithm be designed, developed, deployed, or maintained by vendors or third parties?	Yes, by both vendors/third parties and City staff
How can the City test the vendor’s algorithm before it is put into use?	Vendors were hired and City employees were engaged for correctness of the Google AutoML translation.

Transparency

Question	Response
How do individuals receive a notice in advance of interacting with the system? For example, if a user is interacting with a chatbot, the system lets the user know they are talking to a chat bot instead of a human.	The individuals would not be able to notice any difference as the translation is seamless. Although the correctness of the translation maybe an issue in some cases.
How can third-party auditors easily view the system’s data in order to perform evaluations?	We have a csv files of the name-value pairs of the translation. The file contains, English-to-Spanish, Spanish-to-English, English-to-Vietnamese, Vietnamese-to-English name-value pairs.
How will system operators or residents know if the system outputs an error? What ability will they have to correct or appeal an error?	We have a form we ask users to fill out if they find any incorrect translations on the website.

Equity

Question	Response
What individuals and communities will interact with the system? For example, is the algorithm used on the general City population (technology used in many public areas) or a specific group (e.g., children in a school program, a single neighborhood)?	The SJ 311 website is accessible by anyone with access to the internet. The SJ 311 website targeted for San Jose residents and businesses. You may or may not require a SJ 311 account to access the services the City has to offer.
How likely is it that the system impacts children under the age of 18?	Not likely.
How is this use case, and the information/decisions provided by the algorithm, related to an individual’s right or freedoms (e.g., if the algorithm helps determine if a suspect can be put on bail or must remain in jail)?	Not applicable.
How is this use case, and the information/decisions provided by the algorithm, related to an individual’s economic status (e.g., if the algorithm helps determine if an individual can apply to affordable housing)?	Not applicable.
How is this use case, and the information/decisions provided by the algorithm, related to an individual’s health, healthcare, well-being (e.g., if the algorithm helps determine an individual’s likeliness for colon cancer)?	Not applicable.
How do decisions from the system impact the environment, if at all (e.g., potential impact to carbon emissions, high tech waste)?	Not applicable.
What issues could arise if the algorithmic system is inaccurate?	There could be some angry residents if the language translation is not translated correctly and accurately.

Human Oversight

Question	Response
Please describe the level of autonomy of the system.	System operates automatically with occasional retrospective reviews by humans.
If there is human intervention in the system, is it by the vendor, City department/office, or both?	Both
Please list the roles/divisions that will be “touching” the system, or managing the deployment and use of the system.	Technical lead, Products-Project Manager, Technical Manager from the ITD.
How does the Department provide training and resources to personnel to help them develop the skills they need to effectively operate the system?	We have a zoom session with the SME, in addition to documentation located in Sharepoint site.
In the event that the system does not work or is deemed to be inaccurate, what back-up measures are in place to ensure that the Department can continue to deliver services?	In case of language translation, the corrections are made to the text that is translated incorrectly

LYT.transit Transit Signal Priority System

The Department of Technology oversees the Central Transit Signal Priority project, which provides signal priority for VTA bus routes 66 & 68 for all intersections along the route within City of San José jurisdiction. The project uses the “LYT.transit” system to implement transit signal priority. Transit signal priority gives buses priority at an intersection and creates less idle time waiting for a green light. The LYT.transit system tracks transit vehicles in real-time and communicates with downstream intersections to optimize signal timing, reducing transit vehicle travel time.

DPO Review - LYT.transit

Review summary: Given the demonstrated reduction in travel times, minimal bias of the training data, and ability to view real-time performance metrics, this AI system is approved for usage in the City. The City should continue to monitor the effectiveness of the system as defined by bus travel time before and after LYT.transit implementation. If the system continues to show benefit, the City should explore applying the project to other routes.

Full DPO Review

The Department of Technology is overseeing the Central Transit Signal Priority project, which provides signal priority for VTA bus routes 66 & 68 for all intersections along the route within City of San José jurisdiction. Transit signal priority gives buses priority at an intersection and creates less idle time waiting for a green light. Ultimately, the goal is to reduce travel time and alleviate traffic congestion.

The project uses the “LYT.transit” system to implement transit signal priority. The LYT.transit system tracks transit vehicles in real-time and communicates with downstream intersections to optimize signal timing, reducing transit vehicle travel time. It is built using a supervised machine learning model. A study done on a pilot project in San Jose during 2019 demonstrated that the LYT.transit system may reduce travel times by more than 15%.

The LYT.transit system performs best when the vehicle position data is highly accurate and frequently updated. Performance of the system will likely be poorer if a vehicle's GPS equipment loses accuracy over time or there is poor cellular communication between onboard vehicle equipment and the transit agency data center. The primary consequence of poor performance is lengthier travel times and traffic congestion.

Since the LYT.transit system predicts bus arrival time based on GPS data, there is relatively minimal human bias in the training data. The effectiveness of the system can be measured by comparing the travel time before implementing LYT.transit to the travel time using LYT.transit. With a new software update from LYT.ai expected in 2023, City staff will be able to see this performance metric in real-time.

Given the demonstrated reduction in travel times, minimal bias of the training data, and ability to view real-time performance metrics, this AI system is approved for usage in the City. The City should continue to monitor the effectiveness of the system as defined by bus travel time before and after LYT.transit implementation. If the system continues to show benefit, the City should explore applying the project to other routes.

Vendor AI FactSheet - LYT.transit

A PDF version of the Vendor AI FactSheet is available here.

AI Factsheet
Vendor name	Sinwaves Inc. d/b/a LYT
Model name	Transit vehicle ETA estimator
Overview	The LYT.transit system tracks transit vehicles in real-time and communicates with downstream intersections to optimize signal timing, reducing transit vehicle travel time.
Purpose	This model generates the estimated time of arrival of a transit vehicle at intersections along its route.
Intended Domain	Transportation / transit
Training Data	The model is trained on transit vehicle location, route, and schedule adherence data.
Model Information	The model is a supervised machine learning model, specifically a regression model.
Inputs and Outputs	Inputs: vehicle position, speed, route, schedule adherence Outputs: array of estimated time of arrivals in seconds to the upcoming intersections
Performance Metrics	MAE (mean absolute error) of predicted ETA against test data is used to evaluate the model. Model is trained on a weekly basis as new vehicle data is gathered from deployed transit vehicles.
Optimal Conditions	High accuracy of vehicle position data Frequent updates of vehicle data
Poor Conditions	GPS drift in vehicle GPS equipment Poor cellular communication between onboard vehicle equipment and transit agency data center
Bias	Since this is a model that predicts times of arrival for buses based on GPS updates and other geospatial and timing information, many of the problems of human bias in training data do not enter into this model’s operation. The training set consists of all bus trajectories (within some time window) for exactly those routes upon which the model will be operating during live inference.
Test Data	The model is tested against past transit vehicle position and schedule adherence data.

Algorithmic Impact Assessment Questionnaire
Accuracy
Under what conditions/circumstances has the system been tested?	The LYT.transit system has been tested internally with a corpus of past transit vehicle trajectory data as well as on the field, by intersection and by route.
Have the vendors or an independent party conducted and published a validation report (including the methodology and results) that audits for accuracy and discriminatory/disparate impact? If yes, can the City review the study?	Yes. LYT has published a report on the first deployment of the technology here: https://mailchi.mp/07b1e3eba2b4/llgp2f8piv
Will the model be learning from the information it gets in the field during deployment?	Yes. LYT collects real-time vehicle data from transit vehicles and uses it to iteratively retrain its ETA prediction models.
Equity
What quality control is in place to test and monitor for potential biases in the AI system (e.g., non-representative training data, overfitting, hard-coded rules)?	It is known in advance on which routes LYT’s model will operate. Accordingly, only relevant trips can be selected for training data. Additionally, training data is preprocessed to remove outlier trips. Model testing is conducted against data that occurred after the training data in order to protect against time leakage in evaluating model performance.
How can the City and its partners flag issues related to bias, discrimination, or poor performance of the AI system?	LYT.transit provides a web portal to each customer to show the results of the LYT.transit system and its impact on transit performance in the form of reports and graphs.
Explainability
What performance metrics were selected to judge the model’s effectiveness? What is it optimizing for, and under what constraints?	Mean average error is the primary metric for ETA model evaluation. The model is optimizing for minimal error between its predicted time of arrival and the true time of arrival.
How are the outcomes of the AI system explained to subject matter experts, users, impacted individuals, or others?	The LYT.transit system directly impacts transit route performance, which can be measured in terms of on-time performance using the transit agency's existing performance monitoring systems or LYT's web portal.

AIA Form - LYT.transit

The AIA Form is completed by City staff during the AI Review process. City staff responses to the AIA Form for the LYT.transit Transit Signal Priority System can be found below.

A PDF version of the AIA form is available here.

Project Objective

Question	response
Please describe the project use case, the current process, and the desired outcome.	The Central Transit Signal Priority (CTSP) project provides signal priority for VTA bus routes 66 & 68 with all intersections along the route within City of San Jose jurisdiction. TSP allows buses to get priority at an intersection and provides less idle time waiting for a green light. At this stage of the project, we have implemented TSP at 121 of the 122 intersections along both routes with additional fine-tuning/optimization of the transit signal timing parameters scheduled for late January-early February. The desired outcome of the project is to optimize traffic flow by providing automated signal timing for buses, prioritize traffic related to transit services to improve viability of using public transportation by reducing bus travel time and alleviation traffic congestion, and reduce air pollution.
Owning Department	Department of Transportation
Why does your department choose automation as an approach to this problem? What other approaches to solving this problem were considered (if any) and what led to choosing automation?	There has always been some sort of automation with TSP. One method involves physical equipment in the traffic signal cabinet and antenna on the signal pole to communicate to a GPS device in the bus to provide a "zone" when to send a call to the controller to provide priority call. Everything for this project is done with no physical equipment required and only requires communication to our traffic signal controllers which is something we had in place already. TSP calls are done through LYT's cloud which can then send a remote call to the controller based on past estimated arrival times to the intersection and tracking of VTA buses within the system. Only other approach is to manually place TSP calls when buses approach the intersection which would require more staffing to monitor and observe buses real-time which is unfeasible.

Vendor Details

question	response
Will the algorithm be designed, developed, deployed, or maintained by vendors or third parties?	Yes, solely by vendors/third parties
How can the City test the vendor’s algorithm before it is put into use?	We did several test runs by riding the bus and comparing how LYT's system is aggregating the estimated arrival times to an intersection compared to when we see the bus arrives at an intersection. This way we can make sure LYT's system is accurately determining when the bus will arrive to provide TSP at the correct time.

Transparency

question	response
How do individuals receive a notice in advance of interacting with the system? For example, if a user is interacting with a chatbot, the system lets the user know they are talking to a chat bot instead of a human.	Users of the system is mainly City staff, LYT, and VTA. While there is no advance notice of interacting with the system, City staff can monitor when TSP calls are being placed by using our Advanced Traffic Management System Transcore TCS, logging into the LYT portal to check when buses provide a TSP call to the intersection, or monitoring the controller at the intersection. System runs independently so any interaction with humans would be done by contacting LYT and setting up meetings for troubleshooting.
How can third-party auditors easily view the system’s data in order to perform evaluations?	Auditors can get a login to the LYT portal to access the system which provides real-time bus location, signal status for TSP, and estimated arrival time when bus will arrive at the next intersection.
How will system operators or residents know if the system outputs an error? What ability will they have to correct or appeal an error?	While system operators or residents won't know when something is not working with the system, they will be able to tell something is wrong if buses are waiting at an intersection for a long time or an intersection is out of normal operation (signal flash).

Equity

Question	response
What individuals and communities will interact with the system? For example, is the algorithm used on the general City population (technology used in many public areas) or a specific group (e.g., children in a school program, a single neighborhood)?	The end user of the VTA bus route will be residents of San Jose that live and work and use the two bus routes. Individuals will not interact with the LYT system or traffic signal.
How likely is it that the system impacts children under the age of 18?	Many schools are along the route so children under the age of 18 will use the bus route for school and/or work.
How is this use case, and the information/decisions provided by the algorithm, related to an individual’s right or freedoms (e.g., if the algorithm helps determine if a suspect can be put on bail or must remain in jail)?	N/A
How is this use case, and the information/decisions provided by the algorithm, related to an individual’s economic status (e.g., if the algorithm helps determine if an individual can apply to affordable housing)?	N/A
How is this use case, and the information/decisions provided by the algorithm, related to an individual’s health, healthcare, well-being (e.g., if the algorithm helps determine an individual’s likeliness for colon cancer)?	N/A
How do decisions from the system impact the environment, if at all (e.g., potential impact to carbon emissions, high tech waste)?	The project will have a positive impact to the environment. The project will optimize traffic flow by automating optimal signal timing, proactively prioritize traffic related to transit services to improve the viability of using public transportation by reducing bus travel times and alleviating traffic congestion of single occupancy vehicles, and reduce air pollution.
What issues could arise if the algorithmic system is inaccurate?	Signal timing will cause longer delays for vehicles waiting along side streets the do not benefit TSP. Also, no communication to the traffic signal controller will cause the bus to get no TSP which will mess up the ultimate goals of the project.

Human Oversight

question	response
Please describe the level of autonomy of the system.	System operates automatically with occasional retrospective reviews by humans
If there is human intervention in the system, is it by the vendor, City department/office, or both?	Both
Please list the roles/divisions that will be “touching” the system, or managing the deployment and use of the system.	LYT - Overall management of the system ITS - Monitoring communications to traffic signals Signal Operations - Overall management of signal timing, optimization, and performance of the system
How does the Department provide training and resources to personnel to help them develop the skills they need to effectively operate the system?	Part of the contract includes a couple of training sessions with LYT to allow City staff to understand the system and day-to-day settings to monitor.
In the event that the system does not work or is deemed to be inaccurate, what back-up measures are in place to ensure that the Department can continue to deliver services?	TSP can be turned off and buses will perform the same as was done before the project started. There would be no TSP calls and buses would wait at a signal similar to how signals with no TSP is used.

Zabble Waste Contaminant Identification System

City staff use the Zabble Waste Contaminant Identification System to measure the fullness and contents of trash cans. The system aims to make it faster to identify waste content and aggregate insights across inspections.

DPO Review - Zabble

Environmental Services Division (ESD) is requesting an AI solution that can identify fullness and contents of trash cans to support reducing recycle bin contamination. This information will not be used levy fines or penalties. It will be used to inform future outreach, such as a follow up letter and/or other educational outreach as needed.

Factsheet was requested given computer vision component plays significant role in application.

No PII is involved and AI risk is low given it will only inform general outreach strategy.

Vendor AI FactSheet - Zabble

A PDF version of the Vendor AI FactSheet is available here.

Vendor Name
Zabble Inc.
System Name
Zabble Zero Mobile Tagging
Overview
Zabble is a technology company that helps organizations efficiently manage their waste and workflows using an AI-powered platform, Zabble Zero. One of its products, Mobile Tagging, is a mobile- and web based app to conduct efficient audits using computer vision AI to measure fullness and detect contaminants and engage stakeholders with real-time alerts.
Purpose
Zabble’s AI automatically identifies a bin’s fullness and its contaminants in real-time through a mobile phone’s camera via the Mobile Tagging app. This allows the users to not only spend less time documenting a bin’s fullness and contaminants but also access aggregated insights across the inspections.
Intended Domain
Waste Management
Training Data
The model is trained using top-down images of waste receptacles from customer deployments of the platform in various environments. Images are labeled by professional annotators.
Test Data
The models are tested on a holdout set of customer images, labeled according to the same methodology as the training data. The models are implemented in Zabble Zero’s mobile-based app. Prior to deployment, the models are tested in a test version of the app in a close proximity to the customer environment.
Model Information
Zabble's AI includes object detection and image classification models. The base model for object detection is YOLOv5. There are two image classification models, one for predicting interior fullness and one for predicted exterior fullness. The base model for fullness classification is ResNet18. All models are fine-tuned using Zabble’s training dataset.
Update Procedure
Models are trained twice a year when a critical mass of data has been added. Users are encouraged/required to use the latest version of the Zabble Zero app, so they do not have the option to use an older version of the models.
Inputs and Outputs
1. Inputs: A jpeg image file at least 480x680 pixels (images are resized by the models)
2. Outputs: A JSON of the predicted fullness classes and probabilities; a JSON with a list of objects detected, their bounding coordinates, and the model confidence
Performance Metrics
The deployed models are evaluated on new customer images labeled according to the same methodology as the training and test sets. Zabble computes these metrics overall, by customer, and by receptacle type.
1. For fullness, accuracy and adjacent accuracy (within 10% of the labeled fullness range) are evaluated.
2. For object detection, Zabble reports metrics on the existence of an item in an image. Sensitivity and specificity are reported.
Bias
Human-factor bias is not relevant to Zabble Zero AI. Sampling bias may be present, as Zabble’s training dataset relies on the images of its current and past customers.
Robustness
The Zabble app instructs users to review the outcomes of the AI’s predictions and make adjustments where needed. Zabble’s regular AI evaluations examine areas where the model is not performing well, and seeks to improve this performance in the next version of the model.
Optimal Conditions
Zabble AI performs best on clear images with sufficient lighting, no blurriness, when the top layer of the contents of the receptacle can be seen, and on receptacles where most of the edges of the container are visible.
Poor Conditions
Zabble AI does not perform well on low-light, blurry images, when the contents of the receptacle are otherwise difficult to see, or when it is difficult for a human to estimate the fullness of a container because the top edges are not visible or most of the receptacle is not in view.
Explanation
In live prediction mode before the photo is taken, the predicted fullness % and the boxes and labels of detected items are shown on the camera screen. After taking the photo, the app presents the fullness prediction in the form of a slider, which the user is expected to adjust when not accurate. For detected objects, the app provides these written instructions: “Confirm any items detected from the image or commonly tagged by other users.”

Algorithmic Equity Assessment Questionnaire

How is the AI tool monitored to identify any problems in usage? Can outputs (recommendations, predictions, etc.) be overwritten by a human, and do overwritten outputs help calibrate the system in the future?
The Zabble app instructs users to review the outcomes of the AI’s predictions and make adjustments where needed. Zabble’s regular AI evaluations examine areas where the model is not performing well, and seeks to improve this performance in the next version of the model.
Have the vendors or an independent party conducted a study on the bias, accuracy, or disparate impact of the system? If yes, can the City review the study? Include methodology and results.
A summary of Zabble’s most recent fullness model evaluation is available here: https://www.zabbleinc.com/blog-post/latest-zabble-zero-ai-fullness-predictions

Zabble’s evaluation of its previous object detection model is available here: https://www.zabbleinc.com/blog-post/zabblezero-contamination-object-detection-accuracy
**NOTE: A newer report of the results has currently not been published.
Is the data used to the train the system representative of the communities it covers?
Zabble’s training dataset uses the images of its current and past customers, which is generally representative of other images of waste receptacle data. However, there may be differences in the types of waste/recycling materials present and/or the types of containers holding these materials..
How can the City and its partners flag issues related to bias, discrimination or poor performance of the AI system?
Feedback can be provided by contacting the Zabble Customer Success Manager or by emailing support@zabbleinc.com.
How is the AI tool made accessible to people with disabilities?
People with mobility, auditory, speech, or mild cognitive disabilities can use the Zabble Zero platform with little to no accommodations. The Zabble app is designed for visual waste audits, therefore, people with significant visual impairments may be unable to use the technology.
What other human factors, if any, were considered for usability and accessibility of the system?
Zabble designs for ease-of-use to allow users with only basic training to use the mobile app.

Wordly Real-time AI translation and speech-to-text

The City is using an automated translation system to support translation in live meetings, such as City Council meetings. Wordly allows the City to provide translated text and speech between more than 40 languages including Spanish, Vietnamese, English, Tagalog, and Mandarin. The system will integrate with Zoom and other meeting software so residents can use familiar tools (e.g., Zoom) to the translations in real-time.

DPO Review - Wordly

The translation system is built on existing 3rd-party models for converting text to speech, speech to text, and translating text. The system works as follows:

Someone speaks in one language
That speech is converted into text
The text is translated into another language
The text is spoken in the translated language to the listener

The system is evaluated using standard evaluation metrics, such as the BLEU score for translation and word accuracy rate for speech-to-text. Wordly also incorporates qualitative metrics like user reviews of the system. These metrics are monitored by Wordly to maintain a baseline level of quality.

Wordly offers the City the ability to manually set certain translations, for example, always having “No tengo ganas” translate to “I can’t be bothered”. The City can also set certain speech-to-text/text-to-speech conversion. For example, making sure “José” in San José is pronounced correctly.

The consequences of the system failing is a lack of accessibility for non-English speakers. If the system is no longer working, the City can rely on people translating meetings. Additionally, staff can manually correct the system if it provides a misleading or incorrect translation.

Residents will be notified when translations are provided by an automated system as opposed to a person.

Given the providision of clear notice, known consequences for poor performance, and ability to adjust the model to meet our City-specific needs, this Artificial Intelligence system is approved.

AI Factsheet - Wordly

Download a pdf version of the AI FactSheet.

Vendor Name
Wordly Inc.
System Name
Wordly Transcription and Translation
Overview
The Wordly system makes real-time, spoken language transcription and translation available around the globe. Wordly can support live conferences and virtual meetings of any size, from 2 to 10,000 in over 50 different languages.
Purpose
The primary purpose of the Wordly system is to transcribe and translate spoken language in real-time.
Intended Domain
Wordly is intended to handle all domains of human discourse.
Training Data
Wordly uses models trained by third parties on data they collect. No Wordly customer data is ever used. Different models are used for different languages and all third parties involved stipulate that their data is collected legally.
Test Data
Wordly uses proprietary data sets to test all AI models it uses. This data is not obtained from customers and is collected legally.
Model Information
The AI models Wordly uses for transcription and translation are generally large transformer models.
Update Procedure
The AI models used by Wordly are updated regularly. Wordly tests all models at least quarterly and updates the models in use when new models test better. Updates are generally transparent to customers and they have no need to opt into improvements.
Inputs and Outputs
The input to the Wordly system is spoken audio from live speakers or recorded content. Outputs are text transcriptions and translations and spoken translations if desired. Customers can select from a variety of different interfaces for consuming Wordly services, depending on their specific needs.

Performance Metrics
Measured performance metrics for transcription are Word Error Rate and for translation BLEU score. For the majority of languages, WER is generally under 10% and for the 5 most commonly used languages is under 5%. BLEU scores do not compare well across languages but all languages rate a “good” level and the 5 most common languages are all “excellent”.

Bias
The models used by Wordly are not known to exhibit any particular bias except when translating from ungendered to gendered languages there is a bias toward masculine pronouns.
Robustness
The primary robustness measures are the performance metrics discussed above. When customers encounter issues with transcription or translation of unusual words or phrases (like proper names or acronyms) they are able to compensate by creating a glossary for those terms.
Optimal Conditions
Optimal conditions for the Wordly system occur when input audio is clean and clear with little background noise or music.

Poor Conditions
Poor conditions include very noisy audio or loud background music. In such situations Wordly can have trouble transcribing the original speech. However, the system is generally quite robust even in poor audio situations.
Explanation
The nature of the Wordly system does not require that results be explained. The goal is a faithful transcription and translation.

Algorithmic Equity Assessment Questionnaire

How is the AI tool monitored to identify any problems in usage? Can outputs (recommendations, predictions, etc.) be overwritten by a human, and do overwritten outputs help calibrate the system in the future?
Wordly systems are monitored 24 hours a day to ensure they are working as intended. Wordly transcription and translation accuracy is measured on a regular basis to identify problems or opportunities for improvement. Customers are able to provide feedback via email at any time.
How is bias managed effectively?
Customers can report bias via email or phone and these reports are monitored and dealt with as appropriate.

Have the vendors or an independent party conducted a study on the bias, accuracy, or disparate impact of the system? If yes, can the City review the study? Include methodology and results.
In general, bias that affects accuracy or reliability is not present in the type of models used by Wordly.

How can the City and its partners flag issues related to bias, discrimination or poor performance of the AI system?
Issues related to bias and accuracy can be reported to Wordly via email or phone.

How has the Human-Computer Interaction aspect of the AI tool been made accessible, such as to people with disabilities?
Wordly makes it services available through a variety of interfaces suited to different customer needs.
Please share any relevant information, links, or resources regarding your organization’s responsible AI strategy.
Visit https://www.wordly.ai for general information about the Wordly system, and https://www.wordly.ai/blog/trusted-ai-translation for specific information about our AI policies.

AIA Form - Wordly

The AI Impact Assessment (AIA) Form is completed by City staff during the AI Review process. City staff responses to the AIA Form for the Wordly real-time translation system can be found below.

Download A PDF version of the form.

Project Objective

Question	response
Please describe the project use case, the current process, and the desired outcome.	Wordly offers to translate council meetings and committee meetings in real-time to other languages
Owning Department	Clerk's Office
Why does your department choose automation as an approach to this problem? What other approaches to solving this problem were considered (if any) and what led to choosing automation?	Automation is a magnitude cheaper and allows us to translate into several different languages that we otherwise could not. This way we can provide translation at more meetings, such as committee meetings, than we can previously.

Vendor Details

question	response
Will the algorithm be designed, developed, deployed, or maintained by vendors or third parties?	Yes, solely by vendors/third parties
How can the City test the vendor’s algorithm before it is put into use?	The City is testing the system prior to implementation with City staff.

Transparency

question	response
How do individuals receive a notice in advance of interacting with the system? For example, if a user is interacting with a chatbot, the system lets the user know they are talking to a chat bot instead of a human.	The users will opt-in to using the translation service which is integrated into Zoom's platform.
How can third-party auditors easily view the system’s data in order to perform evaluations?	Auditors can see the transcript from the zoom meeting and compare with the english audio of the meeting.
How will system operators or residents know if the system outputs an error? What ability will they have to correct or appeal an error?	Errors will be identified by residents and periodic staff checks of translation. Can be flagged to City staff which can adjust what certain phrases mean manually.

Equity

Question	response
What individuals and communities will interact with the system? For example, is the algorithm used on the general City population (technology used in many public areas) or a specific group (e.g., children in a school program, a single neighborhood)?	Primarily non-english speaking residents
How likely is it that the system impacts children under the age of 18?	Somewhat, if they are listening to council meetings.
How is this use case, and the information/decisions provided by the algorithm, related to an individual’s right or freedoms (e.g., if the algorithm helps determine if a suspect can be put on bail or must remain in jail)?	N/A
How is this use case, and the information/decisions provided by the algorithm, related to an individual’s economic status (e.g., if the algorithm helps determine if an individual can apply to affordable housing)?	N/A
How is this use case, and the information/decisions provided by the algorithm, related to an individual’s health, healthcare, well-being (e.g., if the algorithm helps determine an individual’s likeliness for colon cancer)?	N/A
How do decisions from the system impact the environment, if at all (e.g., potential impact to carbon emissions, high tech waste)?	N/A
What issues could arise if the algorithmic system is inaccurate?	Residents get poorly translated council meetings.

Human Oversight

question	response
Please describe the level of autonomy of the system.	System operates automatically with occasional retrospective reviews by humans
If there is human intervention in the system, is it by the vendor, City department/office, or both?	City department/office
Please list the roles/divisions that will be “touching” the system, or managing the deployment and use of the system.	City Clerk's Office, City Manager's Office
How does the Department provide training and resources to personnel to help them develop the skills they need to effectively operate the system?	System is fairly straightforward, residents will click the translate button on the zoom page to use the system. Staff will be provided training to use the system by the vendor starting roll-out.
In the event that the system does not work or is deemed to be inaccurate, what back-up measures are in place to ensure that the Department can continue to deliver services?	Provide manual translation

Traffic technology supported by AI

The City uses AI systems to support traffic safety in two main ways:

Collecting data on traffic conditions, such as the number of cars passing an intersection, number of people walking through a crosswalk, and number of near-incidents (e.g., a car almost hitting a pedestrian)
Supporting traffic enforcement and criminal investigations, typically using cameras that can read license plates.

You can see how these systems are used and reports on their accuracy on the Data Usage Protocols page. See the following technology:

Automated License Plate Readers
Vision-based traffic data collection and safety analytics device using artificial intelligence
Red Light Running Cameras
Road safety detection pilot

Additional Algorithmic Systems

People counter cameras at parks, trails, and community centers

The Parks, Recreations and Neighborhood Services Department (PRNS) installed cameras that count the number of people entering and exiting a facility. These cameras provide data that is used to better understand pedestrian traffic and PRNS facilities to support the educational, health, and life outcomes of San José youth and their families, and potentially support incident prevention. The collected data will provide increased knowledge of our facility use and trends to understand visitors' attendance/traffic and enhance their experience. These cameras do not record and do not store any video.

See more detail on the Data Usage Protocols webpage.

Anonymized Foot Traffic Data

Measuring how our community uses facilities is important for the equitable delivery and maintenance of parks, public spaces, and recreational areas. The Parks, Recreation and Neighborhood Services Department (PRNS) is using anonymized foot traffic data to understand where and when people access recreational amenities. We know that this access is uneven across many of our neighborhoods. Systemic and institutionalized racial exclusion and disinvestment create and perpetuate disproportional access to quality parks. This data is another tool to help understand where there is a greater need for new, improved, and safer facilities for equitable resource allocation.

See more detail on the Data Usage Protocols webpage.

Gunshot Detection System

The San José Police Department (SJPD) is piloting the implementation of a gunshot detection system pilot. Police officers may utilize gunshot detections systems to enhance their ability to respond to potential firearm crimes. Gunshot detection systems recognize the typical sound of a gunshot or similar sound (such as breaking glass) and alert police of the sound. When used with existing Automated License Plate Readers (ALPR) to capture photos of passing vehicles, gunshot detection systems can identify the time, location, and associated vehicles surrounding a firearm incident. The integrated use of gunshot detection systems and ALPR can increase police officers’ capacity to respond to incidents of gun violence.

See more detail on the Data Usage Protocols webpage.

Notes

For our purposes, we use “artificial intelligence” and “algorithmic system” interchangeably. The City defines an algorithmic system to be any system, software, or process that automatically generates outputs including, but not limited to, predictions, recommendations, or decisions that augment or replace human decision-making.