The City of San Jose is in the process of creating an Enterprise Data Analytics Strategy to streamline and improve the management of its data assets for fostering and better supporting data-driven business decision making and using data analytics and data sciences to achieve its goals of delivering equitable and efficient services to the residents and businesses of the city. 

The vision for this strategic initiative is to transform data into a service. The City has significant investments in operational transaction processing and case management systems. These systems are designed to support the business operations of various departments serving the public. With advances in information technology, the need to use data assets for analytics purposes has exposed the limitations of transaction processing systems in also serving applications that focus on measuring the efficiency, effectiveness, and results of services. This category of business intelligence, data analytics, and performance measurement applications requires a paradigm shift in how data are governed, stored, catalogued, curated, integrated, aggregated, and made available. 

At the core of this initiative is the development of a Cloud-based Data Lakehouse. The goal is to create a “single source for data” covering all services that the city provides for holistic, integrated, timely, and cost-effective data analytics purposes while maintaining data security and privacy requirements that shall govern all access to data. 

Under this “Data as a Service” model, decision makers in the City will be able to focus on analyzing and solving operational, management, and policy level problems that the City faces instead of spending time and resources on creating duplicate, siloed and isolated data infrastructure solutions. This will also include a data governance model allowing departmental data stewards to maintain full control and be responsible for the lifecycle management of their data assets.  

In addition to fostering and supporting data-driven decision making and analytics by City government managers and policy makers, the Enterprise Data Analytics Platform will also support the publication of pertinent data on the City’s Open Data Portal. This will eliminate standalone processes that only cover open data and will lead to a more efficient and broader scope for the City’s open data assets. 

The City has developed a strategy for implementing the Enterprise Data Analytics Platform. The strategy and the overall architecture for its implementation are based on Data Lakehouse best practices and proven models using non-proprietary software frameworks. A pilot project covering City of San Jose Department of Transportation operational data will be implemented by October 30, 2023, to allow the City’s information technology team to refine and finalize the roadmap for the full implementation of the Enterprise Data Analytics Platform.