Join the free 2-day Microsoft Change Agent course this month

Socitm’s AI benchmarking pilot and Coventry City Council’s use of AI (2024)

Authors and contributors: Socitm, Coventry City Council

Ahead of President’s Conference 2024, Socitm launched an AI benchmarking pilot to understand how AI is changing ICT delivery. There’s a growing need across the sector to measure AI implementation, and this benchmarking module sought to capture actual AI capabilities and performance across the membership, rather than just future AI aspirations.

The AI module gathered data to build a comprehensive benchmark, with metrics and questions covering staffing, costs, savings, governance, and automation.

Coventry City Council were one of the participants and during 2024’s President’s Conference, Tracy Ledwidge, ICT Operational Delivery Manager, shared their experiences of using and managing AI which informed their response to the AI benchmarking module. This included:

  • A three-month Microsoft 365 Copilot trial
  • Development of AI governance, policies and strategies
  • A six-week pilot using AI in social care
Discussion of AI Pilot from President’s Conference 2024, with Matthew Fraser (Socitm) and Tracy Ledwidge (Coventry City Council)

Challenges

Several challenges were identified by Socitm and Coventry regarding AI implementation:

  • There is a need for accreditation frameworks and best practices to guide organisations in their AI journey.
  • There are barriers to AI adoption, including a lack of skilled staff, understanding, and cost concerns.
  • There are also challenges with AI implementation. It is important to ensure the accuracy and reliability of AI-generated data.
  • And there is difficulty in proving cashable savings from AI tools.

Approach

Step 1 – Microsoft 365 Copilot trial

Coventry City Council began their AI journey using Microsoft 365 Copilot. As part of a three-month trial, they acquired 300 licenses for Copilot which were quickly snapped up by staff.

The pilot’s aim was to create enthusiasm and curiosity around AI while exploring potential savings.

During the trial, Coventry implemented practical steps to make the most of the licenses. They:

  • Held bi-monthly sessions with Microsoft to get the most out of the tool.
  • Hosted regular internal meetings and feedback gathering sessions with staff to understand what they did and did not like when using Copilot, and if there were any productivity or cash savings.

Step 2 – AI governance and policies

Coventry scored the lowest in the AI benchmarking module metric around AI governance and policies. However, they are making steps to improve this.

Coventry’s ongoing efforts to develop AI policies and strategies, were driven by their new, at the time, chief executive, Dr. Julie Nugent.

Coventry used Socitm’s sample corporate policy document for generative AI use within their organisation as a starting point for educating their workforce, addressing ethics and bias, and ensuring transparency in AI decision-making.

Step 3 – AI in Social Care

Coventry scored highly here when it came to the benefits of AI. Largely informed by their successful six-week pilot project in children’s services, which aimed to reduce administrative tasks and increase the time social workers spent with families.

Coventry:

  • Implemented a six-week pilot focusing on case notes, chronologies and forms, particularly the Supporting Families forms.
  • Used AI tools to automate and streamline administrative tasks.
  • Conducted regular audits and checks to ensure the accuracy of AI-generated data.
  • Engaged staff in the process, encouraging feedback and collaboration.

Outcomes and benefits

Step 1 – Microsoft 365 Copilot trial

Coventry found some benefits, including improving staff’s ability to focus more and contribute to meetings by producing meeting outputs. But they couldn’t prove cashable savings so they have since reduced their usage to fewer licenses. This could change in the future.

Step 2 – AI governance and policies

Coventry scored the lowest on AI governance in the benchmarking module. Despite having comprehensive discussions and understanding of the importance of having of education programs, ethics, data accuracy, and transparency, they hadn’t released any formal documentation at the time of the benchmarking survey, preferring to wait for a complete vision for AI within the council.

Step 3 – AI in Social Care

  • Successfully flipped the admin-to-direct-work ratio from 70:30 to 30:70.
  • Increased time for social workers to reflect and engage in early intervention with families.
  • Improved staff morale and reduced administrative burden.
  • Enhanced decision-making and early intervention capabilities.

Lessons identified

In these early days of using and analysing AI, immediate cashable savings may not be currently evident. But the long-term benefits in service quality and staff well-being are significant already.

  • Everyone is at a different stage of AI use – most are starting small.
  • Attitudes to risk and governance vary within and between organisations.
  • Few organisations are spending heavily on AI.
  • It is important to involve staff in the AI implementation process to ensure buy-in and collaboration.
  • There is a need for continuous auditing and validation of AI-generated data to maintain accuracy and reliability.

Additional resources

Socitm benchmarking services: You’re busy and benchmarking does take up some of your time. And that can make it the thing you keep meaning to do. Make 2025 the year you get round to it! This (calendar) year it won’t take up any of your budget, as for 2025 only, each member organisation has the complimentary use of one benchmarking module. You can choose between a cost, delivery, performance, or user skills module. Watch our video on benchmarking to learn more about what we offer.

AI@Socitm: Get practical help, skills and a deeper understanding of how you can apply AI responsibly into your public services with a live collection of curated resources.