Australia’s Ministry of Finance conducted a 14-week test on Microsoft 365 Copilot to assess the effectiveness of AI models in the work of employees. Expectations from artificial intelligence were high, but reality was less impressive.
218 volunteers tested Copilot, performing various tasks, and then shared their impressions. The participants initially hoped for the widespread use of AI in their work, but in the end they appreciated the usefulness of the technology below the expected. The main reason is the limited Copilot capabilities in complex processes.
The majority used Copilot only 2-3 times a week, and not daily, as the organizers expected. The Ministry admits that it could create high expectations before testing, and also notes the lack of training of employees for the effective use of AI.
It was originally assumed that Copilot will be useful in four main directions: creating structured content, supporting knowledge management, analysis and placement of priorities in information, as well as the implementation of routine processes.
Copilot demonstrated effectiveness in basic administrative tasks such as searching and summing information, maintaining meetings, and knowledge management. However, more complex processes remained outside the capabilities of the Chat Bot, causing doubts among participants about its usefulness. By the end of the experiment, the effectiveness of the Assistant was assessed to have decreased.
Despite this, AI turned out to be useful for people with disabilities, employees with a single-handed working day, as well as those who are forced to take sick leave. The automatic creation of reports on missing meetings and support in orienting work processes helped some participants feel more confident. Particularly positive reviews were left by junior employees and new employees to whom the AI Assistant helped to adapt.
The Ministry of Finance concluded that successful AI implementation requires not only the selection of suitable employees but also quality training. Another concern relates to security: since Copilot works on a cloud platform, its use in government agencies handling confidential information remains in question. The department doubts whether the Microsoft product can meet strict security requirements without the risk of data leakage.