Reinventing Support with GenAI chatbot
“Every minute we could save for our employees through automation matters.” says Anna Gawlicka, AI and Adoption Lead for HR Digital Services at Schneider Electric. With that mindset, her team joined forces with Schneider’s AI and IT experts to create Jo-Virtual Assistant – a GenAI-powered chatbot supporting Schneider employees in their daily HR and IT questions.
In this episode, Anna shares how her team tackled the complexity of fragmented systems, multilingual and multi-country context, and strict compliance requirements to deliver a scalable, high-accuracy AI solution. With a deep focus on knowledge governance, RAG-architecture, and 90% accuracy, Jo is setting a new standard for enterprise-grade virtual assistants. It has been awarded with HR Digital Awards by Future of HR in France and with a Gold Trophy in the Connected HR category by the Victoire du Capital Humain.
Looking ahead, Anna outlines a vision: in the future, Jo will go beyond answering questions, to taking action and offering personalized, proactive advice.
It’s a story of continuous improvement, collaboration, and responsible introduction of GenAI solutions.
Listen to the AI at Scale podcast
Listen to Anna Gawlicka: Against hallucinations: creating a RAG-based chatbot for HR and IT queries episode. Subscribe to the show on the preferred streaming platform (Apple Podcasts, Spotify).
Or play the live recorded podcast on Youtube !
Transcript
Gosia Gorska: Welcome back to AI at Scale podcast. My name is Gosia Gorska and today I have the pleasure of being joined by Anna Gawlicka. We were actually meeting recently in a very specific setup at the Women in Tech Summit in Warsaw, where Anna and her colleagues from her team were discussing one exciting AI application. So let me introduce Anna properly. She is AI and Adoption Lead in Global HR Digital Services at Schneider Electric. Her focus lies on the implementation of AI initiatives for the HR organization and for all Schneider employees. She’s the spokesperson for the AI Hub from the HR domain and a SPOXO domain contact person for the HR organization on AI use cases. She’s been with us for six years, always focused on HR digital transformation. And prior to joining Schneider, she worked for global technology and strategy consulting companies. So welcome, Anna.
Anna Gawlicka: Hi, Gosia. Hi, everyone. I’m delighted to be invited to your podcast.
Gosia: Yeah, pleasure to have you. So let’s start right away with this flagship project that you are currently coordinating. The name is Jo Virtual Assistant and it’s our internal GenAI chatbot. And just to also properly introduce it to our audience, I’m really super happy that you have been awarded with already two accolades. So with HR Digital Awards by Future of HR in France and with a gold trophy in the Connected HR category at the Victoire du Capital Humain. So congrats for this and tell us now, Anna, how it all started.
Beginning of the project
Anna: So Schneider Electric is a big company. We operate in over 100 countries and we employ over 150,000 employees worldwide. So every minute we could save through automation matters. So Jo Virtual Assistant started as an initiative to streamline and enhance employee support at our company—to save time for our employees and also save time for our support agents so they don’t need to reply to easy, quick, mundane questions, but can focus on faster resolution of issues that really need expertise from humans. We deployed our first generation of virtual assistant in 2022. It was still back then on what today we call the old chatbot architecture based on NLU, and we went with the deployment plan forward along and eventually the GenAI revolution happened in the meantime. So on one hand it disrupted our architecture and our project, but on the other side, it gave us the opportunity to scale even faster. So we decided to switch our architecture to RAG, to GenAI-based architecture and no longer manually curate content but to retrieve source data directly from our knowledge articles that we store in our support platform. We also teamed up with our enterprise IT team who in parallel were working on their chatbot around IT topics. So we also wanted to streamline the overall employee support process and provide two domains, two supports in one channel so that the employee does not need to go to different channels for HR and IT topics. So we started this journey two years ago almost and we, yeah, we are happy to be already deploying it to new countries and to having already good results.
Gosia: OK. So from the beginning you mentioned that you teamed up with the IT team as well and who worked on the bot itself.?
Anna: So this is a collaboration of multiple teams here. So the delivery of the project, the development is owned by our internal AI Hub organization who has talented data scientists, developers, product managers that helped us define the product and deploy it. We collaborate also hand in hand with our IT organization, enterprise IT help desk, to shape the product but also build all the necessary integrations with other IT systems.
Gosia: That’s fantastic. And can you tell us a bit more how it all started? So what was the initial phase of the project? You were mentioning when you were presenting on stage at the Women in Tech Summit that very important stage was to actually gather all the knowledge necessary to feed the bot with the information.
Anna: Yes, so our existing systems are still fragmented. We are trying to unify them as much as possible. But through this unified chatbot, we wanted also to centralize and simplify the support process, making it easier for employees to access all the information they require in one place.
Knowledge management
Gosia: Yeah. And what about the knowledge management? How did you come about this challenge? Because combining HR topics with IT topics, this must be a huge knowledge database.
Anna: Yes, and it’s not only HR and IT, but also within HR and within IT to combine them together. It’s also challenging as, for example, on the HR side, every country has obviously their own local labor law, hence local processes and often local tools. So we need to have the comprehensive knowledge for all these processes and tools for every country, often in local languages, and the same for IT. For every application there is a different instruction and there are the knowledge authors dispersed in the organization. So it’s a big challenge to unify all the content into one place. From the HR side, recently we have developed a policy stating that all HR knowledge—be it policy, instruction, user guide—it should all be stored in our support platform. And so that it is in one unified format as well and in one place it’s easier to control and to ingest.
Gosia: OK, amazing. So it means that not only you have gathered all the information needed, but also you have created kind of a new process of how to enter new information into the system. So you can already guarantee the chatbot is working better and it has access to all this new information coming, right?
Anna: Exactly. And we also, with the GenAI revolution, we needed to understand—and this was one of our main focuses last year—to understand how actually GenAI interprets the text. At first with the revolution we thought that, you know, Gen AI can process any unstructured data and content and, you know, it will just work as magic. And we learned the hard way that’s not the case. So we also needed to diligently test how actually content is ingested, embedded, chunked into our vector database. Because our architecture is based on RAG architecture, so we need to understand every step of the process to understand how words are translated into numeric vectors that are later on being, you know, put up together to form back a text and the reply to the employee. So a lot of, you know, reverse engineering a bit to understand how AI actually works and what is the best format to make it best digestible by GenAI.
Jo’s monitoring process
Gosia: Yeah, I think this is really interesting to understand how it works under the hood and the fact that your team really went through this learning journey to discover and to think about how we should approach it to get the best results. So exactly, let’s talk a little bit about the results. In which countries is the chatbot available now?
Anna: So we are live in over 20 countries today and our ambition is to go live globally by the end of this year. We have planned our releases in batches and in the next months we will also add new languages—Chinese, Spanish, French and German this year.
Gosia: OK, that’s really a lot of countries that are already using or will be using soon. And what are the metrics that you are following? So how do you ensure that the bot is actually working properly?
Anna: So we track adoption—so number of unique users, number of returning users, number of transactions that we call. So these are the interactions that employees have with the bot, but also among those interactions we separate those that were indeed meaningful in the sense that these were actual questions from HR and IT domain and not just the checking in, saying hi, hello, what can you do for me? Thank you. And all the small talk that employees are also chatting with Jo. And out of these meaningful queries, we are measuring the accuracy of the bot. We are especially during this deployment phase putting a lot of effort into these accuracy testing and we do it very diligently. We have separate assessment criteria for various cases—if the bot hallucinates, if the bot mismatches, if there was a technical error, if there was an accurate reply, inaccurate reply, missing some information. So we are really like, you know, in the pharmacy reviewing every interaction. That transcript, which is our goldmine to review the accuracy, is anonymized. So we do not track our employees’ data there. But today we rate every interaction manually because we don’t feel that the GenAI is ready today to test how other GenAI worked. So we already are implementing such automation testing for QA, for quality assurance in stage. But from the production we are still reviewing manually to be able to assess—by often local experts—if there was no mismatch or hallucination, if the reply was indeed full or maybe partially incomplete. And today, only experts can measure that.
Accuracy
Gosia: OK. And in one of the previous episodes, we discussed with Natalya Philogene about the chatbot that their team is using for the financial topics. And she stressed how important it was to decide which is the good moment to actually launch the chatbot because they didn’t want to have accuracy that is super high, but then it would really prolong the way the chatbot was developed. So they agreed on something around 84, if I remember well, accuracy percentage. So how did it work for the HR and IT topics? Did you have the threshold that you were aiming for before you launched the chatbot?
Anna: Yes. And when I said that last year was about understanding how GenAI works, it was in the fight for good accuracy because after our first tests, the POCs and the initial pilot, we saw that the accuracy was around 60%. So we needed to understand what are the reasons behind it and we worked for a few months actually and introduced several technology improvements and also made some trade-offs. For example, in order to reduce hallucinations, on one hand we managed to overcome them with technology changes, but also we took a conscious decision to more often, if the model is not sure if the answer is accurate, not to hallucinate but to come back to the user with a reply that “I’m sorry, I’m not able to help you on the particular topic.” So it’s a trade-off. But we, especially from HR, we have zero tolerance for hallucinations or providing incorrect information regarding, you know, HR policies. So it was a conscious decision to more often come back to the user saying “I cannot help you” than to not help with inaccurate information actually. And within a few months we managed to increase the accuracy to over 80 and this year already around 90%. So we have reached the state-of-the-art from the technical and functional side. Our thresholds to go live, so the acceptance criteria, is 70% and it is also like a benchmark that we see on the market. We are very proud that we are exceeding that. However, we are all the time identifying the knowledge gaps that we need to bridge with updating the content because through our analysis of the transcript, we need to identify those use cases that Jo could not answer to the user. It did not hallucinate, it did not provide incorrect information, but it did not provide any information. So on one side it worked by design and was accurate, but actually it was not helpful for the employee. So now we need to investigate all these cases. We are doing it already from both HR and IT sides and we are enhancing our knowledge base.
Gosia: Yeah, that sounds like a really interesting job and something that you can continuously improve over time and analyse. And out of curiosity, can you share with us what kind of questions are asked most frequently?
Anna: So from HR side, employees are mostly asking about their employment information. Jo is today integrated with our core HCM system, so employees can ask about their, for example, job code, cost centre, legal entity name—some information that is required from time to time and the employees forget where to find it or whom to ask. So today they can ask Jo. Also we see a kind of seasonality of the questions. So if we have a campaign around performance reviews, so we ask our employees to complete their performance reviews, to provide feedbacks, we see questions increasing on this topic. Recently, we had our company shareholder program, so we saw an increase and peak in the questions on this topic. And from IT side, employees are mostly asking about our various internal IT systems. I think that the one that is getting most questions is around procurement—our internal procurement system—and also questions like how to connect to VPN, how to connect to our network from mobile, and all the questions around our Outlook or Microsoft suite tools.
Adoption
Gosia: Yeah, it sounds like it’s super useful. I remember I was using it as well a few times already to ask some questions. So I can confirm first-hand that it’s super useful. And you mentioned before, discussing the accuracy, you also mentioned about the adoption and I was wondering—were there any specific choices that you made in order to increase the adoption? Even the fact that the chatbot is available through Microsoft Teams, that is the tool that we use at Schneider, right? This should help with the adoption.
Anna: So Jo is the application on Microsoft Teams. And it was also a conscious decision not to put it only in our HR support tool or IT support tool, but we wanted it to be universally available to all employees, available in the flow of their work if they need to check or find something. So we hope it will boost the adoption, making it easily accessible for all. There is also a big change management and adoption effort—educating our employees what Jo is. In the meantime, of course, there are other automated conversational AI tools available at the company for our employees. For our developers, we have GitHub Copilot, we have Copilot for Sales. So there are also other tools that can help our employees be more productive. So our focus is also on the education— which tool to use for what. And in the future, we will try to simplify also this landscape as well, to have one chatbot for all corporate topics to help our employees be more productive every day.
Origin of the name
Gosia: Yeah, that sounds like a great plan for the future. I have one more question that we can finish with, and it’s a fun fact that I wanted you to share—like why the chatbot name is Jo?
Anna: So Jo comes after Joseph Eugène Schneider, who co-founded our company in 1836. So it’s amazing how old our company is when you think about it, but also it’s amazing how we managed to reinvent ourselves throughout all the industrial revolutions and also now the current GenAI revolution that we are actually surfing on. So I’m really proud to be part of these times. And also, it’s not only about Joseph Schneider, but as you know, our company uses and saves energy, and the acronym for the energy unit is joules. So it’s “joule”—first two letters is “Jo.” So it’s a wordplay also around “joules.”
Future
Gosia: OK, that’s fantastic. And since you mentioned reinvention, I cannot stop myself from asking you—how was your journey in the project? Did you enjoy it and what are your plans for the future?
Anna: So I am here today representing a big team. As I mentioned, it’s a collective effort of various domains and a lot of talented individuals who are leveraging the latest technology, latest advancements to change the status quo, to help our employees and even our customers be more productive. So it’s amazing to be working in such an inspiring environment with both our HR employees and Schneider Digital or IT organization. So it’s great. We are very proud, and the conference you mentioned only again proved it—that we are quite ahead of the market with our product. So we can really be proud of what we do. Of course, like everyone, we sometimes complain that we are going too slow, that decision-making time takes too long. And, you know, we face also everyday challenges all corporations face as well. But it’s been really, really inspiring to be working with great people, dedicated people, and, you know, building together something really unique. Our plans for Joe Virtual Assistant include expanding its capabilities to perform tasks on behalf of employees and provide personalized advice. So our vision for Jo can be summarized in three A-words: answer, act, and advise. Today, we feel comfortable and confident about the first A—answer—so Jo is accurately replying to employees’ questions. Now ahead of us is enhancing actions on behalf of employees through integrations with our various internal systems, and also advise—to provide personalized reminders, nudges to be more productive and save time every day. Also, we want to incorporate more languages, more content to better serve our diverse workforce. We want to team up with other domains, so no longer just HR and IT, but also other corporate topics to really build this one channel—L0 channel—so self-service for all corporate topics.
Gosia: That’s amazing. Thank you so much, Anna, for being with us and sharing with our audience.
Anna: Thank you, everyone.
Like what you hear?
- Visit our AI at Scale website to discover how do we transform energy management and industrial automation with artificial intelligence technologies to create more sustainable solutions.
- Listen to the other episodes of the AI at Scale podcast.
AI at Scale Schneider Electric podcast series continues!
The first Schneider Electric podcast dedicated only to artificial intelligence is available on all streaming platforms. The AI at Scale podcast invites AI practitioners and AI experts to share their experiences, insights, and AI success stories. Through casual conversations, the show provides answers to questions such as: How do I implement AI successfully and sustainably? How do I make a real impact with AI? The AI at Scale podcast features real AI solutions and innovations and offers a sneak peek into the future.
Add a comment