[Podcast] AI red teaming and right to repair.

This audio was created using Microsoft Azure Speech Services

Identifying and Mapping AI Risks

How to identify risks in AI models? Red teaming is one of the options, says the guest of AI at Scale podcast – Dr. Rumman Chowdhury, CEO of Humane Intelligence, US Science Envoy for Artificial Intelligence. Red teaming roots are in the military. It was used to identify whether the base would hold up, and what are the vulnerabilities.

In the conversation with Gosia Gorska, Rumman guides us through her approach to detecting risks, ensuring transparency and accountability in AI systems, particularly in high-stakes applications such as healthcare and finance.

Rumman also provides invaluable advice for organizations striving to establish responsible AI practices and policies, promoting the right to repair approach.

What’s her take on innovation versus regulation? A good policy is actually more conducive to innovation than no policy whatsoever. The same way as brakes help you drive faster on a highway, says Rumman. As we continue the conversation, she offers her perspective on the role of government and regulatory bodies in addressing ethical challenges associated with AI.

To mention just a few of her recognitions, Rumman was named as one of Time’s 100 most Influential People in AI, BBC’s 100 Women, Worthy Magazine’s Top 100, recognized as one of the Bay Area’s top 40 under 40 and named by Forbes as one of Five Who are Shaping AI.

Listen to the AI at Scale podcast

Listen to Rumman Chowdhury: AI red teaming and right to repair episode. Subscribe to the show on the preferred streaming platform (Apple Podcasts, Spotify).

Transcript

Gosia Górska: I’m very pleased to welcome today Dr. Rumman Chowdhury. She’s a data scientist and social scientist. She is the CEO of the tech nonprofit Humane Intelligence, which builds a community of practice around evaluations of AI models, as well as the US Science Envoy for Artificial Intelligence. 

Rumman is the Responsible AI Fellow at Harvard University’s Berkman Klein Center for Internet and Society. Previously, she was the Director of the Machine Learning Ethics, Transparency and Accountability Team at Twitter, as well as the Global Lead for Responsible AI at Accenture Applied Intelligence. She was named as one of TIME’s 100 Most Influential People in AI, BBC’s 100 Women, Worth Magazine’s Top 100, recognized as one of the Bay Area’s Top 40 Under 40, and named by Forbes as one of five who are shaping AI. Rumman holds two undergraduate degrees from MIT, a master’s degree in Quantitative Methods of the Social Sciences from Columbia University, and a doctorate in Political Science from the University of California, San Diego. I’m very pleased to welcome Rumman. 

Rumman Chowdhury: Thank you so much for having me on the show. 

Why AI and Ethics?

Gosia: I was really looking forward to meeting you and maybe a bit stressed because I was discovering multiple podcasts, shows, articles where you contributed with your expertise already. In one of them, you mentioned that you are a type of person who would skip lunch and continue discussing AI topics. I can absolutely relate to that. 

For me, it’s mainly because I observe that AI is disrupting all aspects of our lives—from shopping to music creation, hiring, and energy management. I wonder what inspired you to pursue a career at the intersection of AI and ethics. 

Rumman: Yeah, that’s a wonderful question. I don’t know if I specifically pursued AI ethics because it really wasn’t a career when I first started. My interest has always been at the intersection of humanity and technology. Specifically, I’m a quantitative social scientist by background. 

My interest is understanding human behavior using data, and that’s always what I’ve done, whether in my master’s program, my PhD, or afterwards. Even when I first started in tech as a data scientist, I’ve always been interested in how we can understand human beings using data. That naturally leads to someone being interested in ethical issues because as technology is increasingly being built in a way that seems misaligned with human values, if you see the connection between the two, you want to put them together. 

Gosia: Yes, I see. I would hope that everyone feels the need of this alignment. That would be a better place for everyone, for sure. 

Identifying and Mapping AI Risks

 
Gosia: As we observe AI—and other technologies—they evolve over time. In the Gartner Hype Cycle methodology, there are five phases: technology trigger, peak of inflated expectations, trough of disillusionment, slope of enlightenment, and plateau of productivity. Now, if I’m reading it right, we start to see the limitations and risks related to AI more clearly. So we might be at the phase of disillusionment. 

You mentioned there has been a lot of work done already by nonprofit organizations to build awareness about risks and flaws in AI systems, and maybe it’s time now to build solutions for these challenges. In your opinion, what are the most pressing ethical challenges posed by the increasing use of AI in various industries? 

Rumman: Well, I think there’s a lot to think about with that question. I think the big thing is, especially with generative AI, prototypes remain just prototypes. It’s not so much that they’re the biggest risks; it’s incredibly difficult to map the risk space. Organizations—governments, nonprofits, for-profits—are all asking, how do I understand what risks are being introduced? 

We have one-off ideas of what may happen, but how can we systematically test and evaluate for the kinds of risks that can manifest? In the world of narrow AI, where you train a model to do a very specific task, it’s easier to understand the risk space. But if you introduce a general-purpose AI or chatbot that can do everything, how do you test risk in a model that basically does anything you ask it to? That’s really the big pressing question. The highest area of risks is how we can even identify them. 

Gosia: Yes, because there are so many possibilities. We are humans, and we are very creative. This creativity goes in the right direction but also can go in the wrong direction. So we can be creative about wrongdoing with AI as well. It’s difficult to map them out. 

Red Teaming and Right to Repair 


Gosia: I’ve also read that you have a specific approach to some of the ethical implications of AI from a global perspective—especially ones that consider diverse cultural and societal norms. Could you explain a bit more about the two R’s: red teaming and right to repair? 

Rumman: Yes, my two favorite topics. Red teaming is a practice that has existed for quite some time, originally in the military, where a small team would try to break into a base to test security. It moved into cybersecurity, where testers see if they can break through firewalls or exfiltrate data. 

In AI, red teaming can identify malicious attacks or, in our work, societal impact issues. Many harms come from normal interactions that produce biased outcomes or hallucinations. Red teaming for societal impact focuses on embedded harms and biases that manifest in everyday use. 

Right to repair has its roots in movements around technology ownership. In today’s world, cars are basically computers. If you don’t have the software, you can’t fix your car. In AI, we often don’t have a mechanism to fix or adjust models ourselves. Red teaming is the first step toward a world in which we can interact with these models in a way that empowers us to repair them. I want us to have a world where we’re not reliant solely on the model developers. Instead, we can have independent communities creating customizations or fixes for AI models. 

Gosia: Indeed, and I think it’s important for our audience to understand the difference between red teaming and user experience testing, which is something we do at Schneider. How is red teaming different from user testing? 

Rumman: The purpose of red teaming aligns with model assurance. Is the model performing as expected? It’s different from pure user testing in that user testing is more like field testing for usability. Red teaming specifically targets model capabilities, model guardrails, and can even test policies or laws against the model. 

You can test for bias, factual accuracy, hallucinations, or attempt malicious prompts. You can also test guardrails like Llama Guard around Llama, for example, to see if they truly protect the base model. Some law firms do legal red teaming to test compliance with specific regulations, like preventing minors from seeing certain content. It’s a versatile methodology. 

Global Collaboration, Multilingual Red Teaming, and Accountability

 
Gosia: What’s particularly interesting is that for some red teaming, you don’t have to be an AI expert. You could be a regular user testing for errors. You also organize events that test different languages. Could you share a bit more? 

Rumman: Yes, we aim to be inclusive. Human in the loop is often lazily implemented. We want structured public feedback, where impacted communities provide feedback in development and testing—not after launch. 

We’ve designed a no-code evaluation platform so if you can type, you can help red-team an AI. We’ve held multilingual and multicultural exercises. Recently, in Singapore, we worked with the Infocomm Media Development Authority. They brought experts from nine different countries to test for biases across cultures and languages. But these events don’t have to be in person. We also have asynchronous exercises, like one for doctors around the world testing AI used in medical contexts. 

Gosia: Definitely I will sign up for any red teaming. You also brought up healthcare. So what strategies are essential for ensuring transparency and accountability in high-stakes AI systems like healthcare? 

Rumman: The purpose of these exercises is to identify where problems can happen and feed that information back to the model owners. The challenge is the broken feedback loop. Right now, we have few legal or regulatory options to engage model owners in meaningful ways. I hope better regulation will formalize these testing practices and help with accountability. 

Regulation and Corporate Responsibility 


Gosia: You mentioned regulation, so let’s talk about the role of government and regulatory bodies, especially given that AI is advancing so rapidly. How can companies prepare and contribute meaningfully to discussions on responsible AI practices and policies? 

Rumman: I love that question because much of the AI safety conversation focuses on foundational model companies. But some of the most powerful companies in responsible AI are those implementing it. In my work leading Accenture’s Responsible AI practice, I saw that the real impact is often with the companies delivering AI solutions to their customers. 

These companies see the clear benefit but also know they’ll bear the brunt of negative externalities if the model fails. We see them investing in user testing, red teaming, scalability planning. And that’s where regulation can help. Clear guidelines let them know how to proceed without fear of compliance surprises. 

Gosia: Right, but sometimes building an ethical model can seem more expensive in the short term. And different regions have different regulations. Could that create an uneven playing field if some must follow stringent laws while others don’t? 

Rumman: Yes, there are a few parts to that question. First, building ethically isn’t actually more costly long term. It’s just that if you do your job well, there’s no headline or scandal—nothing happens. But the counterfactual is costly: you might face fines, legal battles, and lost trust if you ignore responsible practices. 

Next, regions vary. In privacy and security, we already see disparate laws. But organizations like ISO move quickly to create interoperable standards. And about regulation stifling innovation—I don’t believe that. Having no guidance is more stifling because companies slow down or fail to scale. Brakes help you drive faster. You feel safer going fast when you know you can stop. 

The Future of AI 


Gosia: I love that metaphor. One last question: looking ahead, what developments and trends do you foresee at the intersection of AI and ethics, and what broader impacts will AI have on society? 

Rumman: I see a focus on clarifying responsibility across the AI supply chain. Data, model development, deployment—these can all be different partners, so we need to know who’s accountable. I also see more realistic applications of AI. We’re moving past the hype and focusing on productivity tools. 

Yes, people worried about job losses, but historically, new technology shifts jobs rather than erases them. The question is how we balance potentially infinite productivity with the need for a healthy work-life dynamic. We’ve never historically worked less when we’ve had technological advances. 

Gosia: That’s a great closing statement. Thank you, Rumman, so much for your time with us today. I really appreciate that we were able to host you. 

Rumman: Thank you so much for the time. I really appreciate the insightful questions. 

Like what you hear?

AI at Scale Schneider Electric podcast series continues!

The first Schneider Electric podcast dedicated only to artificial intelligence is available on all streaming platforms. The AI at Scale podcast invites AI practitioners and AI experts to share their experiences, insights, and AI success stories. Through casual conversations, the show provides answers to questions such as: How do I implement AI successfully and sustainably? How do I make a real impact with AI? The AI at Scale podcast features real AI solutions and innovations and offers a sneak peek into the future. 

Tags: , , , , , , , , , , , , , , , , , , ,

Add a comment

All fields are required.