Open Research Institute Presentations, 22 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Advances in AI for Web Integrity, Equity, and Well-Being

Description

Presented by Srijan Kumar PhD

The safety, integrity, and well-being of users, communities, and platforms on the web and social media is a critical, yet challenging task. In this talk, I will describe the AI and machine learning methods, advancing natural language processing, graph machine learning, and adversarial machine learning, that my group has developed to efficiently fight malicious users and bad content online. I will talk about the four main pillars of my research: 1) Detection: developing multi-lingual, multi-modal, and multi-platform detection models; 2) Robustness: developing adversarially robust detection models; 3) Attribution: quantifying harms and impact of bad actors; 4) Mitigation: developing solutions and tools to mitigate online harms.

A

B

Okay, great yeah hi everyone, uh I'm opal I'm, serving as the chair of IEEE computer Society, San Diego chapter this year, uh I'd like to welcome everyone uh to the first talk of this invited seminar series organized by our chapter. Today's talk is co-sponsored by IEEE Information, Technology society and computational intelligence Society, uh both Society chairs are present here. Michelle and Doc uh uh are cheaply being the world's largest technical Professional. Organization is dedicated to advancing technology for the benefits of immunity.

B

As part of this Global effort, IEEE San Diego section is regularly organizing a wide range of activities and events for information about the upcoming events. Please visit the IEEE San Diego section website, uh we'll share the link in the chat. uh The IW computer Society is dedicated to engaging Engineers scientists, academicians and Industry Professionals for Global Technical advancement in all areas of computing. This seminar cities is an effort to enable knowledge, sharing and increase networking among our members as well as interested non-members.

B

It is our immense pleasure to have Professor srijan Kumar as our first invited speaker. Dr Kumar is an assistant professor at College of computing at Georgia. Instead of Technology, he did his postdoctoral training at Stanford and his doctoral research at the University of Maryland College Park. He and his research group innovate the efficient methods for detection prediction and mitigation of uh online threats and contents, such as malicious actors, misinformation, hate speech and fake reviews.

B

Dr Kumar Solutions has have been used at Flipkart, which is India's largest e-commerce platform now occurred by Walmart, has influenced tweeters badwash platform and is now being deployed on Wikipedia. He has been recognized as a Cavalry fellow by the National Academy of Sciences and the Forbes 30 under 30.. He is also a CRA Computing Innovation Mentor Facebook faculty research awardee and Adobe faculty research Liberty.

B

His work has been covered in a documentary titled familiar shapes on Apple podcast in radio interviews and by popular press, including wired, CNN, Wall, Street, Journal and many more in today's talk, Dr Kumar will give us an idea about advancements in NLP and ml based approaches for fighting digital misinformation and malicious users without further Ado srijan. The floor is yours.

C

Thank you for inviting me it's my uh utmost pleasure to be here and share some of the work. My students and I have done over the past several years at Georgia Tech to improve the safety and integrity and well-being of users, communities and platforms online.

C

This is a super important topic, because fraudsters Bad actors, online misinformation and polarization presently.

C

Create one of the biggest threats today to Public Health, democracy, science and the entire Society, and the problem is huge. Facebook estimates around five percent of its accounts are whopping 40, 145 million accounts to be fake, and it's estimated that Amazon has around 300 million fake reviews and of course, everyone knows about the immense impact that online misinformation has had in the topics of of broadly in the recent times of covert vaccine misinformation and the threats and the resulting harassment and assault of of Asians that have resulted because of online misinformation uh against them.

C

Such harmful content typically disproportionately disproportionately impacts marginalized and disadvantaged communities, and the problem is particularly alarming today and we need to solve it now. Today, people spend 20 percent of our time online and we get more news and information from our friends and family in social media than on traditional news done from traditional news organizations.

C

As a result, it has become immensely easy for online Bad actors and dangerous online content to manipulate public opinions against even the best available evidence, so distrust against experts and turn people against Democratic processes, notably due to the very recent advancements in generative AI, everyone's familiar with chart, GPT and Dali and others. It has become the cost of Bad actors to create very believable misinformation. That's even quote-unquote, backed by scientific evidence, including texts and images and videos.

C

It has become so easy for Bad actors to create and use them for harmful purposes that if we don't do anything now, the problem will get even bigger going forward. Today we are at a Crossroads where the fabric, the entire fabric of society, uh is being threatened by such bad actors.

C

Therefore, at judge attack my students and I build AI machine learning and data mining methods to combat such online harms and improve the Integra, improved platform, integrity, user equity and Community well-being using online social media data. We innovate natural language processing, graph, neural networks and behavior modeling techniques to develop what we call multi-x detection methods.

C

Essentially, multi-x stands for multi-platform, multilingual and multimodal, which means our models are not focused on the typically studied English language text from Twitter. But we look at data that spans multiple platforms that go beyond English and spread harm using text, images and videos, so we interve state-of-the-art models to detect them, but, more importantly, in the domains of web safety and integrity. While we are trying to catch the Bad actors, The Bad actors are trying to fool and evade the detection systems.

C

They continuously evolve their strategies to fool uh to fool us so that they can continue to make uh make their harms go far and wide.

C

Therefore, we have one of the first groups to develop adversarial machine learning methods to identify the vulnerabilities that current detection systems have and create the next generation of systems that are not only accurate but also robust against adversarial homes.

C

We also create methods to attribute the role that such online harms have in the real world and, finally, and most importantly, we create data-driven Solutions, as well as tools to mitigate such online harms and enhance the the safety, equity and well-being that online platforms create.

C

As Oppo said, some of her work has been deployed at Flipkart, has influenced Twitter and is now being deployed at Wikipedia. We are also creating new tools that Empower ordinary users to correct misinformation, and we have a big NSF Grant to build what we call course correct, which will enable professional fact, Checkers and journalists to be very efficient in monitoring, as well as delivering misinformation Corrections in real time. I'll talk about this uh in the later half of the talk.

C

I'll first cover detection, I'll briefly touch on robustness and then I will I will learn the talk by talking about some of our mitigation Solutions and at any point in time. Please feel free to stop me just unmute yourself and ask any question that you might have, or you can put that on chart and if Apple can can convey the question to me. That would be. That would be great sure.

C

Thank you. So, let's start with detection, the more steady data online is English language text from Twitter. However, and by the way, the reason is because it's super easy- or at least it has been easy until very recently, to collect such data and and Analyze That now, with upcoming changes to Twitter and its apis. We don't know, what's what's going to happen, but so far English language text from Twitter is the most studied online data, at least in research.

C

However, it turns out that 74 of data is actually not in English. uh In fact, and misinformation is a problem that that's that's language, agnostic, you get misinformation in any language, it doesn't just use text and it definitely spreads across social platforms.

C

Therefore, we started a line of work where we, where we looked at uh the impact that misinformation has across multiple platforms, how it spreads and in this particular work, which was at uh triple AI icw SM 2022. We looked at the dispar at potential disparity that might be present in English versus non-english models that are created specifically, since we researchers spend most of our time, creating the latest and greatest English language models and classifiers.

C

We were curious to understand how non-english classifiers stack up against these English language classifiers and whether, if there is any disparity, whether there's any solution that can be created.

C

So let me be more specific.

C

We consider the case where you have multimodal data where you have images and text and the text is written in English and we consider three different scenarios: uh detecting crashes crisis, Humanity humanitarian tasks, which is about detecting when a when a crisis happens just like what's happening in Turkey right now, um because of the earthquake. What uh what are people saying about it in terms of whether they're reporting damage, whether they're reporting about volunteering or rescue efforts or donation efforts, uh or talking about something else?

C

We also have tasks on fake news, detection and emotion, detection, and in each of these tasks we have. We have text and English language. We have English language text and accompanying image. What we wanted to investigate was that is the same misinformation.

C

If the same content is spreading in another language, will machine learning models be able to identify that misinformation as effectively as effectively as it would if it's spreading in English, so what we did was actually we created translations of just English language, misinformation, English language crisis tasks and English language, emotional appeal, uh content to five other languages, so we hired translators to do that and we also used verified automated translators.

C

We created data sets that are super large and translated English language content into Spanish, Portuguese, Chinese, Hindi and one more language French, and what we found was that if you take the best English language to detect uh misinformation- and you take the best non-english language in each of these individual languages and try to identify the misinformation in that specific language, the non-english language models do not perform as well.

C

So the F1 score for this classification task for English is 0.65, and you see this decreasing Trend as you go from left to right across different language. Essentially, this highlights that, even even though the same misinformation content is being spread in the other language, the automated classifiers are not able to identify that as effectively.

C

Now, you can imagine the harm that this can have, specifically on marginalized communities that do not speak English and which are often targets of online misinformation, hate harassment and other societal Harms. And what is this is showing is that, as these as platforms are developing AI models, text-based AI models to identify online hubs in English and in other languages, because of this disparity, the English-speaking population is better served than the non-english language population.

C

There's disparity we tested, as, as I said, we tested this disparity uh on multiple different data sets and we consistently see such a trend.

C

So then we looked at how we can create Solutions how we can improve uh over this and what we did was uh we. We realized that in many of these scenarios we also have images, so we created late Fusion based multimodal models that that extract information from the text, as well as from the image, combine them together and use that for classification and two over uh to our pleasant surprise.

C

What we found was that there was Improvement across the board, but using images helped improve English language content by five percent, while it helped improve non-english language content classification by around 11.5 percent on average, essentially showing that uh this disparity is, is bridged in a way where, where English language is improved, but non-english is improved, more and the gap between English and non-english uh reduces.

C

This was one of our solutions that we created, and uh this was we're, hoping that others would take this forward and come up with other solutions to help bridge multilingual disparity uh in in the task.

C

At the same time, um so this is this is work that we presented as icws in 2022, and we did realize that language alone is not sufficient to help solve these problems. So what we did was we created language agnostic models specifically by using interaction data. So when people interact online when they do anything online, they leave these digital tracers of who is connecting with whom who is talking to whom? Who uh and how? uh Who who is commenting on what post and so on.

C

So what we did was we created these networks out of out of such interactions, so these networks represent users and content and which user is interacting with what content at what time? And these interactions can be writing a post liking, it commenting on it re-sharing and so on, and we created a model which, which I'll describe uh with I'll talk about. In a bit, we created a model called Jody which takes such interaction, temporal interaction networks and creates Dynamic embedding trajectories.

C

So these Dynamic embedding trajectories what they are is a vector representation of every user and of every piece of content that evolves and this Vector representation or embedding evolves over time. And since this, uh since this trajectory is evolving over time, we are able to understand not just what the historical properties of the user or the content was. You can also use these trajectories to forecast what will happen in the future, so using these trajectories, we can make several types of inferences and the specific model that we created is called Jody. uh It takes this.

C

This interaction embedding and it's a the model is a graph neural network based model that takes uh that is a recurrent neural network. That takes is a system of record neural network Frameworks that updates embeddings from previous timestamp to the current timestamp, and it uses a Kalman filter approach or kalman's filter up inspired approach to forecast embeddings in into the future.

C

Using this and using these embedding trajectories, we were able to show that Jody model is able to outperform several existing state-of-the-art, malicious user detection models to uh in the task of detecting Bad actors on multiple data sets that we collected from The Real World, in fact Jody outperformed existing baselines, but by about 12 percent and if someone's interested in checking it out, we have all the code and data available on this website.

C

What was very satisfying and very, uh very, very fulfilling for us, was that, shortly after we created this model in 2019, uh Facebook created a model in 2020. That is a follow-up work on this on this work, uh our model is called Joint, Dynamic interaction. Embedding Facebook followed it up with temporal interaction. Embeddings their model is called ties and they use this model. The ties model currently in production to identify various different types of online harms, including identifying ad payment fraud, misinformation, content and uh um and I think bot accounts.

C

Oh and shortly thereafter, Twitter came up with its own generalized model, which generalizes ties and Jody and several other models and you and it uses that to create to identify Bad actors on their platform.

C

What this is showing is that such graph based models have a lot of potential to scale even to platform. Super large platform scale data with certain optimizations that can be done and have a potential to solve real world to identify Bad actors not just from the content, but also from the interactions that they have.

C

So uh let me pause here and take any questions that you might have about about the multilingual disparity work or about this graph based approach that we have created.

D

um Good evening.

C

D

uh So Professor.

D

English, it's pretty common that we use mixture of languages so like. While we tried something it's very common that you mix up languages when the languages are right,.

C

Sure great question, so the question I'll repeat it, because your voice was not very clear. uh The question was uh people typically on social media. uh They mix languages, so they would start writing in English and then they switched to let's say Hindi and then they would switch back to to English uh that's called code, switching uh and uh and- and that is a well-studied uh Topic in in national language processing.

C

In this work, we specifically looked at complete translation, so completely uh the the content is entirely in English or in one of the other languages. um So I can't really say what the exact numbers are going to be.

C

But since we saw an improvement in all individual languages, my intuition would say that, even if the content is code switched it's a mix of multiple language which still see improvements by using multimodality, and we would also see disparity- uh which was our one of our key observations, so I'm yeah up the work would I expect the work to hold. We haven't done experiments, but that would be actually a very good follow-up for a journal version of the paper.

B

uh Hi I have a quick question: when you uh do, you have a single model that actually can handle all of these languages, or you have.

C

To retrain great question, so we did experiments with monolingual models as well as multilingual models. Monolingual models are the best models that have been. We took the best possible models at the time we did the experiments for each of these individual languages as well as we did experiments where we did where we took the best multilingual model which works on multiple different languages, and the paper has all the details, but we, but we still see this disparity and we still see Improvement when using multimodal data augmentation for uh to bridge the disparity.

A

Okay, yeah thanks.

C

Okay, uh I'll go ahead and again feel free to stop me at any point in time, so continuing on the line of line of research that we have built in terms of uh in terms of uh building detection models. uh Very recently, just last year, we we did one of the first works on detecting band evasion.

C

So one of the biggest challenges that we have today uh that face platform Integrity Solutions, is that as soon as you deploy a model to detect Bad actors uh and it and the model catches Bad actors, it's trivial, it's literally trivial for Bad actors to just create new accounts and continue their malicious activities, and in many cases we have all. We have even seen people creating 15 different accounts to spread out their malicious activities across these different accounts.

C

So in a work that was led by my PhD students, uh crazy student, gaurav Burma and an undergrad at the time, Manoj nagarthy, we conducted one of the first studies to look at the problem of ban evasion where our goal was to study how Bad actors create new accounts after being banned and whether we can create detection and prediction models.

C

uh Whether we can create detection models to identify when such evasion has happened and even predictive models to predict which bad actor Will evade a ban in the future uh by taking uh by by creating models by we first created uh one of the first data sets of ban evasion, uh you with identified ground truth from Wikipedia, and then we created machine learning models that look at linguistic Network as well as behavioral properties, and we were able to create detection models for ban evasion that have very high performance at 0.9 area. Under the ROC Cove.

C

uh We told this to Wikipedia moderators, essentially, people volunteers on Wikipedia Who, currently do platform, Integrity tasks pretty much manually, and they got super excited about this result and they asked us to create a tool for them. So right now we're in the process of creating a tool where the moderators don't have to do all this manual work, but can use the use our predictive models to not only know which account is potentially beneficial. A vision account, but also explore that further, along with along with additional information.

C

In terms of ah in terms of explaining why our model is saying uh whether two accounts is uh are are um are related or not, or Banning, Vision or not. We have delivered the first version of the tool to moderators. They were super excited about it. They gave us a lot of very useful feedback, so we are now incorporating this feedback and we expect that the model uh that the updated version of the tool will be delivered to them in a by the end of the semester.

C

So so far, I have explained: uh I've talked about a few models that we have built to identify Bad actors and harmful content online, uh and while we are creating these models and while some of these models are being deployed, they are very cognizant of the fact that, shortly after some of these models will be deployed, they will become useless and the reason is not because the models are not good. But the reason is that the Bad actors will change their behavior.

C

They will adapt how they behave, how they write, who they connect with and various other properties of their own behavior, and they will change that in order to evade detection.

C

And they will also try to manipulate uh manipulate other accounts, and this is something that the trust and safety professionals and platform Integrity solution developers are very well aware of, but struggle with immensely, because there's no way currently uh to to deal with this.

C

So what we did was we uh we we have. We started a line of research in my group uh where we try to understand what the vulnerabilities of these detection models are, and we try to do that in advance before these models are deployed, so that we can identify these vulnerabilities patch them and uh in such a way that the Bad actors are not able to leverage these vulnerabilities for their for their harm.

C

The goal. My vision for this line of work is to create models that are not just accurate, but also robust against adversarial manipulation.

C

So, along those lines with that mission, we we conducted one of the very first works of adversarial machine learning, of what I call practical adversarial machine learning against such detection systems to identify how robust they are against adversarial manipulation, and this work was led by my PhD student, Bing, hey and uh and was funded by a Facebook faculty award to my group.

C

The solution that we came up with was to create robustness evaluation as well as benchmarking system, uh where we were able to create systems that would be able to identify the robustness of the existing detection systems, Benchmark them and uh and and test how they would perform if an adversary was specifically trying to break the model in a white box or a black box setting, we created one specific adversarial attack in the paper. We call it pet gen. So let's look into that.

C

So the current way in which uh many of the detection systems work is that they take. If they try to do bad actor detection, they would represent a user as a c as as a c as the sequence of posts that they have created. So let's say this lady is is a malicious user and she has posted about movie soccer and couple of restaurants on their on on their social media feed, and these can be post. These can be likes.

C

These can be any sort of interactions and when uh the platforms, bad actor detection model, takes this sequence of interactions into as input it detects that this user is malicious.

C

But what we did was we said if we act as Bad actors. If we act as these malicious users, can we write a post?

C

Can we augment the data with the post in such a way that, after this attack when this, and when this new representation, new sequence, that represents the user, is taken as input by the detection model, it misclassifies the the user, not as malicious but as benign, which would mean that the attacker is successful in fooling the detection model uh by by diluting by adding a post or by diluting or distracting the model uh and and being detect being classified as a benign user rather than a malicious user.

C

So we said what sort of post should the bad actor write and to do that, we created um the model called pet gen, which stands for personalized text generator, and it's a multi-stage multitask learning framework that takes into account the historical posts that the user has written or interacted with the context of those posts, a Target context and uses all of that information to match the style of the user.

C

Make a post relevant both with respect to the recent post that the attacker has made, as well as the target specific discussion that the entire platform is having to create a a a attack post which, when augmented, will lead uh the the will lead to the detection model being fooled.

C

We did many experiments both on Facebook's own model, as well as uh in a benchmark in a sandbox setting, as well as various other various other detection models that are straight apart, and what we found very surprisingly, was that Facebook's Thai system, uh which is which is the one that's deployed on their platform, can be fooled very easily one out of four times by that by this very simplistic attack.

C

This is huge because it's showing that these model, this this specific model, has a 25 failure rate which, if you are from cyber security, is Unthinkable, because uh because you would want a failure rate to be 0.0001, but this model, even though its in production has a failure rate of 25 percent, and it's not the model developers faults even because, um because there is no way in which one could reliably test how robots these systems are, and even these simple attacks are able to put them so to bridge to bridge this Gap.

C

We are in the process of creating these benchmarking systems, where any bad actor detection system, any misinformation detection system can be identified, can be benchmarked against various different types of adversarial, as well as non-adversarial attacks, so that we can so that we can create systems. We can make it a habit not just to test for accuracy, but also for also to test for robustness uh in, in a way that uh that existing uh system, uh in a way that would lead to the creation of next generation of uh systems that are robust as well.

C

We also have conducted other works that was published at cikm uh last year at amnlp uh 2022, as well as a couple of other submissions at ACL this year, where we have looked at the robustness of recommender systems, multimodal detection systems, as well as as well as text-based models and across the board. What we have found is that, while deep learning models are being developed to detect uh to detect various different types of online homes and to uh and being used for critical uh critical tasks, they really are not very, very robust. We need we.

C

We have highlighted the need to create systems to improve the robustness and that's an active, very active area of research in my group.

C

Let me again pause and take any questions that you might have about robustness, uh part of the work that that I just talked about.

D

uh Professor I have a question, so um in many cases the Bad actors uh do not use typical word to you know, put bad comments, uh it's sometimes slang or grammatically, not so correct uh or failing mistake stuff. uh So how does the system you know work for those because, as far as I understand language models generally are trained for correctly spelled words or something which is, you know, messed up.

C

Yeah great Point, that's that's another, um and that is another way in which Bad actors are trying to fool these detection systems, in fact um misspellings or uh um uh what we would uh so, in fact, uh this in the course that I taught previous semester in in Fall 2022. One of the course projects uh students came up with was to look into that.

C

They essentially uh looked at uh at this at how people are trying to fool tick, tocks moderation system by changing certain characters uh in in uh in band words and replacing certain words with certain other words and uh and what they were able to find is, in fact around 15 to 20 percent uh cases. They were able to identify harmful content or or policy violating content where people are are Mis using misspellings uh to uh to to evade this type of of moderation.

C

Uh Current detection systems are not aware of those because they are not trained to identify such harms and that again, that is something uh that some of the benchmarks that we are trying to build uh would help solve, uh would definitely have solved.

E

uh Hi professor I had a question. uh I was wondering: how did you get your ground root labels for melanin or benign uh class of us for classification classification system and I was also curious. How did you, uh how did you get to know that this should be classified as a benign or Melody and while training so as a source truth? So.

C

Yeah great great question, uh so we have, we have used existing data sets that others have created where they have provided ground truth, and we have also crawled our own data set uh where we have used public data set from Wikipedia, where moderators have identified uh manually identified, which accounts are malicious and banned them from the platform along with given given along with certain reasons so, for example, yeah. So in in Jody work we created two new data sets and we released them publicly.

C

One is from Reddit uh where we had 10 000 users, out of which 366 were malicious, and this ground truth was given to us um was was given to us and it was verified. These accounts were in fact banned on the platform uh and on Wikipedia moderators banned 217 accounts out of 8 000 200 accounts, so we use these data sets, along with other data, sets that that others have created. uh There are popular data sets about bots on Twitter uh that that can be used.

C

There are several other data sets uh that that are present. uh We have Crea, we have. If you are interested, we have curated a list of such uh bad actor, harmful content data sets uh in my group, and we have made those available for further research uh and they can be found on my website.

C

So we never make decision of who is bad and who is not or what's misinformation or what's not, we always rely on Experts. We always rely on ground truth. That's that we can verifiably get from from either the platform themselves or from certain exports.

E

All right, thank you.

F

um I have a question so um I was wondering. uh You've talked about uh methods for detecting bad content, as well as methods for detecting bad users and I'm wondering how these methods sort of compare against each other in terms of just like Effectiveness in general. um For for the platforms, uh safety.

C

Great Point um yeah, so we have developed harmful content, detection and harmful user detection models, and they are complementary. They work they work and feed off each other in the sense that this Jody model that's there, it uses it. It can be used to do bad actor detection from this side of the of the graph, but it can be used to do bad content. Detection from the right hand, side of the graph and in fact, uh ties which is Facebook's. Follow-Up work to Jody uses both sides of the graph to detect ad payment fraud.

C

Bot accounts, as well as misinformation, content portions, uh so these can be used up from both sides. Some of the other works, such as the multimodal uh work or that I've talked about uh those works, are content based primarily, and uh they can be used in augmentation, with uh with user-centric models that we have.

C

Any other questions.

C

All right I'll go ahead, uh so so far, I've talked about a few detection models that we have created and some of the work around robustness of these detection models.

C

uh I don't have time to talk about attribution, but in a scientific reports paper that we published last year, we looked at attributing the harm that online misinformation has on uh on on mental health and anxiety, and what we showed was there is a a causal impact on consuming misinformation and believing in misinformation and exacerbation of of anxiety and there's disparity in there, as well, with respect to uh who's, more educated and who's, less less educated we've. Also, we.

C

We have also started some other work um recently funded by CDC to look at uh the impact of online misinformation.

C

Specifically uh anti-violence, provoking this information and attributing that uh attributing the impact of that to uh to violence, provoking intentions on the consumers of such misinformation, in the sense uh that if you read a misinformation that is hateful against Asians, uh do you as a reader, become more likely to either harass online or in the physical world?

C

uh That's something that we that's the project that we that was started last year and we are in the process of of of of working of doing the research on that, uh and we have also started a lot of work. We have also done a lot of work on mitigation Solutions. So let me talk a bit about that.

C

So, given that there's so much uh so so much online harm, uh that's there uh to find Solutions we in in a couple of NSF funded projects. We analyzed uh online misinformation spreading during covert 19., um so I joined, Georgia Tech in January 2020 and in March.

C

Everything went online because of the pandemic, as I just moved to Atlanta, I'm still figuring out was there's good coffee, there's, good Indian, food and so on, and while there was a lot of Chaos in the real world, there's a whole other storm brewing in the online social media world and that storm was about covert misinformation.

C

uh One of my favorite uh or funniest misinformation that I find is that covered vaccines have microchips in it that can track every movement that you have and that you do and therefore you should not take covert vaccines. uh That's not true! There's! No, uh their vaccines are safe. uh They do not have microchips and better than them, but such misinformation did reduce the trust that people have on in vaccines and did reduce uh the the uptake of vaccines.

C

So we started studying online misinformation against of covid-19 back in 2020, and we collected uh in the initial study. 8 million tweets, but we have since expanded that to around 100 million tweets, um and what we found was that, while people are spreading misinformation, people some people are also countering misinformation and in fact, 96 percent of all counter messages are delivered by non-exports.

C

This was surprising to us, because uh what we found was that, while professional fact, Checkers and journalists are creating fact, checks non-experts, ordinary users, like you and me, we are the ones who are actually propagating these fact checks on this on social media. We are the ones who actually are engaging with misinformation, spreaders and correcting them.

C

This was a a really an eye-opening uh statistic for us and we have created a a very big line of research around that my NSF career proposal is also uh focused on that and uh we what we have. What we found in this particular work was that, while almost all counter misinformation messages are by non-experts.

C

Most of these messages are in fact wrong in the sense that they're rude, they are calling people names, they are abusing them and that's very risky because it has it can backfire in the sense that bad actors, uh in the sense that misinformation, Believers would become even more, uh would would essentially be forced to back their own positions and it can backfire in the sense that people will not believe in the counter misinformation just because of the tone that they have. However, uh this statistic uh was very important.

C

uh This work did influence Bird Watch, which is Twitter's Twitter, now calls it Community notes uh and it's a community-based misinformation flagging system, and uh since then we have created uh multiple different solutions. We have been working on multiple, different solutions to empower ordinary users to do better counter misinformation efforts.

C

This is one of the works. uh This is actually an upcoming paper at ACM Web, Conference 2023, where we have created a counter response generation model. This is a real misinformation, uh tweet about covet vaccine from Twitter, and this is a real reply. The misinformation is talking about is claiming that covet vaccines are gene therapy, it Alters your DNA and real and people have replied to it.

C

Saying that you are, you are born to speak, nothing but lies our system, which uh is a counter response generation text generation model, so think of it as chart GPT. But for countering this information uh we we made it much before uh chat. Gpt was released by the way.

C

um What we, what we are able to do is to take uh data training data, take real good replies, Good, counter replies along with crowdsource replies and professional fact, Checkers replies. We are able to take that and create a counter reply, counter response generation model that is structural, that's polite and directly refutes misinformation, claims that are there uh in the in the original post, and this is a reply that our post makes. It says this is not true, it's a it has nothing to do with cloning or DNA.

C

It only uses mRNA and there's more information that it provides. So my vision is that with such a system will be. We can Empower ordinary users to correct misinformation effectively and at scale and in real time.

C

In addition, we have a very big Grant from NSF called convergence accelerator, where we are creating course correct, which is a tool that enables professional fact, Checkers and journalists to monitor misinformation, that's spreading on social media in real time. We.

B

Have a short question on the bridge from Julia: what is the basis for whether or not it is fake or true? Information am I missing something this seems to track. Whether information confirms to a certain majority view, is it it? Possible majority could be wrong.

C

Absolutely majority can be wrong, so that's why our work does not rely on on on crowdsourcing alone. uh In fact, our work relies on labels that are generated by professional fact, Checkers and journalists who are professionally trained to identify uh to dig into into any particular topic and tell us and give us labels. So we are part of ifcns or International fact checking networks uh database.

C

We have access to that database where professional, where International several International fact-checking organizations I, think it's uh or the order of like 50 or more organizations, uh contribute fact checks that they create. uh This was an effort that was started during covert 19., so we, our misinformation, detection systems, are based on data set uh on on fact, checks that have been professionally generated.

C

We take that information into account uh to to label misinformation tweets and to create misinformation. Classifiers from that, so we never we I'm, not trained. None of my students are trained to be the arbitrars of Truth. uh We always rely on Expert labels uh that are there.

C

uh That's also used for our counter response generation as well. So we always we. We always rely on professional labels because, as you correctly said, majority can be wrong or then can be manipulation where you know, Bad actors can create many accounts to uh to essentially create the illusion that majority believes uh in the misinformation, or you know that misinformation is not actually misinformation. Yes, that that actually happens.

C

um That is that's quite common. We would call that a A robustness or adversarial attack, uh and that's why we always rely on professional levels.

B

Okay, thanks all.

C

Right uh so continuing on and I'll wrap up in a minute.

C

um So as I was saying, we are developing course correct, funded by NSF to empower professional fact, Checkers and journalists uh to to not just see which users are consuming misinformation and which users are con, are frequent spreaders of misinformation but in fact, to deliver corrections to them uh in real time via platforms. Add uh ad targeting systems.

C

uh We are developing that we have very positive response from uh from from our end users, and uh we, uh as I said we have a CDC Grant, where we are creating a dashboard for public health experts to monitor, as well as interviewing to counter the impact of violence, provoking uh dangerous misinformation.

C

uh Let me wrap up uh before I. Do I'd like to thank my students, postdocs uh and collaborators, so these are. This is my group uh of postdocs and students. uh My work is funded by by these agencies very grateful uh to them for the support that they provide. We also have uh postdoc, as well as data engineer positions in my group, so, if anyone's looking uh for their next adventure, adventure in this space, please feel free to reach out.

C

Let me wrap up. uh uh I talked about the work that my students and I have done in the area of web uh platform, safety and integrity uh to uh where we have created models to detect online hunts such that the models are not only accurate but also robust, as well as I talked about some of the mitigation solutions that we have created.

C

uh I've enjoyed answering questions uh and I'll be happy to take more if there are any uh and uh thanks again for inviting me uh to to speak with you all today,.

D

Thank you Professor, for the amazing question very specific to corrected what it's been awesome.

C

I can't hear you sorry uh am.

D

I audible now I.

C

Think it's probably with.

D

A mic- yes, um okay, uh so uh uh I was asking. If the misinformation is uh having some mathematical incorrection, for example, let's say uh Queen Elizabeth ruled England for 300 years. We as a human know it's incorrect, but how does bot would firstly know that it's incorrect information and secondly, um if it knows uh the span of years, is generally not found in information. So a typical board would need to you know, evaluate the years of ruling, subtract it and then give the answer.

D

So how how this kind of uh cases are handled with corrected.

C

um Right now it's not uh it's a it's a very difficult problem. It requires. So the current version of the there is counter response generation model is uh is, is the first I would call like version one uh future versions: I Envision would incorporate uh knowledge coming from knowledge uh from databases, knowledge, graphs uh and scientific literature and news articles, and so on right now, it's purely looking at language.

C

It's truly looking at what others have done, what professional fact Checkers and journalists would uh would how they would respond uh to the content right now, detection systems and countering system. There are no countering systems, but detection systems um do not are not smart enough to to uh to work with logical inconsistencies or mathematical, fallacies and so on. So right now uh the the that is an open challenge.

C

I would say if you are interested, you know, I would love to hear what you are able to do and do on it, uh but uh but definitely I think that's a that's a very difficult problem and something that uh that that that definitely needs to be sought.

D

Thank you so much professor.

B

uh Hi so then there is a question on the bridge, uh so Tom asked a great research and presentation, Dr Kumar. uh My question is: who will vet the experts to ensure that they are not introducing their own biased misinformation?.

C

Good question: uh we have thought about it, not in the context of misinformation detection, but in a similar context of fake review detection. Where we we don't. You know, trust. Who is writing what we don't trust? uh Who is saying what so there we created a system called ref2 which stands for reviewing the reviewers, uh where the goal was to create a mathematical, uh a iterative algorithm that would be able to use uh they'll be able to use consistencies to identify whether uh someone's, whether someone is is reliable or Not So.

C

Based on that one would give reliability scores, uh not just based on one piece of content that that someone has written but based on the based on all the things that that all the reviews that they have written. So we were able to develop that we were able to deploy that on Facebook on on Flipkart, which is India's largest e-commerce platform.

C

But I would imagine that an extension of that or a version of that could be used for uh for fact, checking or misinformation uh as well, where we could use uh a reliability score metric to identify or to rate these uh professional organizations or experts to see whether how reliable they themselves are right. Now we trust what they're doing, because they are part of ifcn, which is a very reputed international organization, and they have a very, very, very strict criteria of uh which organizations can become part of the iscn network.

C

I know the people who who do uh the wedding. They are very reputed journalism, professors uh one of them, is in Wisconsin, Madison uh and and so on. So right now we trust the judgments that that these professional fact, Checkers and journalists are providing us, that's not to say that that they might not be wrong and they can be wrong because of uh because of evolving scientific evidence, for example right um or in cases where uh the the scenario evolves or the truth evolves, or uh they have incomplete information. So these errors can be there.

C

uh We are not. We are not uh unaware of such issues right now. uh We do not our systems assume that they are correct, they are reliable, but future extensions of of these systems can be created where, uh where uh where the work can be used to also take into account these uncertainties, these unreliabilities, uh the evolving nature of information, uh content itself and, of course, the reliability of those who are providing valuables.

B

Great yeah, so another question from Prashant, uh exciting research, Professor srijan I was curious to know how frequently do we crawl web Reddit or Twitter to detect possible misinformation to be flagged in systems, like fact, checker, and if there is any SLA service level agreement between when it was posted and detected.

C

Good question so right now our systems are not real time. uh The course correct system that we are building that we aim to to make it real time, in the sense that, when a end user in which, in our case it's a professional fact, Checker um logs into our system and they put in a query, we'll be able to go and fetch uh misinformation on that topic for them right now, we are in the stages of building it. So we don't have the system ready.

C

Yet um we we do have a crawler for crawling covered misinformation that we run real time. We have been running that for the for the past two and a half years or three years now, um yeah and uh it's possible once the data is collected, it's trivial for us to detect misinformation on that, because we have pre-trained misinformation classifiers on these topics.

C

When new misinformation arises, we are able to uh to retrain these classifiers to include new information as as soon as it arrives and um I, don't really understand what you mean by service level agreements, so I, I, I, yeah,.

C

If you can clarify that I'll be happy to answer it.

E

Yeah. Sorry, uh sorry, thanks for clarification, Professor, it's just that we as a term we used here in company uh that the time until which it was actually posted until which it was actually detected so any uh basically the time and seconds or hours so.

C

Yeah, as I said, we don't have the real-time system yet so we can't uh yeah. We don't do this. Real-Time detection got.

E

It got it yeah. Thank you so much and really exciting research. So thanks for presentation and special.

F

um I have a question, um so I was wondering how um these systems have been built for detecting uh misinformation, um how how would they adapt to new topics or new uh misinformation that comes up.

C

Great question, so that's actually a project that we have started just three weeks ago, where we are trying to create generic misinformation classifiers uh that can then be fine-tuned when a uh when, when a new misinformation topic arises with very, very few labels, so we are thinking of it in terms of few short learning NLP models uh in English language for now uh English text, um and we are trying to create few short learning systems that can be used to identify to to update models as new new misinformation topic arises.

C

um That's a very important area of research, I believe because that's something that current systems lack. Currently it takes us um a couple of weeks to update our misinformation classifiers, because it requires us to recall data label data label like thousands of data points and then uh create models.

C

But of course, as you can imagine, that's not scalable. So that's why we are creating these few short learning techniques. uh Very active area of research, I, don't think, there's any solution for that right now, so, hopefully, by the end of the semester, we'll have something exciting.

F

B

Wonderful, uh okay: there is one other question in the chat uh being uh said in practice. Any AI is a huge volume data for training and working day to day. The contents of the websites are also updated. Frequently, I think the biggest workload for the AI system is training and monitoring the correctness or of its results. Instead of the system, design and implementation. How do you evaluate your AI system in the workloads.

C

uh Yes, information evolves, new information comes in information is corrected. uh This real-timeness is something that none no existing system is able to solve, except for the fact that you can update uh retrain your model from scratch again when new information or when when, as as new information arrives, uh what we do on the other hand, uh is, is that when something new happens, that's related to what we have looked into before. We annotate a few more data points and use that as additional training data that works, but it's not perfect.

C

So that's why, as I just uh saying in in response to the previous question, we are working on these few short learning models that a few short learning misinformation detection models that uh can be that can be retrained or rather trained from scratch.

C

Using only a few data points.

B

uh Okay, thanks shrejan uh is there any other question any more question, if not I think uh yeah we pass the 6 30 Mark. uh Thank you so much for uh your wonderful talk. It was very uh interesting and, uh as you mentioned, uh with chat gptali, the things are getting even more like interesting, exciting and also like uh I'd, say. uh Maybe.

A

B

Scary, a little bit so yeah. Let's, let's see what we have the Futures uh thanks uh again uh yeah with that yeah people are writing. It was such a great session. Thank you thanks. uh So yeah I think we can end the talk here today. All.

E

Right. Thank you. Everybody.

C

Thank you for having me and thanks for the questions.

C

A

A