Avoiding AI ‘Hallucinations’ Means Proper Instructions

This story is part two in our series, “What Is AI?”

Stories abound of people using large language models, colloquially referred to as artificial intelligence (AI), to write reports, only to discover that the computer used fake sources and made-up references.   

Last week, we aired the first part of News Director Eric Douglas’ interview with Professor Ali Al-Sinayyid, an assistant professor and director of the West Virginia State University Cybersecurity Innovation Center, about what is AI.  

In part two of that interview, Al-Sinayyid discusses the best way to use AI to get what you want – and not fabricated or false answers that seem real.  

This interview has been lightly edited for clarity.  

Douglas: You talked about building a prompt. You talked about the best way to create a prompt to get the desired effect or the desired information you’re looking for. Can you walk me through that? 

Al-Sinayyid: Usually, we do three steps. Step #1 is defining the rule. Step #2 is tell him exactly, logically, what you want. Step #3 is to tell him what the format of the final output.  

So, for example, if you want to have a (spread)sheet and you want to develop a budget, right? In that case, if you want to hire somebody for this job, who are you going to hire? A finance specialist? So that’s what you’re going to start with. Act as a financial specialist. That’s your rule. And that’s going to help the AI to target this skill instead of searching in all the skills. The problem with searching with all the skills and being not guided thoroughly or logically, he might give you bad results or false information because he, again, he wants to satisfy you, right?  

So that’s number one. It’s the act. And then consider yourself, you are hiring a person to do this job. Later, you’re going to start by telling the AI what to do and how to do it and when to do it. And by that I mean logically.  

So in the attached sheet, which you’re going to attach it to the AI LLM, in this sheet, you will have column #1, column #2, column #3. And you’re going to describe what it has. And that’s so you can remove any assumption made by the AI, which is going to lead to false results. So you’re going to try to remove all these assumptions. And then you’re going to try to do the sum of column #1, column #2, whatever mathematical operation you need to do the budget. 

Later, before you end step two, you’re going to tell him and this is very, very important, “Ask until you are certain 95%.” And in that case, you will guarantee that if there is a knowledge gap before implementing this order, he will ask you. So now we’re going to develop follow-up questions. So instead of it assuming or filling the gaps by himself, now it can start follow-up questions instead of doing it right away. 

At the end, you’re going to tell him what final format you needed. Okay, now I don’t need same sheet. I need a document like a paragraph to tell me how or to predict to me if my salary could be enough for the next one year, right? What should I do in order to increase the revenue or decrease the cost of spending and so on. And I would like it to have as a Word document or as a PDF or just do it in a different sheet, because he can develop whatever file type you can describe to him.  

Number one, he will be specialized in what you want. Number two, he will follow up with the questions with you, so he will not assume or fill any gaps. Number three, and that’s the most important, I believe, he will provide the document that you want, the format that you want, not what he thinks is better for you. And of course, these three steps will guarantee better results.  

Besides these three steps, I’m going to tell the AI to write the prompt itself. Meaning instead of starting “Act like this, do that in this format,” I’m going to start with this. Write a prompt to do the following and then you’re going to put whatever you like. 

Douglas: You mentioned to ask until it’s 95% sure. It’s never going to be 100%. 

Al-Sinayyid: If I may, it depends on the task. If it’s complicated, yes, you are right. It’s unlikely to have 100%. But if it’s like one answer, true or false, no, it can reach 100%. 

Douglas: You’ve used the term, but we’ve heard stories of people writing reports and AI creating its own references or its own citations. The stuff that doesn’t even exist. Humans that don’t even exist that it’s quoting. We call that hallucinations. How do you verify that AI has created what you’re looking for and it’s not making stuff up? 

Al-Sinayyid: If I can go back a little bit one step, the reason why it’s hallucinating, we said because he is going to fill the gaps or any lack of information, instead of asking you, right, he just going to fill it itself. 

It tries to satisfy your desires. Try to satisfy your request by not asking a lot of follow-up questions. Just do it right away. When we use it as a final solution with a final authority to verify, to build, to design, he’s going to do it itself by not verifying with you. So when you say, write me a research paper that has 10 pages with references about this topic, and you press enter, he’s going to do it. Did you specify the references are real or not? No, you didn’t. Did you say that I needed a hot topic with real information? No, you didn’t. So he’s just providing you a fake one.  

After we follow these three steps and after we ask him to do the prompt, here’s my suggestion. Instead of opening one chat session, I open two sessions at the same time. Session one, and let’s call it the prompt one, and session two, the implementation or the execution one. In session one, I’m going to say write the prompt to do this, act like this, do this in the final prompt. He’s going to give you the prompt, right? You’re going to copy it from session one. You’re going to put it in session two, and you press enter. The result is going to be developed, created from session two, the execution one, you’re going to copy it, bring it back to session one, and you’re going to say this, verify. 

Douglas: Verify the prompt you created. 

Al-Sinayyid: Then he’s going to verify. He’s going to tell you either that it’s done, or no, I find something wrong. Here’s the prompt. Do it again to correct it. 

Douglas: Are you talking using the same AI system, or are you talking using Claude versus ChatGPT? 

Al-Sinayyid: I really don’t prefer that for many reasons, but the simplest one that the audience can relate to and can understand, they have different environments, different languages, different sets of rules. The verification is in one platform.