Penn Arts & Sciences · Data Driven Discovery Initiative

AI Resources for Penn Arts & Sciences

A curated guide to AI and LLM tools for researchers: what's available, what it genuinely does well, and where the risks lie.

AI Tools Matrix AI for Learning AI for Research

Before you trust it

How the four fit together

Most online arguments about whether a model is good turn on four words: hallucination, confabulation, unfaithfulness, sycophancy. They answer two independent questions — does it stick to the source you gave it, and does it hold its answer when you push back — which is why you can be caught by one failure while carefully checking for the other.

	Sticks to the source	Invents freely
Holds its answer	Holds its answer · Sticks to the source Worth trusting Grounded in what you gave it, and it does not fold when you push. Still check it, but this is the behaviour you are looking for.	Holds its answer · Invents freely Confidently wrong Hallucination · Confabulation It will defend an invented citation as readily as a real one. Pushing back does not help here, because the problem was never a lack of conviction.
Tells you what you want	Tells you what you want · Sticks to the source Right until questioned Sycophancy The answer was fine. Your doubt was enough to move it. This is the failure that punishes you for checking.	Tells you what you want · Invents freely Tells you whatever you want Both at once It invents, and it re-invents in whichever direction you lean. Leading questions produce this, which is a reason to ask neutral ones.

Sticks to the source

Invents freely

Holds its answer

Holds its answer · Sticks to the source

Worth trusting

Grounded in what you gave it, and it does not fold when you push. Still check it, but this is the behaviour you are looking for.

Holds its answer · Invents freely

Confidently wrong

Hallucination · Confabulation

It will defend an invented citation as readily as a real one. Pushing back does not help here, because the problem was never a lack of conviction.

Tells you what you want

Tells you what you want · Sticks to the source

Right until questioned

Sycophancy

The answer was fine. Your doubt was enough to move it. This is the failure that punishes you for checking.

Tells you what you want · Invents freely

Tells you whatever you want

Both at once

It invents, and it re-invents in whichever direction you lean. Leading questions produce this, which is a reason to ask neutral ones.

The distinction most evaluation posts turn on: a claim can be unfaithful and still perfectly true. If you ask for a summary of a paper and the model adds a correct fact that the paper never mentioned, it has been accurate about the world and unfaithful to your source. Faithful and correct are measured separately, and a model can pass one while failing the other.

Read the 10-minute guide

About this resource

Maintained and edited by Yuxin (Elena) Liang, data scientist at the Data Driven Discovery Initiative (DDDI), Penn Arts & Sciences. Access status is updated as Penn license agreements change. Inspired by Penn Carey Law AI Resources, maintained by R. Polk Wagner. If you have any concerns, contact us at yuxinlg@upenn.edu.

Contributors

Listed alphabetically by surname.

Bhuvnesh Jain
Yuxin (Elena) Liang
Terhi Nurminen
Colin Twomey

With thanks to everyone who reviewed, corrected, or argued with a draft of this guide.