BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//pretalx.iacapconf.org//iacap-2026//speaker//CQ9RLT
BEGIN:VTIMEZONE
TZID:US/Central
BEGIN:DAYLIGHT
DTSTART:20250715T000000
TZNAME:CDT
TZOFFSETFROM:-0500
TZOFFSETTO:-0500
END:DAYLIGHT
BEGIN:STANDARD
DTSTART:20251102T020000
RDATE:20261101T020000
TZNAME:CST
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:20260308T030000
RDATE:20270314T030000
TZNAME:CDT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
SUMMARY:Assessing whether LLMs Provide Acceptable Substitutes for Human Ju
 dgments in a Digital Philosophy Project - Colin Allen\, Nazhah Mir
DTSTART;TZID=US/Central:20260715T140000
DTEND;TZID=US/Central:20260715T143000
DTSTAMP:20260726T091013Z
UID:pretalx-iacap-2026-WJXDCR@pretalx.iacapconf.org
DESCRIPTION:We consider the application of LLMs for a digital humanities p
 roject. The Internet Philosophy Ontology (InPhO) project (inphoproject.org
 ) organizes concepts from the Stanford Encyclopedia of Philosophy (SEP) in
 to a taxonomic hierarchy supplemented by non-taxonomic relationships. The 
 InPhO concept graph is inferred from automated statistical analysis of SEP
  content and human judgments about concept relatedness. The need to collec
 t human judgments made it hard to scale up the original project. The appea
 rance of LLMs raises the question of whether LLM-generated judgments could
  be substituted for human judgments. We tested this idea using five differ
 ent LLMs prompted to adopt different levels of philosophical expertise\, W
 hen prompted to adopt higher expertise levels\, two of the LLMs provided c
 loser matches to human judgments at the corresponding levels than the othe
 r models. We also found that most of the LLMs showed less variance when pr
 ompted to respond at the level of a philosophy doctoral student\, mirrorin
 g the finding in the original project that doctoral students showed more c
 onsistency in their judgments than both higher- and lower-expertise human 
 respondents. We will discuss whether LLM judgments are of sufficient quali
 ty to fulfill the InPhO project’s objectives.
LOCATION:Executive Conference Room
URL:https://pretalx.iacapconf.org/iacap-2026/talk/WJXDCR/
END:VEVENT
END:VCALENDAR