Triple navigation evaluation

The evaluation of the triple search was carried out on a list of ==100 questions== that where generated by gpt-4-turbo. The requirement for these questions was, that they can be answered by providing a semantic triple. For each question the navigating agent and the generating agent performed a search of a ==maximum of ten iterations== for a triple that is answering the question. They started from the triple ["Physics", "is a branch of", "Science"] and the topic "Physics".

The whole process was conducted twice. Once with gpt-3.5-turbo as navigating and generating agent and once with gpt-4-turbo as navigating and generating agent. The examining agent was in both cases gpt-4-turbo.

Here is an example for a successful triple search of the gpt-4-turbo model: The search for the answer of the question "What protects Earth from solar radiation?"

Iteration	Triple	Topic
0	[5]	Physics
1	["Magnetism", "is studied as a branch of", "Science"]	Electromagnetism
2	["Magnetism", "is a key component of", "Electromagnetism"]	Electromagnetism
3	["Dynamo theory", "explains creation of", "Electromagnetism"]	Magnetic fields
4	["Protective shield around Earth", "is created by", "Electromagnetism"]	Earth’s magnetic field
We can see how the navigation process starts from the initial triple and navigates through the knowledge base in four iteration until it reaches the triple that fulfills the condition of answering the question.

This example also shows that the examiner agent is not completely strict in the requirements that a tripe has to fulfill to answer a question. The final triple ["Protective shield around Earth", "is created by", "Electromagnetism"] does not explicitly state, that the shield protects the earth from solar radiation.

Here is an list of five typical questions from the same dataset and the according triples that have been found by the navigation process:

Question	Triple
What does the Tesla unit measure?	["Magnetic flux density", "is quantified in", "Tesla"]
What does a laser emit?	["Laser", "utilizes", "Electromagnetic waves in Science"]
What allows particles to overcome potential barriers in quantum mechanics?	["Quantum tunneling", "exploits", "wave-particle duality"]
Which planet has an extensive ring system?	["Saturn", "has the most extensive", "Planetary rings"]
What state of matter is composed of charged particles?	["Ionosphirical plasma", "yields", "highly ionized particles"]

Out of the 100 triple search runs, gpt-3.5-turbo completed 27 successful while gpt-4-turbo completed 43 successful. The calculated decision cost of non interrupted searches was:

	gpt-3.5-turbo	gpt-4-turbo
decision cost	91.1 ± 17.0	50.7 ± 7.1

These numbers correspond to the average number of navigation decisions the navigation agent has to make to find a question answering triple if there are 10 options available per decision.

Data Code