This is, I think, one of the most important thought experiments a person can do.
I mean this quite seriously and earnestly.
Very seldom do we onfront the map-territory distinction in such a raw manner. There's a Scott Alexander post, "What Developmental Milestones Are You Missing?" (http://slatestarcodex. com/2015/11/03/what-developmental-milestones-are-you-missing/) (broken link) that gets some detail into the theory of mind. He discusses the map-territory problem briefly, along with other interesting examples.
There's a attractive trap that shows up in human minds -- all human minds, your and mine included -- that changing the map can somehow change the territory as well. Call it the Voodoo-Doll Bias to be evocative and memorable, but it's present all over the place. Our minds are associative. We see a word, and see the image of the word. That strong mental connection leads us into the belief that by changing the word/map/signifier/representation/doll, we can effect genuine change in the object/territory/signified/represented object/person. Magic words move the world. A "grammar" used to be a book of spells before it was a description of syntax and morphology.
We still see this today. All over the place. Members of the Sovereign Citizen movement think they can navigate a government-free path through the legal system by knowing the right phrases. Classic map-territory problem. Or take the people who have trouble believing 0.999... does not equal 1. Obviously the two representations look very different. Nevertheless, in the real numbers they are the same thing. The actual "territory" of mathematics is a centuries-old discussion between genuine mathematicians working out ideas. The numbers on the page are simply ways of expressing the underlying ideas, and those two representations quite obviously (to those in the know) represent the same point on the territory. Two different maps, but both representing the same place on the number line. Internet math cranks have problems with this, precisely because they believe that the numbers they are writing down are themselves the territory, rather than just a representation of that territory.
(It's interesting that analogies to two written fractions such as 1/2 and 2/4 representing the same underlying number never seem to help these people. Why? Well, the decimal map works very different from the fractional map in their minds. In almost all everyday cases, to write a different number in decimal notation is quite literally to refer to a different spot on the territory. That's how it always works at the store, right? Add to that everyday experience a certain confusion with how mathematicians handle infinity, and then toss in more than a fair share of idiot human pride, and you have a 20-page thread where a pig ignorant person cannot understand that there are two different symbols representing one underlying idea. Map-territory problem. The Voodoo-Doll Bias.)
Another example of this is the insistence on the Right Terminology from political partisan. Again, this is a human universal and not particular to any one subgroup. Many hardcore economic rightists like to say "Taxation is theft!", whereas many hardcore economic leftists like to say "Property is theft!"
Exactly the same deal. It's argument by connotation. They have a word with a strong negative connotation -- theft, in this case -- and they want to associate that negative feeling with a concept that they dislike. Obviously, definitions cannot be wrong as a law of the universe. A definition can be anything that groups of people agree to use. But it's similarly obvious that these definitions are not attempts to understand the nature of the underlying territory. These folks think they already got the world sufficiently figured out, thanks, which is why the negative connotation comes out in force. Map-territory problem.
The little scientist you create has to make a map. The map that they create is not the same as the territory. They might never get a fully understanding of the underlying territory. The program you wrote for that scientist is (likely to be) very small compared to the program for the world in which the agent is embedded.
Your task, in programming the digital scientist, is not to find the absolute truth. You're probably not going to succeed at that. Your task is to program an algorithm that might efficiently discover truth. In order to do that, you are going to have to have your scientist make choices in the world. The agent will make choices based on their "desires" resembling curiosity -- an urge to explore. And the agent will make choices based on their "beliefs" -- their map -- of how the world works. The agent might step on an ant, while avoiding a dragon.
Again, this is not about touching truth directly. It's about having a theory of truth which can help your agent do slightly better. In order to do that, your agent is going to start with certain guidelines built right in. When they take their first step, they will have no direct experience of the world, but will be operating under the initial instructions you gave them. They will not have had any time to build up a reservoir of experiences from which to judge what is plausible in that world, vs what is not plausible. They will instead be using, in their first moment, the prior plausibilities that you gave them. And then they will update those plausibilities according to the rules you gave them.
You might have some interest in their success as a simulated creature. As an embodiment of science and learning and understanding. In satisfying this, you might develop interest in having their beliefs update in an efficient manner.
How would you do that?
There are various choices you might make.
One possibility might be to make 20 random mutations of your original scientist, and then have each one run through the game. Take the best three of those, and make 20 more random mutations. Take the best three of those, and repeat. Most of the mutations will be bollocks. (Maybe you'd need 100 instead of 20.) But some of them could be helpful. Some of them could result in a "science score" that was higher than previous iterations. You can run this again and again.