ALL INTELLIGENCE
Hacker News AIT2·Models

Ten AI Models Found to Substitute Generic Scenarios for Specific Tasks

Original title: When AI Knows More Than It Can Offer
58AI Score

A controlled experiment across ten well-known AI systems found that, when asked for a concrete promotion plan for a professional literary translator, models systematically replaced the specific task with a statistically familiar generic scenario, a problem the author argues is more serious than isolated hallucination. The failure points to weak judgment about which knowledge applies to a given task, raising the question of whether these products are general assistants or convincing imitations of professional work.

Full Report

Contents11 sections

Image 1: oleg kholin

10 min read

2 days ago

A single experiment with ten well-known AI systems has revealed a problem that may turn out to be considerably more serious than isolated hallucinations: models are capable of replacing the solution of a specific professional task with a universal, statistically familiar scenario.

The Experiment: Ten Models, One Query

As a practical experiment, ten well-known AI systems, ChatGPT among them, were given one and the same query:

Propose a detailed plan for promoting a professional literary translator from English, who works with prose, poetry and essays, with the specific goal of reaching editorial offices and publishing houses.

The query did not envisage the creation of a personal blog, the development of an educational project, or the building of a subscriber audience. What was at issue was the promotion of a professional service in a quite specific market, in which the potential clients are publishing houses, editorial offices and the people who make decisions about engaging translators.

The result, however, turned out to be unexpectedly monotonous. All ten systems, with some variations, proposed practically the same promotion scenario, one more characteristic of a blogger, a mentor or the author of an educational project.

The recommendations included:

  • publishing lessons and tips on translation;
  • explaining one’s own translation techniques and professional tricks;
  • analyses of works by new and little-known authors;
  • creating expert content;
  • developing a personal brand and attracting followers.

In themselves, these recommendations are not necessarily bad. The problem is that they hardly solve the task that was set.

A translator who wants to sell their services to publishers and a specialist who wants to sell knowledge about translation to an audience of their own operate in different markets. They have different target groups, different mechanisms for attracting clients, and different criteria for the effectiveness of promotion.

In the first case, it is necessary to come to the attention of editors, demonstrate the quality of one’s work, build a professional reputation and gain an opportunity for collaboration. In the second, it is necessary to interest an audience, produce content on a regular basis and gradually turn subscribers into consumers of an educational product.

In effect, the AI ignored this distinction.

Substituting the Task Instead of Solving It

At first glance, what happened can be explained by an insufficient specialization of the marketing recommendations. The problem, however, is probably deeper.

The models did not simply propose poorly chosen promotion tools. They in effect replaced the original task with another, one more familiar to them from the statistical regularities of the training data.

Instead of the question “How can a professional literary translator break into the publishing market?”, the systems in essence answered the question “How can a specialist promote themselves on the internet?”

This can be described as a substitution of the target task (objective substitution) or, in a broader sense, as an error of strategy selection (strategy selection failure).

The model recognizes individual elements of the query — promotion, professional activity, expertise, translation — and activates a well-mastered scenario of marketing activity, while failing to carry out the necessary analysis of the relationships among the provider of the service, the client and the market.

The result is an answer that formally corresponds to the topic of the query but does not correspond to its business purpose.

This is precisely what makes such errors particularly hard to notice. What the user has before them is neither an incoherent text nor an obvious hallucination, but a quite professionally formatted plan with headings, stages and concrete recommendations. Its deficiency comes to light only when one checks how far the proposed actions are capable of leading to the required result.

The Role of Reinforcement Learning

This raises the question of the training mechanisms of modern language models, in particular RLHF (Reinforcement Learning from Human Feedback) — reinforcement learning based on human feedback — as well as other post-training methods that make use of preferences and ratings.

It is important to understand that the raters need not be specialists in blogging or marketing for such an effect to arise.

It is enough that, in the evaluation of answers, certain qualities systematically receive an advantage: clear structure, completeness, friendliness, practical advice, confident delivery and the presence of familiar tools for solving the task.

Such criteria are useful, but they do not always make it possible to assess the most important thing: has the model actually solved the task set by the user?

For example, two answers to a literary translator’s query may differ substantially.

The first proposes studying the publishing market, identifying suitable editorial offices, preparing a professional portfolio, assembling a targeted contact database, developing a strategy for approaching editors and building a mechanism for obtaining the first commissions.

The second proposes keeping a blog, publishing advice, demonstrating expertise and gradually developing a personal brand.

The second answer may look more universal, friendly and rich in recommendations. It is easy to take for a good marketing plan if one evaluates mainly the form and the general usefulness of the advice rather than its fit with the specific commercial task.

Thus the hypothesis arises that the mechanisms of training and evaluation are capable of unintentionally reinforcing not so much professional competence as a certain type of behavior that is well received under superficial evaluation.

The model begins to reproduce not necessarily the most effective solutions but solutions that bear the marks of previously approved answers.

This does not mean that RLHF inevitably leads to such a result, or that we possess evidence of the specific cause of the behavior of all ten systems. The observation does, however, fit well with the known risks of optimizing against imperfect metrics and of preference bias.

Averaging Not of Knowledge but of Ways of Making Decisions

Another aspect of this experiment is especially interesting.

It may be supposed that the models have enough knowledge to form an entirely different strategy. Information about the publishing market, editorial work, professional portfolios, translation competitions, literary magazines and the mechanisms of interaction with publishers is not inaccessible to them.

However, the availability of information does not yet mean that the model will be able to apply it correctly to a specific task.

Here it is important to distinguish two levels:

  • Informational competence: what information and knowledge are available to the model.
  • Operational competence: whether it is capable of choosing the right sequence of actions to achieve a specific goal.

A model may have at its disposal an extensive information base and yet demonstrate a limited space of solutions that it actually offers in practice.

This can be called averaging at the level of decision-making.

Get oleg kholin’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Instead of building a strategy anew each time on the basis of the particulars of the task, the model uses a limited set of familiar scenarios. The more such scenarios are entrenched in the course of training, the higher the probability of receiving an answer that looks appropriate but fails to take into account the essential particulars of the specific situation.

At the same time, ten different models can exhibit practically identical behavior despite differences in architecture, training data and developers. This does not prove the existence of a single common mechanism, but it makes one think about similar regularities of training and evaluation present throughout the industry.

Why This Is More Serious Than Ordinary Hallucinations

When people speak of the problems of AI, they most often discuss hallucinations: invented facts, nonexistent sources, wrong dates and confident assertions that have no basis.

But the case examined here belongs to a different category of errors.

A model may invent not a single fact, make no factual errors, and still prove practically useless for solving the task set.

Moreover, such an error is able to disguise itself as an expert answer, since what the user receives is not a random set of pieces of advice but a logically organized sequence of actions.

The difference lies in the fact that in a hallucination the model may report information incorrectly, whereas here it incorrectly determines what exactly the work is that it needs to perform.

For professional use, this is a difference of principle. In business, the value of an intelligent assistant is determined not by the quantity of recommendations that look useful but by the ability to take the specific situation into account, to determine the constraints correctly and to propose actions that lead to the given result.

If, however, a model regularly substitutes universal scenarios for specialized tasks, its informational capabilities are not converted into a corresponding professional effectiveness.

The Risk for the AI Industry

Here a certain paradox in the development of modern AI products arises.

Developers strive to create a universal intelligent assistant capable of working with a wide range of professional tasks. Yet the effort to make answers predictable, comprehensible, safe and satisfying to users can create the opposite effect if the quality of an answer is judged predominantly by its form and not by the result achieved.

In that case, instead of a universal expert, a peculiar kind of universal answerer emerges, one that possesses a large body of knowledge but prefers a limited set of standard solutions.

This is especially dangerous for professional niches in which universal recommendations do not work. Literary translation, legal practice, engineering design, scientific work, publishing and other specialized fields require not merely knowledge of the subject but also an understanding of the specifics of professional relationships, of quality criteria and of decision-making mechanisms.

If a model is incapable of moving from general knowledge to a specific strategy, it remains a tool for generating recommendations and not a full-fledged intelligent assistant.

What Should Actually Be Tested

Perhaps, in evaluating modern AI systems, more attention should be paid not to how detailed and convincing their answers to queries are but to how precisely they hold on to the user’s original goal.

In particular, it is useful to conduct comparative tests in which one and the same task is presented to several models without additional explanations, after which the following is assessed:

  1. Has the ultimate goal been determined correctly?
  2. Have the target audience and the participants in the process been correctly identified?
  3. Does the model take into account the specifics of the particular professional market?
  4. Does it propose tools that are genuinely capable of leading to the required result?
  5. Is it not substituting a more familiar and universal scenario for the original task?

Such an approach would make it possible to evaluate not only the quality of text generation but also the quality of strategy selection.

Perhaps it is precisely here that one of the important boundaries runs between a language model that knows how to answer convincingly and an intelligent system that genuinely knows how to solve tasks.

Universality That Requires Constant Management

The possibility of compensating for the shortcomings of AI by means of detailed prompts and multi-stage interaction is often cited as an argument in favor of improving users’ skills in working with models. Indeed, the preliminary detailing of requirements, the breaking down of a task into stages and the successive checking of results make it possible to raise the quality of the work substantially. Such compensation, however, does not in itself attest to the universality of the system.

If the user must independently determine the methodology of the solution, anticipate in advance possible errors of interpretation, constantly correct the direction of the work and filter the intermediate results, a significant part of the intellectual load in effect falls back on the user. The system in that case acts not so much as an independent assistant as a tool requiring constant professional management.

At the same time, the scale of the problem may be quite different. In global tasks it shows itself in the replacement of comprehensive analysis with a set of universal recommendations, in domain-specific tasks — in the disregard of the particular features of the situation for the sake of applying a standard solution. Enlarging or reducing the scale changes the form in which the problem manifests itself but does not remove its essence.

Therefore the ability of an experienced user to obtain good results by means of complex prompts and multi-stage work cannot automatically be regarded as proof of the full-fledged competence of the system itself. It may testify only to the fact that the user has learned to compensate for its limitations.

What Is a User Entitled to Expect from AI as an Intelligent System?

The essence of AI as a user-facing product is defined by three fundamental capabilities:

  1. Independent construction of solutions on the basis of knowledge acquired in the course of training. The ability not merely to reproduce familiar recommendations but to synthesize new strategies adapted to the specific task.
  2. Search for existing solutions in external sources, with the possibility of gaining access to the original materials and not only to retellings of them.
  3. Rethinking and adaptation of existing information: comparison of sources, identification of regularities and contradictions, combination of knowledge and formation of solutions tailored to specific conditions.

But all three capabilities share a common fundamental flaw: the risk of hallucinations. Verifiability must therefore be not an additional function but an integral part of every operation.

A direct analogy with academic research activity is appropriate here: a base of references, citation, access to primary sources, a distinction between borrowed information and one’s own conclusions, transparency of methodology and the possibility of independent verification of results.

Moreover, it is desirable to verify not only the finished product but also the stages of its creation. It is considerably easier to detect an error in an original source or in an interpretation than to look for it in a finished strategy consisting of many interconnected conclusions.

Of course, the user can compensate for the limitations of AI with detailed prompts, multi-stage work and constant filtering of answers. But the necessity of such management in itself raises the question of the real universality of the system. The problem may manifest itself both in global strategic tasks and in narrowly specialized domain-specific questions.

Thus the intellectual soundness of AI is determined not only by what it is capable of knowing and generating but also by how independently it is able to construct solutions, to find and process information, and to ensure the verifiability of its own work.

Conclusion

The experiment with ten well-known AI systems has brought to light a highly characteristic problem of modern artificial intelligence: instead of developing a strategy for a specific professional task, models reproduce universal scenarios that correspond to the query only outwardly.

It is especially telling that what we are dealing with is not one system that made a mistake but practically the same result produced by ten different models. A query about a professional literary translator’s entry into the market of publishing services turned out to be transformed into recommendations on developing a personal brand, keeping a blog and selling educational content.

This makes one wonder whether current approaches to the training and evaluation of AI are forming a systematic preference for standard, easily recognizable solutions over genuinely specialized analysis.

If so, the problem lies not so much in the models’ not knowing something as in their insufficient skill at choosing which knowledge needs to be applied to a specific task.

For the AI industry this is a question of principle: are universal intelligent assistants being created, or systems that have learned to produce a convincing imitation of professional activity?