CONCEPTUAL SCAN: LEARNING WITH AND ABOUT RULES Anonymous authors Paper under double-blind review

Abstract

The ability to learn from a mix of rules and examples and to reflect on the learned abstractions is an important aspect of human intelligence. At the same time, there is a lack of benchmarks that systematically test for this ability, which makes it hard to evaluate the degree to which it is present in state-of-the-art ML architectures. We introduce a novel task format for such benchmarks by using an example structure that allows us to explicitly provide and ask about rules that are relevant for the given task. We present a simple dataset illustrating this format, and we use it to analyze the performance of a variety of T5-based ML models. We identify three challenge areas in this setup: maintaining consistency between learned rules and their application, scaling to larger rule sets, and compositional generalization.

1. INTRODUCTION

Machine learning algorithms are typically designed to be able to learn functions from examples. This is a very general paradigm, but it does not explicitly capture some aspects of human learning. Humans, in contrast, are able to learn both by being shown examples of the task to accomplish and by being told rules or instructions about this task. They can even provide relevant rules to others once they have learned the task from examples. As a realistic illustration of this ability, consider the task of a personal assistant who, among other things, is expected to make movie recommendations based on the age and interests of a user. Even for a task such as this that would currently be considered a standard use case for an example-based recommender system, as humans, we do not learn how to perform this task exclusively by observing examples of movie recommendations. Instead, we can accomplish this task much more efficiently by also taking into account relevant knowledge in the form of rules that have been communicated to us explicitly, i.e., by "learning with rules". For recommending a movie to a girl called Anna, we may, among others, use the rules (and facts, which we consider a special case of rules) illustrated on the left side of Figure 1 . In addition to the ultimate goal of providing movie recommendations (e.g., "What movie could Anna watch?"), we would also expect a human to be able to answer the intermediate questions shown on Figure 1 : Personal assistants answer questions using knowledge consisting of rules and facts. Note that the last knowledge bullet point above can also be construed as an example of the underlying "movie recommendation" task, while the other bullet points represent other relevant knowledge. The first bullet point is a traditional "rule" that states conditional knowledge that can apply to many different movies. The second is a concept definition, which can be equivalently construed as a rule relating two different pieces of information about a person. The other bullet points are facts stated at varying levels of granularity.

