This is a guest post by Professor José Hernández-Orallo from the Technical University of Valencia. He shares his experience of working on metrics of machine intelligence two decades ago when there was little interest in measuring the intelligence of AI. He discusses the metrics of intelligence linked to algorithmic information theory, where intelligence was formulated using theories of inductive inference.

He goes on to talk about the recent advancements in AI, particularly in the field of machine learning, and the increasing interest in evaluating artificial general intelligence (AGI) systems. He mentions the introduction of AI evaluation platforms such as Microsoft’s Malmö, GoodAI’s School, OpenAI’s Gym and Universe, DeepMind’s Lab, Facebook’s TorchCraft, and CommAI-env. These platforms provide a standard interface for connecting reinforcement learning agents and creating different tasks for evaluation.

The author highlights the importance of a system’s ability to reuse representations and skills from one task to new ones, similar to how humans learn. He refers to this capability as “compositionality” and emphasizes its relevance in evaluating AI agents. He suggests that platforms like Malmö and CommAI-env are well-suited for evaluating compositionality.

He explains that CommAI-env stands out from other platforms as it focuses on communication skills and keeps interactions simple. He praises its use in the warm-up round of the General AI Challenge, where participants can focus on evaluating RL agents without the complexities of vision and navigation. The author acknowledges that vision and navigation are important but can create additional complications when evaluating gradual learning. He believes that starting with a minimal interface to test incremental learning is a challenging and important problem for general AI.

The author mentions that bits in CommAI-env are packed into 8-bit characters to make tasks more intuitive and transparent to the agents. This allows for the composition of actions and observations in solving tasks. He compares this approach to Turing machines and symbolic AI, and suggests that techniques such as Neural Turing Machines and neural networks with symbolic memory can be suited for this problem.

He concludes by mentioning that deep reinforcement learning enthusiasts can adapt their techniques to participate in the warm-up round. He believes that this challenge opens up the opportunity for various AI techniques, including natural language processing, evolutionary computation, compression-inspired algorithms, and inductive programming. He expresses his excitement to see how the round develops and what participants are able to achieve.