Knowledge Observation Group

The Knowledge Observation Group (KOG) intends to test and use Large Language Models (LLMs) to investigate the relationship between knowledge acquired through human socialisation and knowledge acquisition by machines.  It combines social science and computer science, that combination being reflected in the membership.  KOG invites application from interested parties with relevant knowledge and expertise.

The founding project is to understand the capabilities and flaws in current LLMs.  Many groups are already doing this but our approach, which begins with comparing human and machine socialisation, is unique and, we believe, will lead to a deeper understanding than more haphazard approaches.  Continual invention of new kinds of revealing prompts will be part of the project but structured by a classification of types and causes of failure based on comparison with human abilities and exposure to knowledge.  A panel of human volunteers will be established to check our beliefs about how humans would respond to various prompts.

One feature of the social science background  – ‘Studies of Expertise and Experience (SEE)’ – is that human language is determined to be a much more important component of human socialisation than it is in other philosophical treatments of human abilities, such as those that emphasise practice.  The idea of ‘interactional expertise’ is central to KOG: it shows that fluency in language is crucial to the division of labour and that language captures and facilitates many features of practice even when fluency does not lead to the ability to execute the corresponding practices (consider the physically challenged as an extreme case).  Therefore, SEE is a natural perspective through which to approach the study of LLMs.  A further consequence is that the Turing Test, understood as a comparison of human and machine fluency though with more carefully worked out protocols than Turing’s 1950s description, is still an adequate test of machine abilities and our exploration of LLMs abilities can be seen as refinements of the Turing Test.

Other projects include testing LLMs on specific branches of knowledge such as gravitational wave physics and using LLMs to explore human socialisation in new ways.