Pupils are celebrating today after receiving an astonishing 43 offers from either the University of Oxford or the University of Cambridge – the highest number in the College’s history. We are immensely proud of our pupils - a huge congratulations!
Last Thursday, Dr Saldanha, an expert in statistical machine learning, gave an informative talk on Natural Language Processing (NLP) and how text is generated using Large Language Models (LLMs). ChatGPT is an example of an LLM, creating full paragraphs of text that seem almost identical to human conversation.
Dr Saldanha explained how small, specialised text generators were easy to build and produced plausible text outputs. These were based on sufficient data and an appropriate probability model.
A specific example showed how a bigram language model (pairs of letters) was used to generate names. Using the probability of specific letters following each other, words close to names that have been created by humans, such as ‘Devon’ or ‘Jordi’, could be generated quickly and easily.
Neural networks can be used in conjunction with such models to create more complex systems that can better generate readable text. Dr Saldanha went on to cover a larger model, where something close to Shakespearean text was generated. Remarkably, this was undertaken on an old (2012) laptop with only around 20 minutes of training.
These models use tokenisation, which maps each letter or character that it uses to a number, so the computer can store and utilise the information in a more efficient manner.
They also use a ‘self-attention mechanism’, which allows the models to generate relevant letters or words by adding weight to words based on their relevancy to the current sentence; as the datasets used are so large, only small portions of the millions of characters and words are sampled.
The talk was brilliant and answered many questions about how sites like ChatGPT worked and how they could improve in the future, due to imperfections, such as ridiculous answers and easily bypassed filters.
Dr Saldanha left us with his view that a true AGI (Artificial General Intelligence) is “something in silicon that behaves in much the same way as a reasonably intelligent human being in any situation” and that what we have currently is not truly AI.