Section 1.1: The Origins of "Delve"

Returning to the question, "What caused ChatGPT to adopt the term 'delve' so prominently?" If ChatGPT's usage of "delve" is disproportionate compared to everyday language and online content, it suggests that its language patterns were modified after being trained on a broad data set.

Following extensive training on vast amounts of data, measures are implemented to ensure the AI remains focused and aligned with human expectations. This is achieved through a process called Supervised Learning, where human annotators assess the model's outputs. Their evaluations are crucial for refining the model.

Here's a concise overview of this process:

1. A foundational Language Learning Model (LLM) is trained using a comprehensive data set.
2. A Reward Model (RM) is introduced to discern what humans deem "good" and "aligned."
3. The LLM produces multiple outputs, from which humans select the most appropriate.
4. The RM learns from these human selections, translating preferences into rewards or scores.
5. The LLM receives feedback and adjusts its behavior to garner more rewards.
6. This cycle continues: the LLM generates outputs, humans provide feedback, and the model iteratively improves.

This general framework of Reinforcement Learning from Human Feedback (RLHF) is what helps ChatGPT remain cautious and focused. However, this effective alignment hinges on human evaluation. This evaluation can involve simple approval or disapproval of outputs or providing ideal responses that the LLM is expected to generate.

The volume of feedback given is minuscule compared to the vast text corpus used for the initial training. Nonetheless, providing sufficient feedback is labor-intensive and costly, leading large AI firms to outsource this work to regions in the global south, where English-speaking labor is more affordable.

As noted by The Guardian, an extensive network of cost-effective freelancers in Africa undertakes these annotation tasks. In Nigeria, the term "delve" appears more frequently in business English than it does in the UK or the US. Consequently, the workers who are training the AI systems provide input that reflects this linguistic preference, resulting in an AI that communicates in a manner reminiscent of African English.

This situation exemplifies poor sampling (Selection Bias), where the evaluators differ from the intended user demographic, potentially skewing the writing style. To ensure a better sampling for an AI Assistant meant for global English speakers, it is essential to include annotators from diverse backgrounds and styles.

Section 1.2: Addressing the Biases

While it's unclear if this leads to subtle biases in ChatGPT's writing style, the underlying issue likely stems from the RLHF phase rather than the initial training. The writing style of ChatGPT is already somewhat robotic and easily identifiable, regardless of its use of "delve." However, there's a significant lesson here: understanding where things could go wrong in this process can help prevent similar pitfalls in future research and development efforts.