Skip to content

For narrow tasks, a small, specialized model is often enough.

A decision diagram based on a plumbing supply wholesaler that sorts its central email inbox. A small classification model, trained on hand-labeled emails from the company’s own archive, assigns each email a request type and picks out the order number and amount. If confidence is above the set threshold, the email goes straight to the responsible department; otherwise an employee reviews it, and corrected cases go into the next training run. Below: a large language model only drafts replies to complaints.

Natural language processing covers every method computers use to handle written and spoken language. Classic tasks include sorting texts into categories, spotting names, places, dates, and amounts (named entity recognition), translation, summarization, and sentiment analysis. Speech recognition, which turns speech into text, is a neighboring field.

NLP is a field, not a product. Large language models (LLMs), such as the models behind ChatGPT, are one tool within it and currently the best known. Small models trained for a single task and hand-written rule sets sit alongside them. Neuro-linguistic programming, a communication and coaching method, shares nothing with the field but the acronym.

NLP concerns executives because much of a company’s work is text: email, contracts, tickets, reviews. Leaders who know which task a method solves, and how reliably, can compare proposals and see where a smaller model will do.

Chapter 1 of 7

An NLP system breaks the work into steps that can be tested.

Classic NLP systems work as a pipeline. Tokenization splits text into words, word pieces, and punctuation. Lemmatization reduces inflected forms to their base form, so “invoices” becomes “invoice.” Named entity recognition tags proper names; spaCy’s documentation describes a named entity as a real-world object that’s assigned a name, such as a person, a country, or a product. A classifier then assigns a category.

German makes some steps harder: it produces many forms of the same word and long compounds such as “Kundendienstanfrage” (customer service request). A system that only matches identical strings misses hits.

Neural models now handle many of these steps in one pass. The breakdown still helps with testing: it shows where errors creep in.

Chapter 2 of 7

Early systems ran on dictionaries and rules.

On July 15, 1949, Warren Weaver of the Rockefeller Foundation sent out a memorandum titled “Translation.” He proposed computers as translators and named the core problem: a word read through a narrow slit is often ambiguous. “Fast” can mean “rapid” or “motionless.” With enough words visible on either side, the meaning becomes clear. Weaver therefore called for statistical studies of language.

On January 7, 1954, Georgetown University and IBM demonstrated Russian-to-English translation on an IBM 701 in New York. According to IBM’s press release, the system translated more than sixty sentences with a dictionary of 250 Russian words and six rules. Leon Dostert of Georgetown estimated that scientific writing generally might need up to one hundred such rules.

Rule systems are easy to trace, but they fail on any phrasing nobody wrote a rule for. Later, statistical methods learned probabilities from large text collections.

Chapter 3 of 7

Embeddings and transformers rebuilt the field.

In January 2013, Tomas Mikolov and colleagues at Google described models that represent each word as a vector of numbers; the method became known as word2vec. Words used in similar contexts end up close together. Most semantic search today runs on such vectors, called embeddings.

In June 2017, Ashish Vaswani and colleagues published “Attention Is All You Need.” Their transformer architecture relies entirely on attention: for each word, the model weighs which other words in the text matter for its meaning. The authors tested it on English-to-German and English-to-French translation.

BERT, from Jacob Devlin and colleagues, followed in October 2018. It learns from unlabeled text, reading the context on both sides of a word at once; one added output layer adapts it to tasks such as question answering. In May 2020, Tom Brown and colleagues showed with GPT-3 that a large language model can solve tasks described in plain text with a few examples, without extra training.

Chapter 4 of 7

NLP is the field; an LLM is one tool in it.

Artificial intelligence is the umbrella term. Machine learning is the branch in which software learns from example data; deep learning uses many-layered neural networks. NLP cuts across these layers because it is defined by its subject, language, rather than by a method. A rule set that pulls invoice numbers out of emails is NLP without machine learning; a large language model is NLP with deep learning.

Within the field, natural language understanding (NLU) means recognizing intent and meaning, and natural language generation (NLG) means producing text. An LLM does both.

The distinction helps in vendor conversations. A claim that a product “uses NLP” says little. The questions are which task it solves, with which model, and with what measured accuracy on the company’s own texts.

Chapter 5 of 7

For narrow tasks, a small, specialized model is often enough.

Narrow means fixed categories, many similar texts, and enough examples labeled correctly by people. Sorting an inbox or extracting invoice data often meets that bar. A model like this responds quickly, costs little per text, and can run on in-house servers.

DistilBERT showed in 2019 how much a small model can do. Victor Sanh and colleagues shrank BERT by 40 percent; according to the paper, the model kept 97 percent of BERT’s language understanding and ran 60 percent faster.

An LLM pays off when tasks keep changing, examples are scarce, or it has to write text itself. It needs no training of its own, but every call costs money and its output needs review. The two can be combined: an LLM handles the task until enough reviewed examples exist, then a small model takes over.

Chapter 6 of 7

What good implementation looks like.

It starts with a test set of the company’s own texts, labeled by people. Vendor sample data says nothing about the company’s own customers and how they write.

The model reports a confidence score with every decision. Below a set threshold, a person reviews the case. Where the threshold sits depends on what an error costs: a misfiled marketing email is harmless, a missed contract cancellation is not.

Every decision is logged. When products or forms change, the test set runs again, and corrected cases feed the next training run. If the texts contain personal data, data protection law applies; where processing happens must be settled up front.

Costs come from data preparation, integration, review, and upkeep; LLM services also bill per token.

Chapter 7 of 7

Where the term stops working

NLP processes language statistically and produces probabilities, not certainty; fluent output proves no understanding. The field differs from the LLM, one tool within it, from machine learning, a method that also handles images or sensor data, and from speech recognition, which first turns speech into text.

Frequently asked questions

What is natural language processing (NLP)?

Natural language processing is the field in which computers process human language: they classify, analyze, translate, summarize, or generate text. It ranges from fixed rules to large language models.

What is the difference between NLP and AI?

AI is the umbrella term; NLP is the branch that deals with language. Other branches work with images or sensor data.

Is ChatGPT an NLP model?

ChatGPT is an NLP application built on large language models. The models behind it are transformers pretrained on large amounts of text; they analyze and generate text.

What is the difference between NLP and NLU?

NLU (natural language understanding) is the part of NLP that recognizes intent and meaning. Generating text is called NLG (natural language generation); NLP covers both.

What do companies use NLP for?

Companies use NLP wherever text piles up: sorting emails and tickets, extracting data from invoices and contracts, analyzing customer reviews, and running chatbots. Only a test on the company’s own texts shows the benefit.

Does NLP require a large language model?

No, for narrow tasks with fixed categories a small, specialized model is often enough. A large language model pays off when tasks change, text has to be written freely, or few training examples exist.

Does NLP have anything to do with neuro-linguistic programming?

No, the two share only the acronym. Neuro-linguistic programming is a communication and coaching method; natural language processing is a field of computer science.

    ↑ ↓ select · Enter open · Esc close