Philosopher Ludwig Wittgenstein asserted that “the meaning of a word is its use in the language,” an insight linguist J.R. Firth famously codified: “You shall know a word by the company it keeps.” Vector semantics operationalizes this distributional hypothesis by embedding discrete linguistic symbols into high-dimensional geometric vector spaces, unlocking continuous semantic representations.
1. The Distributional Hypothesis and Vector Spaces
Traditional symbolic representations suffer from the curse of one-hot orthogonality. If a vocabulary contains (|V| = 50,000) words, representing each word as a sparse one-hot vector (mathbf{w}_i in {0, 1}^{|V|}) implies that all word vectors are equidistant and completely orthogonal:
[
mathbf{w}_{ ext{cat}}^ op mathbf{w}_{ ext{dog}} = 0, quad mathbf{w}_{ ext{cat}}^ op mathbf{w}_{ ext{refrigerator}} = 0
]
Orthogonal vectors fail completely to capture semantic relatedness. The Distributional Hypothesis posits that words occurring in similar linguistic contexts share similar semantic meanings. By projecting words into a dense real-valued embedding space (mathbb{R}^d) (where (d ll |V|), typically (d in [100, 300])), semantic similarity maps to geometric closeness measured by the cosine similarity:
[
ext{Cosine}(mathbf{u}, mathbf{v}) = rac{mathbf{u} cdot mathbf{v}}{|mathbf{u}| |mathbf{v}|} = rac{sum_{i=1}^d u_i v_i}{sqrt{sum u_i^2} sqrt{sum v_i^2}}
]
2. Mikolov’s Word2Vec: Skip-Gram with Negative Sampling (SGNS)
In 2013, Tomas Mikolov and colleagues at Google introduced Word2Vec, demonstrating that high-quality dense vector representations can be trained efficiently over billions of tokens using shallow two-layer neural architectures.
In the Skip-Gram architecture, the model takes a center target word (w_t) and predicts the probability of surrounding context words (w_{t+j}) within a sliding window of radius (C). Parameterized by target embedding matrix (mathbf{W} in mathbb{R}^{|V| imes d}) and context matrix (mathbf{C} in mathbb{R}^{|V| imes d}), the exact softmax probability is:
[
P(w_c mid w_t) = rac{xp(mathbf{v}_c^ op mathbf{v}_t)}{sum_{w in V} xp(mathbf{v}_w^ op mathbf{v}_t)}
]
Because computing the full vocabulary partition function in the denominator is computationally prohibitive ((mathcal{O}(|V|))), Word2Vec introduces Negative Sampling (SGNS). SGNS re-casts the multi-class prediction problem as binary logistic regression: distinguish true target-context pairs from (k) randomly drawn noise words (drawn from the unigram distribution raised to the (3/4) power, (P_n(w) propto U(w)^{3/4})).
The SGNS objective function maximized for each target word (w) and true context word (c) is:
[
mathcal{L}_{ ext{SGNS}} = ln sigma(mathbf{v}_c^ op mathbf{v}_w) + sum_{i=1}^k mathbb{E}_{n_i sim P_n} left[ ln sigma(-mathbf{v}_{n_i}^ op mathbf{v}_w)
ight]
]
where (sigma(z) = rac{1}{1 + e^{-z}}) is the sigmoid function. SGNS trains in linear time (mathcal{O}(k cdot d)), scaling effortlessly to multi-billion-token corpora.
3. GloVe: Global Vectors for Word Representation
While Word2Vec samples local sliding windows, Jeffrey Pennington, Richard Socher, and Christopher Manning (Stanford, 2014) introduced GloVe, combining the global statistical advantages of matrix factorization with the local context window strengths of Skip-Gram.
GloVe constructs a global word-word co-occurrence matrix (X), where (X_{ij}) counts how often word (j) appears in the context of word (i). GloVe minimizes a weighted least-squares objective:
[
J = sum_{i, j=1}^{|V|} f(X_{ij}) left( mathbf{w}_i^ op ilde{mathbf{w}}_j + b_i + ilde{b}_j – ln X_{ij}
ight)^2
]
where the weighting function (f(X_{ij}) = min(1, (X_{ij}/x_{max})^alpha)) (with (alpha = 0.75)) prevents common stop words (like “the”, “is”) from dominating the objective.
4. Emergent Vector Arithmetic and Linear Analogies
The most astonishing property of trained dense word embeddings is the spontaneous emergence of linear compositionality in geometric vector space. Semantic and grammatical relationships correspond to invariant directional translation vectors:
[
mathbf{v}_{ ext{King}} – mathbf{v}_{ ext{Man}} + mathbf{v}_{ ext{Woman}} approx mathbf{v}_{ ext{Queen}}
]
[
mathbf{v}_{ ext{Paris}} – mathbf{v}_{ ext{France}} + mathbf{v}_{ ext{Italy}} approx mathbf{v}_{ ext{Rome}}
]
This remarkable isomorphism proves that deep statistical co-occurrence models uncover the intrinsic topological structure of human conceptual representation.
