الانتباه والأقنعة والموضع

from math import exp, sqrt
query = [1.0, 0.0]
keys = [[1.0, 0.0], [0.0, 1.0]]
values = [2.0, 8.0]
scores = [sum(a*b for a,b in zip(query,k))/sqrt(2) for k in keys]
m = max(scores)
weights = [exp(s-m)/sum(exp(t-m) for t in scores) for s in scores]
print([round(w,6) for w in weights])
print(f"context={sum(w*v for w,v in zip(weights,values)):.6f}")

[0.669762, 0.330238]
context=3.981431

ينتج استعلام واحد أوزاناً مطبعة لمفتاحين وسياقاً عددياً، دون تعلم أو تجزئة نص.

from math import exp
scores = [1.0, 2.0, 100.0]
allowed = [True, True, False]
maximum = max(s for s, ok in zip(scores, allowed) if ok)
unnormalized = [exp(s-maximum) if ok else 0.0 for s,ok in zip(scores,allowed)]
weights = [v/sum(unnormalized) for v in unnormalized]
print([round(w,6) for w in weights])
print("masked weight", weights[2])

[0.268941, 0.731059, 0.0]
masked weight 0.0

تستبعد الدرجة العالية المحجوبة قبل التطبيع ويصبح وزنها صفراً تماماً في المثال.