works
|
about
Mechanistic Interpretability
on (multi-task) Irreducible In
teger Identifiers
Noah Syrkis
January 8, 2025
1 |
Mechanistic Interpretability
2 |
Modular Arithmetic
3 |
Grokking on
𝒯
miiii
4 |
Embeddings
5 |
Neurons
6 |
The
𝜔
-Spike
“This disgusting pile of matrices is actually just an astoundingly poorly written, elegant and consice
algorithm” — Neel Nanda
¹
¹
Not verbatim, but the gist of it
1 |
Mechanistic Interpretability
▶
Sub-symbolic nature of deep learning obscures model mechanisms
▶
No obvious mapping from the weights of a trained model to math notation
▶
MI is about reverse engineering these models, and looking closely at them
▶
Many low hanging fruits / practical botany phase of the science
▶
How does a given model work? How can we train it faster? Is it safe?
1 of
19
1.1 |
Grokking
▶
Grokking
[1]
is “sudden generalization”
▶
MI (often) needs a mechanism
▶
Grokking is thus convenient for MI
Figure 1: Grokking
2 of
19
2 |
Modular Arithmetic
▶
“Seminal” MI paper by
Nanda et al. (2023)
focuses on modular addition (
𝒯
nanda
)
▶
Their final setup trains on
𝑝
=
1
1
3
▶
They train a one-layer transformer
▶
We call their task
𝒯
nanda
▶
And ours we call
𝒯
miiii
𝒯
nanda
=
(
𝑥
0
+
𝑥
1
)
mod
𝑝
,
∀
𝑥
0
,
𝑥
1
(1.1)
𝒯
miiii
=
(
𝑥
0
𝑝
0
+
𝑥
1
𝑝
1
)
mod
𝑞
,
∀
𝑞
<
𝑝
(1.2)
3 of
19
2 |
Modular Arithmetic
▶
𝒯
miiii
is non-commutative …
▶
… and multi-task:
𝑞
ranges from 2 to 109
¹
▶
𝒯
nanda
use a single layer transformer
▶
Note that these tasks are synthetic and trivial to solve with conventional programming
▶
They are used in the MI literature to turn black boxes opaque
¹
Largest prime less than
𝑝
=
1
1
3
4 of
19
Figure 2:
ℕ
<
𝑝
2
multiples of 13 or 27 (left) 11 (mid.) or primes (right)
3 |
Grokking on
𝒯
miiii
▶
For two-token samples, plot them varying one
on each axis (
Figure 3
)
▶
When a matrix is periodic use Fourier
▶
Singular value decomposition
Figure 3: Top singular vectors of
𝐔
𝑊
𝐸
𝒯
nanda
(top),
varying
𝑥
0
and
𝑥
1
in sample (left) and freq. (right)
space in
𝑊
out
𝒯
miiii
5 of
19
3 |
Grokking on
𝒯
miiii
▶
The model groks on
𝒯
miiii
(
Figure 4
)
▶
Needed GrokFast
[3]
on compute budget
▶
Final hyperparams are seen in
Table 1
rate
𝜆
wd
𝑑
lr
heads
1
1
0
1
2
1
3
256
3
1
0
4
4
Table 1: Hyperparams for
𝒯
miiii
Figure 4: Training (top) and validation (bottom)
accuracy during training on
𝒯
miiii
6 of
19
4 |
Embeddings
How the embedding layer deals with the difference between
𝒯
nanda
and
𝒯
miiii
7 of
19
4.1 |
Correcting for non-commutativity
▶
The position embs. of
Figure 5
reflects that
𝒯
nanda
is commutative and
𝒯
miiii
is not
▶
Maybe: this corrects non-comm. of
𝒯
miiii
?
▶
Corr. is
0
.
9
5
for
𝒯
nanda
and
−
0
.
6
4
for
𝒯
miiii
Figure 5: Positional embeddings for
𝒯
nanda
(top)
and
𝒯
miiii
(bottom).
8 of
19
4.2 |
Correcting for multi-tasking
▶
For
𝒯
nanda
token embs. are essentially linear
combinations of 5 frequencies (
𝜔
)
▶
For
𝒯
miiii
more frequencies are in play
▶
Each
𝒯
miiii
subtask targets unique prime
▶
Possibility: One basis per prime task
Figure 6:
𝒯
nanda
(top) and
𝒯
miiii
(bottom) token
embeddings in Fourier basis
9 of
19
4.3 |
Sanity-check and task-mask
▶
Masking
𝑞
∈
{
2
,
3
,
5
,
7
}
yields we see a slight
decrease in token emb. freqs.
▶
Sanity check:
𝒯
baseline
has no periodicity
▶
The tok. embs. encode a basis per subtask?
Figure 7:
𝒯
baseline
(top),
𝒯
miiii
(middle) and
𝒯
masked
(bottom) token embeddings in Fourier basis
10 of
19
5 |
Neurons
▶
Inspite of the dense Fourier basis of
𝑊
𝐸
𝒯
miiii
the
periodicity is clear
figs/neurs_113_miiii.svg
), caption: [Activations of first three neurons for
𝒯
nanda
(top) and
𝒯
miiii
(bottom)],
)
11 of
19
5 |
Neurons
▶
(Probably redundant) sanity check:
Figure 9
confirms neurons are periodic
▶
See some freqs.
𝜔
rise into significance
▶
Lets log
|
𝜔
>
𝜇
𝜔
+
2
𝜎
𝜔
|
while training
Figure 9: FFT of Activations of first three neurons
for
𝒯
nanda
(top) and
𝒯
miiii
(bottom)
12 of
19
Figure 10: Neurons as archive for
𝒯
basline
Figure 11: Neurons as algorithm
𝒯
miiii
Figure 12: Number of neurons with frequency
𝜔
above the theshold
𝜇
𝜔
+
2
𝜎
𝜔
6 |
The
𝜔
-Spike
▶
Neurs. periodic on solving
𝑞
∈
{
2
,
3
,
5
,
7
}
▶
When we generalize to the reamining tasks,
many frequencies activate (64-sample)
▶
Those
𝜔
’s are not useful for memory and not
useful after generalization
time
256
1024
4096
16384
65536
|
𝝎
|
0
0
1
0
1
8
10
Table 2: active
𝜔
’s through training
Figure 13:
Figure 12
(top) and validation accuracy
from
Figure 4
(bottom)
13 of
19
6 |
The
𝜔
-Spike
▶
GrokFast
[3]
shows time gradient sequences is (arguably) a stocastical signal with:
▶
A fast varying overfitting component
▶
A slow varying generealizing component
▶
My work confirms this to be true for
𝒯
miiii
…
▶
… and observes a strucutre that seems to fit
neither
of the two
14 of
19
6 |
The
𝜔
-Spike
▶
Future work:
▶
Modify GrokFast to assume a third stochastic component
▶
Relate to signal processing literature
▶
Can more depth make tok-embedding sparse?
15 of
19
TAK
References
[1]
A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra, “Grokking: Generalization Beyond
Overfitting on Small Algorithmic Datasets,” no. arXiv:2201.02177. arXiv, Jan. 2022. doi:
10.48550/
arXiv.2201.02177
.
[2]
N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt, “Progress Measures for Grokking via
Mechanistic Interpretability,” no. arXiv:2301.05217. arXiv, Oct. 2023.
[3]
J. Lee, B. G. Kang, K. Kim, and K. M. Lee, “Grokfast: Accelerated Grokking by Amplifying Slow
Gradients,” no. arXiv:2405.20233. Jun. 2024.
16 of
19
A |
Stochastic Signal Processing
We denote the weights of a model as
𝜃
. The gradient of
𝜃
with respect to our loss function at time
𝑡
we denote
𝑔
(
𝑡
)
. As we train the model,
𝑔
(
𝑡
)
varies, going up and down. This can be thought of as
a stocastic signal. We can represent this signal with a Fourier basis. GrokFast posits that the slow
varying frequencies contribute to grokking. Higer frequencies are then muted, and grokking is indeed
accelerated.
17 of
19
B |
Discrete Fourier Transform
Function can be expressed as a linear combination of cosine and sine waves. A similar thing can be
done for data / vectors.
18 of
19
C |
Singular Value Decomposition
An
𝑛
×
𝑚
matrix
𝑀
can be represented as a
𝑈
Σ
𝑉
∗
, where
𝑈
is an
𝑚
×
𝑚
complex unitary matrix,
Σ
a rectangular
𝑚
×
𝑛
diagonal matrix (padded with zeros), and
𝑉
an
𝑛
×
𝑛
complex unitary matrix.
Multiplying by
𝑀
can thus be viewed as first rotating in the
𝑚
-space with
𝑈
, then scaling by
Σ
and
then rotating by
𝑉
in the
𝑛
-space.
19 of
19
Mechanistic Interpretability
on (multi-task) Irreducible In
teger Identifiers
Noah Syrkis
January 8, 2025
1 |
Mechanistic Interpretability
2 |
Modular Arithmetic
3 |
Grokking on
𝒯
miiii
4 |
Embeddings
5 |
Neurons
6 |
The
𝜔
-Spike
“This disgusting pile of matrices is actually just an astoundingly poorly written, elegant and consice
algorithm” — Neel Nanda
¹
¹
Not verbatim, but the gist of it
1 |
Mechanistic Interpretability
▶
Sub-symbolic nature of deep learning obscures model mechanisms
1 of
19
1 |
Mechanistic Interpretability
▶
Sub-symbolic nature of deep learning obscures model mechanisms
▶
No obvious mapping from the weights of a trained model to math notation
1 of
19
1 |
Mechanistic Interpretability
▶
Sub-symbolic nature of deep learning obscures model mechanisms
▶
No obvious mapping from the weights of a trained model to math notation
▶
MI is about reverse engineering these models, and looking closely at them
1 of
19
1 |
Mechanistic Interpretability
▶
Sub-symbolic nature of deep learning obscures model mechanisms
▶
No obvious mapping from the weights of a trained model to math notation
▶
MI is about reverse engineering these models, and looking closely at them
▶
Many low hanging fruits / practical botany phase of the science
1 of
19
1 |
Mechanistic Interpretability
▶
Sub-symbolic nature of deep learning obscures model mechanisms
▶
No obvious mapping from the weights of a trained model to math notation
▶
MI is about reverse engineering these models, and looking closely at them
▶
Many low hanging fruits / practical botany phase of the science
▶
How does a given model work? How can we train it faster? Is it safe?
1 of
19
1.1 |
Grokking
▶
Grokking
[1]
is “sudden generalization”
Figure 1: Grokking
2 of
19
1.1 |
Grokking
▶
Grokking
[1]
is “sudden generalization”
▶
MI (often) needs a mechanism
Figure 1: Grokking
2 of
19
1.1 |
Grokking
▶
Grokking
[1]
is “sudden generalization”
▶
MI (often) needs a mechanism
▶
Grokking is thus convenient for MI
Figure 1: Grokking
2 of
19
2 |
Modular Arithmetic
▶
“Seminal” MI paper by
Nanda et al. (2023)
focuses on modular addition (
𝒯
nanda
)
▶
Their final setup trains on
𝑝
=
1
1
3
▶
They train a one-layer transformer
▶
We call their task
𝒯
nanda
𝒯
nanda
=
(
𝑥
0
+
𝑥
1
)
mod
𝑝
,
∀
𝑥
0
,
𝑥
1
(1.1)
𝒯
miiii
=
(
𝑥
0
𝑝
0
+
𝑥
1
𝑝
1
)
mod
𝑞
,
∀
𝑞
<
𝑝
(1.2)
3 of
19
2 |
Modular Arithmetic
▶
“Seminal” MI paper by
Nanda et al. (2023)
focuses on modular addition (
𝒯
nanda
)
▶
Their final setup trains on
𝑝
=
1
1
3
▶
They train a one-layer transformer
▶
We call their task
𝒯
nanda
▶
And ours we call
𝒯
miiii
𝒯
nanda
=
(
𝑥
0
+
𝑥
1
)
mod
𝑝
,
∀
𝑥
0
,
𝑥
1
(1.1)
𝒯
miiii
=
(
𝑥
0
𝑝
0
+
𝑥
1
𝑝
1
)
mod
𝑞
,
∀
𝑞
<
𝑝
(1.2)
3 of
19
2 |
Modular Arithmetic
▶
𝒯
miiii
is non-commutative …
▶
… and multi-task:
𝑞
ranges from 2 to 109
¹
▶
𝒯
nanda
use a single layer transformer
▶
Note that these tasks are synthetic and trivial to solve with conventional programming
▶
They are used in the MI literature to turn black boxes opaque
¹
Largest prime less than
𝑝
=
1
1
3
4 of
19
Figure 2:
ℕ
<
𝑝
2
multiples of 13 or 27 (left) 11 (mid.) or primes (right)
3 |
Grokking on
𝒯
miiii
▶
For two-token samples, plot them varying one
on each axis (
Figure 3
)
▶
When a matrix is periodic use Fourier
▶
Singular value decomposition
Figure 3: Top singular vectors of
𝐔
𝑊
𝐸
𝒯
nanda
(top),
varying
𝑥
0
and
𝑥
1
in sample (left) and freq. (right)
space in
𝑊
out
𝒯
miiii
5 of
19
3 |
Grokking on
𝒯
miiii
▶
The model groks on
𝒯
miiii
(
Figure 4
)
▶
Needed GrokFast
[3]
on compute budget
▶
Final hyperparams are seen in
Table 1
rate
𝜆
wd
𝑑
lr
heads
1
1
0
1
2
1
3
256
3
1
0
4
4
Table 1: Hyperparams for
𝒯
miiii
Figure 4: Training (top) and validation (bottom)
accuracy during training on
𝒯
miiii
6 of
19
4 |
Embeddings
How the embedding layer deals with the difference between
𝒯
nanda
and
𝒯
miiii
7 of
19
4.1 |
Correcting for non-commutativity
▶
The position embs. of
Figure 5
reflects that
𝒯
nanda
is commutative and
𝒯
miiii
is not
Figure 5: Positional embeddings for
𝒯
nanda
(top)
and
𝒯
miiii
(bottom).
8 of
19
4.1 |
Correcting for non-commutativity
▶
The position embs. of
Figure 5
reflects that
𝒯
nanda
is commutative and
𝒯
miiii
is not
▶
Maybe: this corrects non-comm. of
𝒯
miiii
?
▶
Corr. is
0
.
9
5
for
𝒯
nanda
and
−
0
.
6
4
for
𝒯
miiii
Figure 5: Positional embeddings for
𝒯
nanda
(top)
and
𝒯
miiii
(bottom).
8 of
19
4.2 |
Correcting for multi-tasking
▶
For
𝒯
nanda
token embs. are essentially linear
combinations of 5 frequencies (
𝜔
)
▶
For
𝒯
miiii
more frequencies are in play
▶
Each
𝒯
miiii
subtask targets unique prime
▶
Possibility: One basis per prime task
Figure 6:
𝒯
nanda
(top) and
𝒯
miiii
(bottom) token
embeddings in Fourier basis
9 of
19
4.3 |
Sanity-check and task-mask
▶
Masking
𝑞
∈
{
2
,
3
,
5
,
7
}
yields we see a slight
decrease in token emb. freqs.
▶
Sanity check:
𝒯
baseline
has no periodicity
▶
The tok. embs. encode a basis per subtask?
Figure 7:
𝒯
baseline
(top),
𝒯
miiii
(middle) and
𝒯
masked
(bottom) token embeddings in Fourier basis
10 of
19
5 |
Neurons
▶
Inspite of the dense Fourier basis of
𝑊
𝐸
𝒯
miiii
the
periodicity is clear
figs/neurs_113_miiii.svg
), caption: [Activations of first three neurons for
𝒯
nanda
(top) and
𝒯
miiii
(bottom)],
)
11 of
19
5 |
Neurons
▶
(Probably redundant) sanity check:
Figure 9
confirms neurons are periodic
▶
See some freqs.
𝜔
rise into significance
▶
Lets log
|
𝜔
>
𝜇
𝜔
+
2
𝜎
𝜔
|
while training
Figure 9: FFT of Activations of first three neurons
for
𝒯
nanda
(top) and
𝒯
miiii
(bottom)
12 of
19
Figure 10: Neurons as archive for
𝒯
basline
Figure 11: Neurons as algorithm
𝒯
miiii
Figure 12: Number of neurons with frequency
𝜔
above the theshold
𝜇
𝜔
+
2
𝜎
𝜔
6 |
The
𝜔
-Spike
▶
Neurs. periodic on solving
𝑞
∈
{
2
,
3
,
5
,
7
}
▶
When we generalize to the reamining tasks,
many frequencies activate (64-sample)
▶
Those
𝜔
’s are not useful for memory and not
useful after generalization
time
256
1024
4096
16384
65536
|
𝝎
|
0
0
1
0
1
8
10
Table 2: active
𝜔
’s through training
Figure 13:
Figure 12
(top) and validation accuracy
from
Figure 4
(bottom)
13 of
19
6 |
The
𝜔
-Spike
▶
GrokFast
[3]
shows time gradient sequences is (arguably) a stocastical signal with:
▶
A fast varying overfitting component
▶
A slow varying generealizing component
▶
My work confirms this to be true for
𝒯
miiii
…
▶
… and observes a strucutre that seems to fit
neither
of the two
14 of
19
6 |
The
𝜔
-Spike
▶
Future work:
▶
Modify GrokFast to assume a third stochastic component
▶
Relate to signal processing literature
▶
Can more depth make tok-embedding sparse?
15 of
19
TAK
References
[1]
A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra, “Grokking: Generalization Beyond
Overfitting on Small Algorithmic Datasets,” no. arXiv:2201.02177. arXiv, Jan. 2022. doi:
10.48550/
arXiv.2201.02177
.
[2]
N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt, “Progress Measures for Grokking via
Mechanistic Interpretability,” no. arXiv:2301.05217. arXiv, Oct. 2023.
[3]
J. Lee, B. G. Kang, K. Kim, and K. M. Lee, “Grokfast: Accelerated Grokking by Amplifying Slow
Gradients,” no. arXiv:2405.20233. Jun. 2024.
16 of
19
A |
Stochastic Signal Processing
We denote the weights of a model as
𝜃
. The gradient of
𝜃
with respect to our loss function at time
𝑡
we denote
𝑔
(
𝑡
)
. As we train the model,
𝑔
(
𝑡
)
varies, going up and down. This can be thought of as
a stocastic signal. We can represent this signal with a Fourier basis. GrokFast posits that the slow
varying frequencies contribute to grokking. Higer frequencies are then muted, and grokking is indeed
accelerated.
17 of
19
B |
Discrete Fourier Transform
Function can be expressed as a linear combination of cosine and sine waves. A similar thing can be
done for data / vectors.
18 of
19
C |
Singular Value Decomposition
An
𝑛
×
𝑚
matrix
𝑀
can be represented as a
𝑈
Σ
𝑉
∗
, where
𝑈
is an
𝑚
×
𝑚
complex unitary matrix,
Σ
a rectangular
𝑚
×
𝑛
diagonal matrix (padded with zeros), and
𝑉
an
𝑛
×
𝑛
complex unitary matrix.
Multiplying by
𝑀
can thus be viewed as first rotating in the
𝑚
-space with
𝑈
, then scaling by
Σ
and
then rotating by
𝑉
in the
𝑛
-space.
19 of
19