Being an identical twin has its advantages. You always have someone who understands you and always have someone to blame.
It also creates some pretty confusing moments.
One winter, I was on a ski trip. As I got off the chairlift, a guy
skied straight over to me with the biggest smile on his face.
"Hey! How've you been?"
Now, I had absolutely no idea who this guy was.
But I didn't want to be rude, so I smiled back and started doing
what any normal person would do...
I began fishing for clues.
"Doing great! So... how do we know each other?"
The poor guy looked at me like I had lost my memory.
Finally, after a minute or two of awkward conversation, he asked,
"Who are you?"
"Ray and you know my brother Phil"
He burst out laughing.
"Well, that explains everything!"
Turns out he'd known my identical twin for years. To him, we were
the same person. To me, I had spent the last two minutes wondering if
early-onset memory loss had just kicked in.
Once we figured out who was who, everything made sense.
Years later, I ran into the networking version of that exact same
situation.
Only this time, the twins were MAC addresses.
I was troubleshooting a strange VoIP problem in the administration
office of a college. The complaints were frustratingly inconsistent.
"Sometimes my phone works."
"Sometimes it hangs up."
"Sometimes I can't even get a dial tone."
Nothing about it made sense.
We started capturing packets and waited.
Sometimes troubleshooting isn't glamorous. It's a lot like
fishing. You stare at the screen, waiting for something interesting to swim by.
Then it happened.
One lady answered her phone and started talking.
A few seconds later, another lady picked up her phone.
Instant chaos.
The first call was disconnected.
The second phone never even received a dial tone.
And sitting quietly in the packet trace was the smoking gun.
Following the Packet Trace
Following the packet trace for Phone 1, everything looked
normal—until it wasn't.
The trace suddenly filled with ICMP messages:
- Destination Unreachable
- Host Unreachable
- Network Unreachable
The Smoking Gun
Phone 1 was in an active conversation, sending and receiving RTP
voice packets.
Then Phone 2 went off-hook and sent a request to the call
controller.
The controller attempted to send SIP signaling (INVITE and NOTIFY)
to Phone 2.
Instead, Phone 1 began receiving those SIP messages.
Phone 1 was now confused. Why was it receiving signaling intended
for another phone?
Phone 2 was equally confused because it never received the
responses it expected.
The result was a stream of ICMP "Destination
Unreachable," "Host Unreachable," and "Network
Unreachable" messages as the network attempted to deliver packets to two
devices claiming the same identity.
At first, we thought one of the phones was defective.
We opened the configuration menu on both phones and started
comparing their settings.
That's when we saw it.
Both phones had exactly the same MAC address.
We replaced Phone 2.
The problem disappeared immediately.
This is the story of one of the most unusual network problems I've
ever solved—and why, when packets are allowed to tell their story, the truth
usually isn't far behind.
At Porifera.io, we help software teams become heroes. Our platform continuously watches over your applications, collecting billions of live data points to show you exactly what's happening, why it's happening, and how to fix it.
