Verizon FIOS Doesn’t Talk to Verizon FIOS?
I have a bit more information about the connection difficulties I’ve been having with my ISP, Verizon FIOS. I have a residential account in Palmetto, FL with Verizon FIOS. Mostly, it works fine. I can get to a host of web sites without difficulty, and the transfer speeds are great.
I do remote system administration on two servers in the Dallas-Fort Worth Metroplex. Those servers get their connection via a Verizon FIOS Business plan link. (Yes, Verizon, the servers are on an account where serving is usual and expected.) One server provides my regular email, the other serves a whole bunch of web sites via virtual hosting. And things there are mostly working, where the outside world can merrily get pages served on demand.
But…
As of sometime early last Tuesday morning, January 10th, Verizon FIOS stopped reliably talking to Verizon FIOS. I can tell the approximate time of the outage as the last email message my computer here picked up from the server there was at 1:09 AM CT. The problem is very likely to have manifested within a very few minutes after that. And the problem’s characteristics are just plain weird. One expects most ‘problems’ with connections to be user error. Certainly that’s the primary basis of Verizon FIOS’s residential account tech support, who are ready to quit if the problem isn’t solved by having the user clear their browser cache or resetting the router. This problem, though, is more complex and is not localized to my particular account. First, not all connectivity is gone, just *most* connectivity. I can use SSH to log in remotely and use commands that return small amounts of information. Once I try a command that would return a page or more of text, the connection drops with a ‘Broken pipe’ message. There’s a web page that is static and is only a few hundred characters in size that I can successfully retrieve. But none of the web sites that rely on web applications (Drupal, WordPress, and IkonBoard) do anything but spin forever while the browser displays ‘Waiting on …’.
So let me jot down some things I’ve learned about this so far.
* It isn’t a DNS issue, as ‘nslookup’ finds any of the domain names and returns the correct IP address quite rapidly.
* It isn’t a single port failure. Ports 22, 25, 80, and 587 are, at a minimum, included in the affected list.
* It isn’t a complete break, as connections on the scale of a single packet of data at a time work.
* Using traceroute for other websites shows three hops taken within the Verizon routing center in Tampa. Traceroute for the affected servers shows two hops taken similarly, but the third times out.
* My parents live in Lakeland, Florida, a goodly distance away from where I live, and have Verizon FIOS as their ISP. I visited there this past weekend and asked my dad if he had been able to check this blog recently. He said no, not for about the past week. I tried traceroute from their connection, and it behaved the same way as from my home connection. The problem is not localized, it affects other Verizon FIOS customers.
* I’ve heard from Texas where another Verizon FIOS user of the email system cannot connect to the email server. I don’t have a traceroute result from them to compare.
I have two open tickets on this problem with Verizon, FLCP08NT6J and FLDQ090SXY. There are some other people who have posted to the web saying that they are having network difficulties with Verizon FIOS in the same time frame, but I haven’t seen a report that exactly matches what I am seeing. I’m writing this post by the expedient of using a proxy for my browser, which is a nuisance. (While it is on, my Google search results tend to come back in German, which I can’t read.) It’s a bit of a Catch-22, since I’d like to get feedback from Verizon FIOS users, but if the problem is of the nationwide scale that I expect it is, this post will be unaccessible to them from that account. On the other hand, if it is accessible via Verizon FIOS elsewhere, that would be useful information to have. If you are a Verizon FIOS user, I would appreciate it if you could run traceroute from the Verizon account to baywing.net and copy the results into a comment here. I’ll copy my traceroute results into a comment here shortly.
How to invoke traceroute:
Under Windows, open a command prompt. In the command prompt, type in the following:
tracert baywing.net > tr_baywing.txt
It will take a few minutes to complete if you also have the problem I’m having. The result ill be in a text file, ‘tr_baywing.txt’, in that directory. Copy and paste the text in a comment here if you aren’t seeing the problem, or contact me if you are having the problem.
On Mac or FreeBSD, open a terminal window. At the command prompt, type in:
traceroute baywing.net > tr_baywing.net
On Ubuntu Linux, open a terminal window. At the command prompt, type:
tracepath baywing.net > tr_baywing.net
Here’s my email, if you can’t leave a comment here (remove spaces and convert to symbols as indicated): w e l s b e r r at b a y w i n g dot n e t
Tracing route to baywing.net [71.123.242.56]
over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms myrouter.home [192.168.1.1]
2 6 ms 6 ms 6 ms L100.TAMPFL-VFTTP-138.verizon-gni.net [96.243.148.1]
3 10 ms 9 ms 8 ms G0-9-2-6.TAMPFL-LCR-22.verizon-gni.net [130.81.140.68]
4 * * * Request timed out.
5 40 ms 39 ms 38 ms 130.81.23.204
6 42 ms 43 ms 43 ms P13-0.DLLSTX-LCR-06.verizon-gni.net [130.81.199.45]
7 42 ms 42 ms 41 ms P14-0-0.DLLSTX-LCR-08.verizon-gni.net [130.81.27.43]
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace complete.
Here’s a little something from just west of Boston (I have an extra router behind my Verizon router to operate a VPN).
Tracing route to baywing.net [71.123.242.56]
over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 192.168.13.1
2 2 ms 2 ms 2 ms 192.168.1.1
3 26 ms 3 ms 3 ms 10.9.44.4
4 4 ms 3 ms 3 ms P9-0.BSTNMA-LCR-02.verizon-gni.net [130.81.37.250]
5 4 ms 4 ms 4 ms so-7-0-0-0.BOS-BB-RTR2.verizon-gni.net [130.81.29.166]
6 56 ms 55 ms 55 ms P14-0-0.DLLSTX-LCR-08.verizon-gni.net [130.81.27.43]
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace complete.
Have you found a solution to this issue? i believe it is Verizon residential which is the source of the problem. On my side its Verizon residential fios to ATT business server location. Myself and other employees with Verizon home fios service are able to connect to server remotely via VPN, but MS exchange will not communicate with remote computer. I am confident that it is a Verizon issue because all other staff remote computers via other ISP have no connectivity issues.
Tracing route to baywing.net [71.123.242.56]
over a maximum of 30 hops:
1 24 ms 1 ms 1 ms Wireless_Broadband_Router.home [192.168.1.1]
2 6 ms 4 ms 5 ms L100.DLLSTX-VFTTP-48.verizon-gni.net [71.123.176.1]
3 5 ms 5 ms 6 ms G3-3-748.DLLSTX-LCR-07.verizon-gni.net [130.81.107.16]
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
11
Try setting your MTU for packets to a value of 1492 or lower. I thought I had that set on the servers. I’ll have to check that.
The somewhat longer explanation is that Verizon FIOS network links don’t all use the industry-standard Maximum Transmission Unit (MTU) setting of 1500. They sometimes use a shorter MTU of 1492. This appeared exactly once in any official part of the Verizon FIOS site, on a page describing their Speed Optimizer tool, where it says:
Apparently, none of the Verizon tech support people knew a thing about this. I only found out because a friend happens to be a network guru and he looked over packet-capture logs as a favor. He suggested looking at the MTU settings, and after that I was able to find the reference on the Verizon site. I’ve requested the highest-ranking tech person I talked to at Verizon to add the MTU information to the FIOS documentation more prominently and to have it included in FIOS tech support scripts. I can’t tell that any of that has happened yet.
Not all FIOS pathways enforce the restricted MTU settings, which explains the stochastic nature of the problem. So far as I know, the Actiontec routers provided by FIOS don’t have the MTU=1492 restriction in them. I was able to change MTU on my clients and restore access, then followed it up with changing MTU on the servers. I suspect that it may be the case that some FIOS pathways restrict the MTU to a smaller size than 1492, which might be the case for your issue. If that’s so, please comment again with the MTU size on your client machine that gets you access again. I’ll switch the servers to use that.