Safari Connectivity and Security Updates; potential causes, fixes, and more
Last week we resumed coverage of an issue where Safari can't connect to some sites on the first try, but (for most users) will eventually load the desired page after several tries.
There is a strong correlation, but probably not a causation, between the inability to load pages in Safari (and perform other network operations) and the release of certain Apple Security Updates.
The primary cause of this issue now appears to be an apparent flaw in the way that Mac OS X -- specifically the BIND networking component -- handle DNS requests. It appears that the Mac OS X DNS lookup mechanism (which queries a DNS server for the appropriate IP address of a given domain name, e.g. 64.236.16.84 for "cnn.com") times out too quickly. Simply put, Mac OS X queries the domain name, waits a brief time period for the response, and times out before the response happens.
This partially explains why manually entering your ISP's DNS servers (which can generally be obtained from your ISP's Web site or a phone call to their support line) in the Network pane of System Preferences works to resolve this issue. With manually entered DNS addresses, Mac OS X does not have to take the extra step of locating the servers and hence processes more DNS query requests on the first try.
In some cases, the DNS addresses must also be configured on a router if one is present.
MacFixIt reader Brent writes "I was able to resolve the problems by manually entering the DNS Servers for my ISP (Cox Kansas) in the Network Settings and on my router."
Reader Mike adds "We have gotten into the habit of entering the DNS server numbers for our ISP (Road Runner) in order to solve some connection problems. We set the DNS numbers in the (D-Link DI-604) broadband gateway router, and we also set the DNS numbers in all computers of our office LAN."
Finally, MacFixIt reader Set notes that you can sometimes find local DNS servers through a Web search "I did a search on the internet for DNS servers pertaining to my local area and service provider (Time Warner RR in Raleigh, NC), and found these 3:
- 24.25.4.107
- 24.25.4.108
- 24.25.4.109
"I added those to my Network Preferences, and the ones ending in 107 and 108 to my Airport Express DNS server list (because only 2 are allowed) using the Airport Admin Utility. In the last 6 hours, I have had only 1 error where the server couldn't be found (a second try found the site, however)."
Some users, like Doug Eldred, have different ISPs produce different results on the same Mac. This it seems that some ISPs may have slower DNS conversion mechanisms that others"
"One more observation on the Safari/DNS situation... I have Qwest/MSN DSL. For the past day or so, NO DNS queries worked at all. I tried repeatedly unplugging both the DSL modem and my AirPort Express Base Station, once for several hours, and rebooting the iMac, to no avail.
"One or more or my neighbors have unprotected 802.11* WAPs (one is "linksys", another is 'default'), so I 'borrowed' one long enough to convince myself that it wasn't the Internet as a whole, nor my computer or setup per se. Oddly enough, while using the 'borrowed' connection, I had few if any DNS failures on the first try which immediately worked on a second try, which I've been having on and off since one of the security updates was installed.
"[...]I was thinking of calling Qwest to see if they knew of any problems, when to my surprise I found an old document that gave the Qwest DNS server addresses back in January 2000. To my slight surprise (but not really "shock") they were still valid. Since specifying them, I've had no solid failures, few "first time fails, second time works" incidents, but I -do- still have an -occasional- first/second time incident, though much much less frequently."
A more complex, but more reliable fix MacFixIt reader Ken has discovered a clever workaround that involves modifying the operation of Mac OS X's "named" daemon -- the DNS server that is part of the BIND set of UNIX DNS utilities.
The theory behind why this fix works is as follows: root domain servers appear to have recently been given IPv6 capability, and are now returning AAAA records in response to name lookups.
Ken writes "The simple upshot is that for whatever reason, the first time named tries to go do a DNS query, it seems to decide to try sending to an IPV6 server address, which is pretty much guaranteed to fail for most users. Eventually this times out and it retries, but by that time Safari has usually given up on resolving the address and you get an error. The second time you try it, the correct address has already been cached by the system and everything works.
"I found that a very simple fix (if you don't mind editing OS config files) was to modify the /System/Library/StartupItems/BIND/BIND file to add the '-4' option to named, which forces it to only use IPV4. For example, the first part of the file normally looks like this:
StartService ()
{
if [ "${DNSSERVER:=-NO-}" = "-YES-" ]; then
ConsoleMessage "Starting named"
named
fi
}
I changed the above to:
StartService ()
{
if [ "${DNSSERVER:=-NO-}" = "-YES-" ]; then
ConsoleMessage "Starting named"
named -4
fi
}
"You should probably do the same thing for the RestartService section in the file. You'll either need to restart named by hand with the new option, or simply reboot your system to have the above take effect (probably the safest thing to do). I haven't yet tried digging into the BIND code deeply enough to determine why it seems to favor IPV6 address over and over again even though it never gets a valid response. It appears as though there is some code within BIND to sort the servers on response time, but IPV6 servers seem to always wind up at the front of the list."
Previously visited sites do not exhibit the problem Several readers have reported that one a site successfully (usually after multiple requests), it will continue to work on the first try in subsequent attempts. This occurs because Mac OS X is able to cache the correct DNS address for a given URL. However, once the cache no longer exists, the problem re-occurs.
MacFixIt reader Steve writes "Shutting down my cable modem/airport, and powering back up fixed the Safari 'two-try' syndrome. However, that was short-lived. It did fix it for a few hours, but then... back to the usual problem. Previously visited sites are fine, but newly-visited sites almost always exhibit the 'cannot find server' drop down, but connect always on the second try."
Why are these issues occurring after Security Updates? Again, the update most often implicated as a cause -- Security Update 2004-09-30 -- was a relatively minor update in terms of the number of files replaced; specifically, the Update only changed files relating to NetInfo Manager, File Sharing, QuickTime, postfix, and cupsd. None of these files should affect Safari connectivity.
Also, it should be noted that we've received a dozen or so reader reports (out of the hundreds regarding this issue) from users who have not applied any of the recent Apple Security Updates.
In a quick scan through changes included in the recent updates, only one -- Security Update 2004-09-07 -- has significant changes to TCP/IP operation.
This update fixes a vulnerability where maliciously crafted IP fragments can use too many system resources preventing normal network operation, i.e. the "Rose Attack."
According to Security Update 2004-09-07's documentation "The TCP/IP implementation has been modified to limit the resources consumed and prevents this denial of service attack."
Even this change does not seem to be directly related to the issue at hand, though Security Update 2004-09-07 did generate an increase in the number of connectivity problem reports.
One possible theory -- Security Update 2004-09-07 was revised from its initial release in mid-September. A mere two weeks later, the "most problematic" update, Security Update 2004-09-30, was released. Some have suggested that Security Update 2004-09-07 actually caused the problem, but users did not immediately experience it because of already cached DNS servers. A subsequent update -- Security Update 2004-09-30 for some, another revision or even a general system maintenance routine for others -- cleared the DNS cache and the problem began to manifest. Currently the theory is pure speculation, but it may help us track down the actual trigger for this issue.
Feedback? Late-breakers@macfixit.com.
Resources
home and occasionally at work. I did notice, however, that this problem also
occurs when I try to preview web pages I'm creating with Dreamweaver in
Safari. Its not going to an outside server, it won't load the local page.
I have SBC Yahoo! DSL, and other customers of this ISP having the same DNS issue might want to look at this from SBC/Y!'s help site:
http://help.sbcglobal.net/article.php?item=3510&os=mac103
(I have linked to the OS X 10.3 version, but there is a menu at the top where the OS choice can be changed.)
Apparently, SBC is dropping (or plans to drop, or may drop (?)) their old DNS servers that customers were previously given static IPs for. When I started to experience this bug, I noticed I was still using the old servers. Once I deleted the DNS server IP address(es) from my System Preferences > Network > Built-In Ethernet > TCP/IP tab, and rebooted (which may or may not be necessary, clicking "Apply Now" might be sufficient), the bug has mostly disappeared.
There is still a site or two that sometimes requires a reload (and, yes, I'd say the behavior is the same across Safari, Firefox, and Camino). But not nearly as many as before.
(the Bind deamon) does not actually run by default in OS X (unlike OS X
Server). If you do a "ps aux | grep named" in terminal, there is no named
process running. You can start named explicitly but it is not running
normally. So unless you opt to start running it, the edit will not make any
difference...
"StartUpItems" in the root system library - no such file. And none appears
with a "Find" command - visible or invisible. Any help on this?
Cathy
"StartupItems" - no capital "U".
Best idea is to type in the beginning of the path (ie "Sta") then hit 'Tab' key.
The system will try to fill in the rest.
But as I pointed out in my post - named neither runs normally in OSX (client)
nor is the builtin v. 9.2.2 capable of running with the -4 option...???
If your LAN is running DNS on an OSX server then you CAN update Bind to 9.3
and run it with the -4 option. This has been shown to work, see this thread in
Apple Server Forum:
http://discussions.info.apple.com/webx?
14@116.8tAlaG0JAv4.1@.689de754/0
OS X comes with Bind Version 9.2.2 for which, FAIK, the -4 option is not
valid. I'm not sure about v.9.2.3 but 9.3 can operate with the -4 option. I'm
therefore not sure what sort of system the poster of this suggestion has tried
it on then...
Specifically, trying Ken's suggested edit in a 10.3.5 environment elicits the
following:
Restarting named
usage: named [-c conffile] [-d debuglevel] [-f|-g] [-n number_of_cpus]
[-p port] [-s] [-t chrootdir] [-u username]
named: unknown option '-4'
Following recent posts in the Apple Server forum, I can confirm that installing
Bind Version 9.3 onto an OS X 10.3.5 Server running DNS services, and
running it with the -4 option DOES appear to work (for me) for all machines
on the network.
OS X Client does come with Bind 9.2.2 as part of the BSD core but this is not
actually activated on the client. One CAN activate it (using an additional
startup script) and therefore use yourself as your own DNS server, upgraded
to v9.3 running with -4 option, and presumably it would also work.
the security updates per se, but the REBOOT!? Most OS X users rarely reboot,
except to install patches that require it. Perhaps the caches get cleared and/
or a reconnection gets a new name server and the timeouts begin.
Just a guess. I had not had this problem until the most recent update/reboot,
but see it occasionally now.
Rebooted and cleared cathes and the problem still exists. Nice thought but no
solution.
my Mac's network config today and the problem has not re-occured after
several hours of surfing.
I agree. I added my DNS addresses to the Network Pane and to my Airport
base station. Not only am I not getting timeouts, but everything seems to be
loading MUCH faster.
have reported that on(c)e a site successfully (loads) (usually after multiple
requests), it will continue to work on the first try in subsequent attempts.
This occurs because Mac OS X is able to cache the correct DNS address for a
given URL. However, once the cache no longer exists, the problem re-occurs."
Wish it were so for me, BUT, this morning when I clicked on the "MacMinute"
link in my Safari Toolbar, it couldn't load the site. When I clicked on it the
second time it did. I visit MacMinute every day, sometimes more than once
and I've visited MacMinute a number of times since this problem cropped up
for me. I've NOT trashed the cache either.
last night), and I haven't yet tried entering my ISP's DNS server addresses in
my Network preferences or the BIND config edit. I've noticed for at least 24
hours, though, that my DSL connection has been going down and coming
back up spontaneously (and I no longer think it's just the kittens in the room
where the DSL jack is playing with the wire they severed and I repaired a few
weeks ago). Obviously, mine is not a situation where it'll be easy to pinpoint
the root cause, but perhaps there's a common DNS problem among more
than one ISP (mine is Earthlink / Covad DSL)?
there. I installed the <code>named -4</code> hacklet and my system is
behaving normally, i.e., no timeouts.
problem. At one point I did think it was fixed but after a time the problem
reoccurred. I don't think the problem ever actually went away, I just think I hit
some sites where it didn't happen for awhile. As of today though I'm still
experiencing the problem.
The one site I visit where the problem always occurs on the first try (95% of
the time) is the log in box at Ameritrade at https://
wwws.ameritrade.com apps/LogIn/. Try it
if you think you're fixed and let me know. Thanks.
that I am stupid. In order for Apple and OSX to surpass the nay sayers and
convert people from WINTEL, these kind of update problems must be tested
and retested before being released. For me, I have to keep two systems
running because I cannot trust that these updates will not cause my vital
programs to malfunction.
Sorry state for unix programmers.
needed to revise an eBay item for sale. Made the changes and clicked the
SAVE CHANGES button. Page refreshes but revision does not hold. 3X. Gave
up and went to IE which took the change with no problem on the first try. Not
sure if this is a new bug sunce the Security Update as I don't revise items
often.
- by Simon Hackett November 17, 2004 6:22 PM PST
- Gentlepersons,
- Like this Reply to this comment
-
(20 Comments)This thread was very useful for me - I found it via google, having run in to the
same problem following the process of updating my Mac (2x2Ghz G5) up to
10.3.6 and applying all the recent security patches. Consistent with other
peoples' experience, the problem described here then started to occur for
me, too.
I have had enough DNS experience to have decided to try to track the details
of problem down to prove the cause is 'as it seems', to prove that its
operating system and not browser specific, and I've done those things.
Deeper, that lead to me coming up with a simple workaround, and (for me at
least) this workaround solves the problem - with no system file editing
required.
So lets cut to the chase - how do you fix the problem?
Easy: list your ISP DNS server IP address *multiple times* (I listed mine three
times, for the heck of it).
Yes, I mean that you take whatever your ISP says is your primary DNS resolver
IP address, and just type it in three times in a row (in three successive rows)
in the 'DNS servers' box under TCP/IP in your 'Network' control panel.
That simply convinces the operating system to try three times (to the same
DNS resolver) to answer the question.
That , in effect, triples the effective timeout for the DNS queries.
For me, it has solved the problem entirely. No editing of startup files
required.
For those that care, I'll explain how I proved the issue is operating system
based, and how I proved that the above fixes it (below). Those that don't care
- just do the above and enjoy your restored surfing experience :)
(How *do* you report this sort of thing to Apple so that they actually take
note of it and fix it??)
<hr>
Ok, here are the gory details.
You can prove that the problem exists, and that its operating system (not
browser) specific by operning a window as root (e.g. using sudo tcsh) and
monitoring your own systems' DNS queries, like this:
tcpdump -s1500 -vv port 53
Now, just go surfing.
In particular, the trick is that to show the problem up, you need to surf to a
URL with two important characteristics:
1) Its not in your ISP's DNS cache already (so you can only do this once per
DNS name, per ISP-cache-timeout-period)
2) The name concerned has to take a while to resolve (the simplest way to do
that is to pick names served in another country to your own, preferably one
far away)
For me, in Australia, that's simpler than for folk in the USA - I just go for USA
names, or European ones. These commonly take several seconds to manage
initial resolution (after which they are cached).
If you are short of ideas on such names (and you are OUTSIDE of the USA),
pick any dictionary name and wrap it in 'www' and 'com' and it'll probably be
a valid DNS name :) Else, well, be creative. Treat it like a game. Look up your
own bookmark history, find obscure URLs and try them :)
To make the resulting tcpdump clearer, its also best if you type in
<tt>www.whatever.com</tt> with a period (.) after the dns name.
That stops your resolver from trying to tack your local/isp domain name on
the end and attempting that first, before falling back to exactly the name you
typed. It makes the resulting tcpdump output clearer, in effect.
So, try
<tt>www.whatever.com.</tt> where 'whatever' is something not already in
your ISP's cache (or your own).
Now, lets look at the resulting tcpdump output where you've managed to pick
a name that isn't already cached and which takes more than about 4 seconds
to initially resolve. Note that this example was on my system, using this same
domain name may NOT work for you. I reproduced this many times, just keep
trying until you strike a name not in your cache:
<tt>
11:47:16.376803 IP (tos 0x0, ttl 64, id 37576, offset 0, flags [none], length:
65) MYNAMEHERE.50373 > MYRESOLVERNAME.domain: [udp sum ok]
47858+ A? www.avisrewards.com. (37)<br>
11:47:17.082036 IP (tos 0x0, ttl 64, id 37582, offset 0, flags [none], length:
65) MYNAMEHERE.50373 > MYRESOLVERNAME.domain: [udp sum ok]
47858+ A? www.avisrewards.com. (37)<br>
11:47:17.787116 IP (tos 0x0, ttl 64, id 37587, offset 0, flags [none], length:
65) MYNAMEHERE.50373 > MYRESOLVERNAME.domain: [udp sum ok]
47858+ A? www.avisrewards.com. (37)<br>
11:47:18.492206 IP (tos 0x0, ttl 64, id 37593, offset 0, flags [none], length:
65) MYNAMEHERE.50373 > MYRESOLVERNAME.domain: [udp sum ok]
47858+ A? www.avisrewards.com. (37)<br>
<i>my browser returned an error at this moment, just before the answer
comes back (below):</i><br>
11:47:20.559098 IP (tos 0x0, ttl 62, id 4142, offset 0, flags [none], length:
129) MYRESOLVERNAME.domain > MYNAMEHERE.50373: [udp sum ok]
47858 q: A? www.avisrewards.com. 1/2/0 www.avisrewards.com. A
www.avisrewards.com ns: avisrewards.com. NS ns2.ipera.com.au.,
avisrewards.com. NS ns1.ipera.com.au. (101)<br>
</tt>
Take a look at the above. The system made four attempts to resolve the
name, over about two seconds. Then my browser said that it couldn't resolve
the name, and THEN the last line in the above trace came in - with the right
answer in it - but 1.5 seconds too late for the (now) impatient operating
system resolver library.
Next attempt works, because now your ISP's cache, and your own, have the
answer stored away, and it'll keep working for about a day or so (or longer, if
someone else with your ISP happens to hit the same name to refresh the
cache before you need it).
So there is the problem. Apple, for whatever reason, have configured their
resolver library in recent security updates and/or operating system updates,
with a shorter timeout, in particular the resolver tries four times over two
seconds and then gives up if no answer has come back. Way too short a
timeout for the real world. Especially for those of us who live outside of the
contiinental USA. Way too short. Oops, naughty Apple, no biscuit.
So, lets see what happens when you put my simple workaround in (list the
same DNS resolver IP address three times - or more if you're even more
paranoid). Now, lets try again (with another DNS name, since the above is
already in cache):
<tt>
12:04:46.342008 IP (tos 0x0, ttl 64, id 51574, offset 0, flags [none], length:
59) MYNAMEHERE.50670 > MYRESOLVERNAME.domain: [udp sum ok]
48170+ A? www.sloth.com. (31)<br>
12:04:47.047680 IP (tos 0x0, ttl 64, id 51579, offset 0, flags [none], length:
59) MYNAMEHERE.50671 > MYRESOLVERNAME.domain: [udp sum ok]
48170+ A? www.sloth.com. (31) 12:04:47.752787 IP (tos 0x0, ttl 64, id
51585, offset 0, flags [none], length: 59) MYNAMEHERE.50672 >
MYRESOLVERNAME.domain: [udp sum ok] 48170+ A? www.sloth.com.
(31)<br>
12:04:48.457893 IP (tos 0x0, ttl 64, id 51592, offset 0, flags [none], length:
59) MYNAMEHERE.50670 > MYRESOLVERNAME.domain: [udp sum ok]
48170+ A? www.sloth.com. (31)<br>
12:04:49.163082 IP (tos 0x0, ttl 64, id 51598, offset 0, flags [none], length:
59) MYNAMEHERE.50671 > MYRESOLVERNAME.domain: [udp sum ok]
48170+ A? www.sloth.com. (31)<br>
12:04:49.868147 IP (tos 0x0, ttl 64, id 51603, offset 0, flags [none], length:
59) MYNAMEHERE.50672 > MYRESOLVERNAME.domain: [udp sum ok]
48170+ A? www.sloth.com. (31)<br>
12:04:50.573202 IP (tos 0x0, ttl 64, id 51609, offset 0, flags [none], length:
59) MYNAMEHERE.50670 > MYRESOLVERNAME.domain: [udp sum ok]
48170+ A? www.sloth.com. (31)<br>
12:04:51.064328 IP (tos 0x0, ttl 62, id 19305, offset 0, flags [none], length:
140) MYRESOLVERNAME.domain > MYNAMEHERE.50670: [udp sum ok]
48170 q: A? www.sloth.com. 2/2/0 www.sloth.com. CNAME sloth.com.,
sloth.com. A pegasus.sslcatacomb.net ns: sloth.com. NS
ns1.sslcatacomb.net., sloth.com. NS ns2.sslcatacomb.net. (112)<br>
</tt>
See what happened? This domain name took 7 attempts (3.5 seconds) to
return a result.
It previously would have given up (and returned a browser error) after the first
four attempts. Now, with the resolver IP listed three times, it worked.
(and the browser page loaded perfectly first time).
<br>
Notes:
1) This is NOT a Safari a bug as such. Its the underlying resolver library.
I have proven this by duplicating the bug, reliably, in MS-IE as well as Safari.
Same network-level activity occurs.
2) (the subtle one) you can't reproduce this failure mode using 'nslookup'!
It seems (perhaps understandably) to run with a non-default, higher, query
timeout. So it generates the false impression that the DNS resolution isn't the
issue, when it really is (with real GUI apps using default resolver
configuration, not nslookup).
<hr>
So - there you go!
Cheers,
Simon Hackett