Generally when errors occur during authentication for network services such as file sharing or screen sharing in OS X the connection will either time out or immediately give a warning, but sometimes connections may hang when authenticating. When this happens the connection will not be established and the authentication window will remain on-screen as a floating window that will block other windows and show a continuous revolving activity indicator next to a "Connecting..." status message.
While the window usually has options for canceling the connection, the hang prevents these from being active and results in the only way to manage the problem being either to drag the window to the side of the screen or to log out and log back in. During the hang the user will not be able to establish other connections to network services.
Because this window is invoked by choosing sharing options in the Finder, people experiencing this issue may initially suspect the Finder and try force-quitting it to resolve the issue. Doing so will only close and re-open the Finder windows but the authentication window will remain.
Network authentication for GUI-related services such as file sharing and screen sharing is not done through the Finder, and instead is managed through the network authentication agent process. This agent gathers credentials for a service, securely passes them to remote systems, and then quits once the connection betwen the server and a client program such as "Screen Sharing" is established. The process is then ready to authenticate additional connections.
When authentication hangs it is usually this process that is not working properly, instead of the Finder or other service-specific programs such as "Screen Sharing," and as a result the agent will need to be forced to quit to fix the problem.
To do this, launch Activity Monitor and locate the process called "NetAuthAgent," which will likely be using a minimal amount of CPU and will be one of the current users's processes (it will have the current username listed next to it). The process may or may not be colored red, but regardless of its status in Activity Monitor, select it and click the "Quit Process" button at the top of the Activity Monitor window.
When you do this the hung authentication window will disappear and you should then be able to establish new connections and try the connection again.