Hope you all are doing quite well. Today I want to share with you one case study that is still haunting me. Day in and day out I tried to solve a very unique Incident raised by one of our customer.

In the morning I received one call that the users from one of our several locations are not able to browse using proxy server all of a sudden. Please note that we use Microsoft ISA for proxying.

As rest of the users from other locations were successfully browsing through same proxy server, so I thought there might be some problem relating to either Network connectivity or desktop firewall policy.

Accordingly I contacted network team who confirmed network path is okey. Though I was sure that no one has made any change, still I checked my desktop firewall policy and found nothing.

I contacted IT Administrator of that location and took remote control of one of the systems (172.30.1.15). I observer that PING and TRACERT were working fine. No reachability issue.

I did TELNET to the proxy server (172.30.3.50) on http port and got stuck. It was not happening. I checked Desktop firewall log and there was no sign of connection drop. What else could be that was not allowing me to connect to port 80 to proxy server. I checked logs in proxy server. I found something strange !!! When I was PINGing from that problematic system, accept log was there. But when I was TELNETing on http port, there was neither ACCEPT nor REJECT/ DROP log.

We have a backup ISA proxy (172.30.3.51) in our environment and it is in the same zone with LIVE ISA Proxy. I did a TELNET on HTTP port to the BACKUP Proxy Server from the same system and it worked!!! So now I started scraching my head and took a pen and pad to jot-down the findings.

1. No desktop firewall issue.

2. No perimeter firewall issue as PROXY server and client system are in same zone.

3. No network issue.

4. No ISA Firewall issue.

What next???

Suddenly one point came across my mind. It seemed to me very illogical but I still did that.

A couple of week back, I have pluged in a Bluecoat SG appliance (810-20) in our environment for doing POC. It was connected in the same zone (IP Address: 172.30.3.56) with ISA Proxy and I configured it to take user authentication credentials from my BACKUP Proxy (172.30.3.51) and it was listening on TCP - 80.

I tested from client system and found that TELNETing on http port to that appliance was successful. I removed that appliance immediately from network and WOW.......the issue got resolved then and there. Telneting to LIVE ISA Proxy server on http port was successful now and Internet Browsing was okey.

Now there are lots of queries floating in my head. Was my client system someway getting redirected to Bluecoat SG appliance? Is this network outage created by SG appliance? Or there were some intermittent network connectivity issue which got resolved at the same time I disconnected Bluecoat appliance from network - an unwanted co-incident from a network administrator's perspective. I once again tried to re-generate the issue in my test environment, but failed to replicate the scenario.

If any one of you folks can have any idea on this please share it with me. Thanks in advance.