Here’s a problem that had me puzzled for longer than I care to admit. I have a C# application that periodically makes rather light-weight calls on a web service. To be a good UI citizen, the app makes the calls on a background thread, and the standard .NET HttpWebRequest mechanism is used to make the calls. The thread maintains a single queue of requests, processes one at a time, creating a new HttpWebRequest object for each, and the results are returned to the UI (if necessary) by a callback.
My problem was that intermittently the calls were timing out (I’d set a 10-second timeout on them). Furthermore, when I examined the server logs, the requests were arriving and being processed, but often much later than they were sent, sometimes by as much as a minute. Yet when I attached a sniffer to the client machine, it showed the request packets leaving the machine right when my program was sending them. What on earth was going on?