WP7 HTTP Using Sockets & Problem With HttpWebRequest

Why Would You Want To Do This?

The first question you will have unless you found this article because you know what the problem is already is: why would you want to use sockets when there is a perfectly good way of sending and receiving data using WebClient and HttpWebRequest objects? Well the problem explains it.

The Problem

I recently had to integrate a WP7 Mango app with the Disqus API. I set out thinking it would be fairly straight forward, but as soon as I started work on it I ran into a big problem. I was using HttpWebRequests to do a simple HTTP GET, here's the code:

private void button1_Click(object sender, RoutedEventArgs e)

        {

            HttpWebRequest req = (HttpWebRequest)WebRequest.Create(new Uri("http://disqus.com/api/3.0/threads/listPosts.json?forum=abcd&thread:ident=12345&api_key=abcdef0123456789"));

            req.BeginGetResponse(EndGetResponse, req);

        }

        private void EndGetResponse(IAsyncResult a)

        {

            try

            {

                HttpWebRequest req = a.AsyncState as HttpWebRequest;

                HttpWebResponse res = req.EndGetResponse(a) as HttpWebResponse;

                Stream s = res.GetResponseStream();

                StreamReader str = new StreamReader(s);

                string data = str.ReadToEnd();

            }

            catch (Exception ex)

            {

                string s = ex.ToString();

            }

        }

Normally this should work fine, but an exception was being thrown at EndGetResponse:

System.Net.WebException: The remote server returned an error: NotFound. ---> 

Now this exception is not very helpful so you need to use a protocol analyser (I use WireShark, I gave up with Fiddler because I never got it working with the phone!) to find out what the problem is. It turns out that it's 400 Bad Response code:

Now if you profile a browser making the same request you notice some differences:

For some reason (I don't know what) WP7 (.Net doesn't do this) adds a "Referer" header with a reference to some location on the device. The API server doen't like this and rejects the request. This is the problem.

Possible Solutions

One obvious solution would be to remove the offending header, but this is not possible. Another solution would be to change the "Referer" header value, this is possible but doesn't work. After some thought I realised that since HTTP is only a layer on top of TCP and since MS have kindly given us Sockets to use in the Mango update that Sockets are the solution.

Sockets

First job is to get a new client to handle asynchronous TCP requests, there's a nice MS example for a Tic-Tac-Toe game (Noughts and Crosses if you're from the UK!). This is the example download link: http://go.microsoft.com/fwlink/?LinkId=219075

The AsynchronousSocketClient is the bit we're interested in, but it needs some adjustment because we need the socket to keep receiving data until there is no more. I changed the ProcessReceive method to this:

// Called when a ReceiveAsync operation completes  

        private void ProcessReceive(SocketAsyncEventArgs e)

        {

            Socket sock = e.UserToken as Socket;

            if (e.SocketError == SocketError.Success)

            {

                // Received data from server 

                dataFromServer += Encoding.UTF8.GetString(e.Buffer, 0, e.BytesTransferred);

                

                // More data to receive

                if (e.BytesTransferred > 0)

                {

                    //Read data sent from the server 

                    sock.ReceiveAsync(e);

                }

                else

                {

                    sock.Shutdown(SocketShutdown.Both);

                    sock.Close();

                    sock.Dispose();

                    ResponseReceivedEventArgs args = new ResponseReceivedEventArgs();

                    args.response = dataFromServer;

                    OnResponseReceived(args);

                }

            }

            else

            {

                if (retries < MAX_RETRIES)

                {

                    retries++;

                    try

                    {

                        dataFromServer = string.Empty;

                        sock.ConnectAsync(socketEventArg);

                    }

                    catch (SocketException ex)

                    {

                        throw new SocketException(ex.ErrorCode);

                    }

                }

                else

                {

                    sock.Close();

                    sock.Dispose();

                    ResponseReceivedEventArgs args = new ResponseReceivedEventArgs();

                    args.isError = true;

                    OnResponseReceived(args);

                }

            }

        }

The method calls ReceiveAsync again if BytesTransferred is not 0. I also put in a retry loop in because I found on the emulator, the connection seemed to get reset a lot which was annoying (it works nicely on the phone).

HTTP Over Sockets

Now we have a way of sending and receiving data over TCP, we need to work out how to do an HTTP GET. This is where  WireShark comes in again to look at the structure of the message.

The implemented code looks like this:

private void button2_Click(object sender, RoutedEventArgs e)

        {

            // GET string

            string getString = "GET /api/3.0/threads/listPosts.json?forum=abcd&thread:ident=12345&api_key=abcdef0123456789 HTTP/1.0\r\n";

            // Headers

            getString += "Host: disqus.com\r\n";

            getString += "Connection: keep-alive\r\n";

            // This carriage return is important to separate the content

            getString += "\r\n";

            // If this was a POST, content goes here

            AsyncSocketClient client = new AsyncSocketClient("disqus.com", 80);

            client.ResponseReceived += new ResponseReceivedEventHandler(GetCommentsRequestComplete);

            client.SendData(getString);

        }

        private void GetCommentsRequestComplete(object sender, ResponseReceivedEventArgs e)

        {

            // Client

            var client = sender as AsyncSocketClient;

            client.ResponseReceived -= this.GetCommentsRequestComplete;

            client = null;

            if (!e.isError)

            {

                // Strip off http preamble

                int start = e.response.IndexOf('{');

                string data = e.response.Substring(start);

            }

        }

I started with a full set of headers from the browser analysis, then removed the ones that aren't needed one by one to keep the code more manageable. The "Connection" header is really important, if you use value as "close" the socket will shut down before all data is received, so "keep-alive" must be used.

Conclusion

I've actually used this solution twice already now and think it will come in useful again and again. I hope other people having this same problem find this as it works nicely!