Thursday 19 January 2012

WP7 HTTP Using Sockets & Problem With HttpWebRequest

Why Would You Want To Do This?
The first question you will have unless you found this article because you know what the problem is already is: why would you want to use sockets when there is a perfectly good way of sending and receiving data using WebClient and HttpWebRequest objects? Well the problem explains it.

The Problem
I recently had to integrate a WP7 Mango app with the Disqus API. I set out thinking it would be fairly straight forward, but as soon as I started work on it I ran into a big problem. I was using HttpWebRequests to do a simple HTTP GET, here's the code:

private void button1_Click(object sender, RoutedEventArgs e)
        {
            HttpWebRequest req = (HttpWebRequest)WebRequest.Create(new Uri("http://disqus.com/api/3.0/threads/listPosts.json?forum=abcd&thread:ident=12345&api_key=abcdef0123456789"));
            req.BeginGetResponse(EndGetResponse, req);
        }

        private void EndGetResponse(IAsyncResult a)
        {
            try
            {
                HttpWebRequest req = a.AsyncState as HttpWebRequest;
                HttpWebResponse res = req.EndGetResponse(a) as HttpWebResponse;

                Stream s = res.GetResponseStream();
                StreamReader str = new StreamReader(s);
                string data = str.ReadToEnd();
            }
            catch (Exception ex)
            {
                string s = ex.ToString();
            }
        }

Normally this should work fine, but an exception was being thrown at EndGetResponse:

System.Net.WebException: The remote server returned an error: NotFound. ---> 

Now this exception is not very helpful so you need to use a protocol analyser (I use WireShark, I gave up with Fiddler because I never got it working with the phone!) to find out what the problem is. It turns out that it's 400 Bad Response code:


Now if you profile a browser making the same request you notice some differences:


For some reason (I don't know what) WP7 (.Net doesn't do this) adds a "Referer" header with a reference to some location on the device. The API server doen't like this and rejects the request. This is the problem.

Possible Solutions
One obvious solution would be to remove the offending header, but this is not possible. Another solution would be to change the "Referer" header value, this is possible but doesn't work. After some thought I realised that since HTTP is only a layer on top of TCP and since MS have kindly given us Sockets to use in the Mango update that Sockets are the solution.

Sockets
First job is to get a new client to handle asynchronous TCP requests, there's a nice MS example for a Tic-Tac-Toe game (Noughts and Crosses if you're from the UK!). This is the example download link: http://go.microsoft.com/fwlink/?LinkId=219075

The AsynchronousSocketClient is the bit we're interested in, but it needs some adjustment because we need the socket to keep receiving data until there is no more. I changed the ProcessReceive method to this:

// Called when a ReceiveAsync operation completes  
        private void ProcessReceive(SocketAsyncEventArgs e)
        {
            Socket sock = e.UserToken as Socket;

            if (e.SocketError == SocketError.Success)
            {
                // Received data from server 
                dataFromServer += Encoding.UTF8.GetString(e.Buffer, 0, e.BytesTransferred);
                
                // More data to receive
                if (e.BytesTransferred > 0)
                {
                    //Read data sent from the server 
                    sock.ReceiveAsync(e);
                }
                else
                {
                    sock.Shutdown(SocketShutdown.Both);
                    sock.Close();
                    sock.Dispose();

                    ResponseReceivedEventArgs args = new ResponseReceivedEventArgs();
                    args.response = dataFromServer;
                    OnResponseReceived(args);
                }
            }
            else
            {
                if (retries < MAX_RETRIES)
                {
                    retries++;

                    try
                    {
                        dataFromServer = string.Empty;

                        sock.ConnectAsync(socketEventArg);
                    }
                    catch (SocketException ex)
                    {
                        throw new SocketException(ex.ErrorCode);
                    }
                }
                else
                {
                    sock.Close();
                    sock.Dispose();

                    ResponseReceivedEventArgs args = new ResponseReceivedEventArgs();
                    args.isError = true;
                    OnResponseReceived(args);
                }
            }
        }

The method calls ReceiveAsync again if BytesTransferred is not 0. I also put in a retry loop in because I found on the emulator, the connection seemed to get reset a lot which was annoying (it works nicely on the phone).

HTTP Over Sockets
Now we have a way of sending and receiving data over TCP, we need to work out how to do an HTTP GET. This is where  WireShark comes in again to look at the structure of the message.

The implemented code looks like this:

private void button2_Click(object sender, RoutedEventArgs e)
        {
            // GET string
            string getString = "GET /api/3.0/threads/listPosts.json?forum=abcd&thread:ident=12345&api_key=abcdef0123456789 HTTP/1.0\r\n";
            // Headers
            getString += "Host: disqus.com\r\n";
            getString += "Connection: keep-alive\r\n";
            // This carriage return is important to separate the content
            getString += "\r\n";
            // If this was a POST, content goes here

            AsyncSocketClient client = new AsyncSocketClient("disqus.com", 80);
            client.ResponseReceived += new ResponseReceivedEventHandler(GetCommentsRequestComplete);
            client.SendData(getString);
        }

        private void GetCommentsRequestComplete(object sender, ResponseReceivedEventArgs e)
        {
            // Client
            var client = sender as AsyncSocketClient;
            client.ResponseReceived -= this.GetCommentsRequestComplete;
            client = null;

            if (!e.isError)
            {
                // Strip off http preamble
                int start = e.response.IndexOf('{');
                string data = e.response.Substring(start);
            }
        }

I started with a full set of headers from the browser analysis, then removed the ones that aren't needed one by one to keep the code more manageable. The "Connection" header is really important, if you use value as "close" the socket will shut down before all data is received, so "keep-alive" must be used.

Conclusion
I've actually used this solution twice already now and think it will come in useful again and again. I hope other people having this same problem find this as it works nicely!

5 comments:

  1. While I never have had to bother with this problem, setting the Referer header to something else (as long as it's not null!) is really easy and shouldn't be a problem for you.

    I feel that even if this works for you, it's a bit like trying to reinvent the wheel.


    GET XXX HTTP/1.1
    Accept: */*
    Accept-Charset: utf-8;
    Accept-Encoding: identity
    Referer: http://I.like.cats/
    User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.12 Safari/535.11
    Host: XXX
    Connection: Keep-Alive

    request.Headers["Referer"] = "http://I.like.cats/";

    ReplyDelete
  2. If I remember rightly the disqus API would not accept requests with the referer header and it's not actually possible to completely remove it using httpwebrequest

    ReplyDelete
    Replies
    1. While I cannot find a single reason not to allow the referer header for Disqus I do find it even more weird that you cannot completely remove it using httpwebrequest, but yes, I did get an exception if I tried setting it as null or "blank" in an attempt to remove it.

      Delete
  3. Yes it's strange that disqus is so picky and that the headers can't be fully controlled. I spent a whole day trying to get it to work so decided to use sockets :-)

    ReplyDelete
  4. HI Geoff,
    Am just getting into Wp8 now and this issue still persist and your article and explanation has helped me so much that I cant thank you enough.
    Additionally, wanted to put forward a point that my connections for https through port 443 is not working. In the sense its connecting by response am getting a "blank" string.
    Regards,
    Purab

    ReplyDelete