How to use the token obtained from GoogleAuthUtil with Web API 2 default template project

My intent is to authenticate using Google and once authenticated access my resource server (written in Web API 2).

I am able to get the authentication token from Google for the ClientID that I have created in Google's developer console. Here is the code that does it (courtesy: http://ift.tt/1F7hdwa):

private class RetrieveIdTokenTask extends AsyncTask<String, Void, String>  {
    protected String doInBackground(String... params) {
        String account = params[0];
        String homeServerClient = "276706532100-mygoogleclient.apps.googleusercontent.com";
        try {
            String scope = "audience:server:client_id:" + homeServerClient;
            final String token = GoogleAuthUtil.getToken(LoginActivity.this, account, scope);
            return token;
        } catch (GooglePlayServicesAvailabilityException playEx) {
            // In this case you could prompt the user to upgrade.
        } catch (UserRecoverableAuthException userAuthEx) {
            // This should not occur for ID tokens.
        } catch (IOException transientEx) {
            // You could retry in this case.
        } catch (GoogleAuthException authEx) {
            // General auth error.
        }
        return null;
    }

    protected void onPostExecute(String result){
        Log.d("Token", result);
    }
}



@Override
public void onConnected(Bundle bundle) {
    // get the email address
    Person currentPerson = Plus.PeopleApi
            .getCurrentPerson(mGoogleApiClient);
    String email = Plus.AccountApi.getAccountName(mGoogleApiClient);
    new RetrieveIdTokenTask().execute(email);

}

So far so good, but here is the issue:

Once I got the token I used it in fiddler to send it to the ValuesController. The ValuesController in default template has [Authorize] attribute and I was thinking that following should authorize the request:

1. Uncommenting following lines and adding the missing details in Startup.Auth.cs.

       app.UseGoogleAuthentication(new GoogleOAuth2AuthenticationOptions()
       {
           ClientId = "276706532100-mygoogleclient.apps.googleusercontent.com",
           ClientSecret = "nbQjjRnivx1hjmQnkuBEbhbj"
       });
   
   

2. Sending the token in HTTP request as "authorization" header

But it didn't work and I keep on getting 401 (Unauthorized). Here is how my fiddler request looks like:

**URL :** 
http://localhost:60864/api/values/

**Header section:**
User-Agent: Fiddler
authorization: bearer eyJhbGc...REMOVED FOR BREVITY..iOiJSUzI1NiIsIzuhlgIE-ZC5mlmyNnpXEOP2qplbYCmw
content-type: application/json
Host: localhost:60864

Any idea, what is it that I am missing? I am getting a feeling that either I am so close and missing one last little step, or, I have got it all wrong :)

Help!