Why do I get Asset Store errors in my Unity Editor?

For a long time – perhaps the last year – I have been getting occasional errors in the Unity Editor. These errors happen to everyone on the team, not just me. The errors occur sometimes in edit mode, sometimes in play mode. They confuse people — is the game broken? Is a plugin malfunctioning?

It happened to me again a few days ago. I decided it was time to go dig.

Step 1: Look at logs

I had just started up the Unity editor and opened a project. I was greeted with the following error message:

Not very informative… there were problems performing some HTTP communication. At a time when nothing in our editor extensions has any reason to talk over HTTP. The callstack doesn’t say much. Where to next?

We use an editor extension called UberLogger. It offers an improved editor console, in-game console, plugin support, filtering by channels, etc. It also has its own callstack resolution logic.

Here is the result of UberLogger’s callstack resolution:

Okay, so this is a problem within Unity’s own Asset Store code. We get source file names and line numbers. Perhaps we can understand something by looking at the source code…?

Step 2: Inspect source code

Unity Technologies have helpfully released the C# source code for the Engine and Editor on GitHub for read-only purposes. (Thanks, Unity! It makes this kind of troubleshooting a lot easier!)

Let’s have a look in AssetStoreClient.cs, around line 710:

static AssetStoreResponse ParseContent(AsyncHTTPClient job)
{
  ...
  if (!AsyncHTTPClient.IsSuccess(state))
  {
    ...
    return resp;
  }

  ...
  resp.dict = ParseJSON(content, out status, out message);
  ...

  if (status == "error")
  {
    Debug.LogError("Request error (" + status + "): " + message);
    return resp;
  }
  ...
  return resp;
}

We would only get that “Request error (error):” error message, if the HTTP request has been successful. Under which conditions are status set to “error”? Let’s have a look at ParseJson():

static Dictionary<string, JSONValue> ParseJSON(string content, out string status, out string message)
{
  ...
  status = null;

  ...

  try
  {
    JSONParser parser = new JSONParser(content);
    ...
  }
  catch (JSONParseException ex)
  {
    Debug.Log("Error parsing server reply: " + content);
    Debug.Log(ex.Message);
    return null;
  }

  ...
  
  if (dict.ContainsKey("status"))
    status = dict["status"].AsString(true);
  else if (dict.ContainsKey("error"))
  {
    status = dict["error"].AsString(true);
    if (status == "")
      status = "ok";
  }
  else
    status = "ok";
  ...
}

Deciphering this, we see that the JSON parsing was successful (otherwise status would remain null). Also, there was either a { "status": "error" } or a { "error": "error" } key-value pair within the JSON body.

That is as far as we can go just inspecting code. To get any further we need to look at the network traffic.

Step 3: Capture network traffic

The traffic is over HTTPS. The SSL encryption makes it unwieldy to use tools like Wireshark. However, Fiddler is the perfect tool for a situation like this. (Thanks, Telerik! It is a great tool!) It can act as a web proxy and decrypt HTTPS traffic between programs on your machine and servers on the Internet.

Here is an example of the network traffic when starting up the Unity Editor, without any HTTP-level errors signalled. One of those calls might be the error, one might not be.

Browsing through those requests show that the HTTP request to shawarma.unity3d.com is the only that provides a JSON response with the appropriate format.

A couple of test launches of Unity later, and we have examples of both successful and failed calls.

This is a successful response:

{
 "query":{
  "limits": [],
  "offsets": [],
  "query": "bootcon",
  "groups": [],
  "class": [],
  "labels": []
 },
 "status": "ok",
 "groups": [
  {
   "matches": [],
   "time": "0.000",
   "name": "free",
   "total": 0,
   "words": {
    "bootcon": {
    "hits": 0,
    "docs": 0
   },
    "free":{
    "hits": 200186,
    "docs": 173406
   }
  },
  "total_found": 0,
  "error": "",
  "warning": "",
  "label": "Free Assets"
 },
 {
  "matches": [],
  "time": "0.000",
  "name": "paid",
  "total": 0,
  "words": {
   "paid": {
   "hits": 4262110,
   "docs": 4262028
   },
   "bootcon": {
   "hits": 0,
   "docs": 0
   }
  },
  "total_found": 0,
  "error": "",
  "warning": "",
  "label": "Paid Assets"
  }
 ]
}

What’s that? “Paid assets” and counts? “Free assets” and counts? This request is probably used to populate the ‘paid assets’ info which shows up when we search among our assets in the Project view.

This is a failed response:

{
 "query":{
 "limits": [],
 "offsets": [],
 "query": "bootcon",
 "groups": [],
 "class": [],
 "labels": []
 },
 "status": "error",
 "error": "Failed to open connection to localhost:9312: Connection refused... ConnectRetries exceed..."
}

Hm, what’s that? an actual error message? failed to open connection to localhost:9312 ? I don’t have anything running on my machine on that port.

No, this is an error message from the server side. Some backend service behind shawarma.unity3d.com has problems talking with another backend service. What could it be?

Step 4: Google and hope for the best

Are there any popular programs that run on port 9312? Google to the rescue. Apparently a software called Sphinx Search Server commonly runs on that port. And Sphinx Search Server is an open source full text search server. A good guess is that the search server’s database contains all products in the Unity Asset Store. Another good guess is that the connectivity problems are either because search servers in the cluster sometimes stop working, or because the search servers sometimes become overloaded.

Step 5: Put together the pieces

If we take the above findings and weave it into a story, we get this:

The Unity Editor has a feature in the Project browser’s Search view, where it searches the Unity Asset store in addition to your local project whenever you perform a free-text search. This search service can be reached at shawarma.unity3d.com . Sometimes the search servers that power the service refuse requests from the frontfacing API.

Those errors are not caught by the front-facing API, but sent on as-is to the Unity Editor client. Those errors in turn result in error messages in the Console, Play mode randomly pausing, build processes occasionally failing, and a bit of confusion within the team.

Step 6: File a bug with Unity

Search for a few minutes on Unity’s Issue Tracker, and it turns out that this was reported at least one year ago. Alright, let’s add the extra details and hope that Unity will get around to solving it sometime soon.

Leave a Reply

Your email address will not be published. Required fields are marked *