python - Serializing output to JSON - ValueError: Circular reference detected

Question

I'm trying to output results of my mysql query to JSON. I have problem with serializing datetime.datetime field, so I wrote small function to do that:

def date_handler(obj):
    if hasattr(obj, 'isoformat'):
        return obj.isoformat()
    else:
        return obj

and then in main code I'm just running:

products_json = []
for code in best_matching_codes:
    cur = db.cursor()
    query = "SELECT * FROM %s WHERE code LIKE '%s'" % (PRODUCTS_TABLE_NAME, product_code)
    cur.execute(query)
    columns = [desc[0] for desc in cur.description]
    rows = cur.fetchall()
    for row in rows:
        products_json.append(dict((k,v) for k,v in zip(columns,row)))   

return json.dumps(products_json, default = date_handler)

However, since I wrote date_handler function, I'm getting "ValueError: Circular reference detected"

127.0.0.1 - - [10/Jan/2013 00:42:18] "GET /1/product?code=9571%2F702 HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1701, in __call__
return self.wsgi_app(environ, start_response)
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1689, in wsgi_app
response = self.make_response(self.handle_exception(e))
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1687, in wsgi_app
response = self.full_dispatch_request()
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1360, in full_dispatch_request
rv = self.handle_user_exception(e)
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1358, in full_dispatch_request
rv = self.dispatch_request()
  File "/Library/Python/2.7/site-packages/flask/app.py", line 1344, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/pisarzp/Desktop/SusyChoosy/susyAPI/test1.py", line 69, in product_search
return json.dumps(products_json, default = date_handler)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 238, in dumps
**kw).encode(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 201, in encode
chunks = self.iterencode(o, _one_shot=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 264, in iterencode
return _iterencode(o, 0)
ValueError: Circular reference detected

What did I break? Is there a better way to serialize output to JSON?

question from:https://stackoverflow.com/questions/14249115/serializing-output-to-json-valueerror-circular-reference-detected

Answer 1

The function you pass as the default argument will only be called for objects that are not natively serializable by the json module. It must return a serializable object, or raise a TypeError.

Your version returns the same object you were passed if it's not of the one type you're fixing (dates). That is causing the circular reference error (which is misleading, since the circle is between one object and itself after being processed by date_handler).

You can start to fix this by changing date_handler to raise an exception in its else block. That will still probably fail, but you can probably then find out what object it is that is in your data structure causing the problem using code like this:

def date_handler(obj):
    if hasattr(obj, 'isoformat'):
        return obj.isoformat()
    else:
        raise TypeError(
            "Unserializable object {} of type {}".format(obj, type(obj))
        )