Retry on MySQL InnoDB deadlock

[awestendorf]

The following stack trace shows what happened when migrating data from a redis histogram to mysql innodb. Although this bug could be fixed in the torus migration tool, any application inserting data could run into this and so kairos should take care of it if that's possible.

<Greenlet at 0x2dd15f0: spawn_target(1380844800L, {'524ecf6fadaa6d7a624a3f5e': 24, '524afa5044d90511)> failed with InternalError

Traceback (most recent call last):
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/gevent/greenlet.py", line 328, in run
    result = self._run(*self.args, **self.kwargs)
  File "bin/migrate", line 89, in spawn_target
    store(stat, k, timestamp)
  File "bin/migrate", line 98, in store
    target_schema.store(stat, value, timestamp)
  File "/home/aaron/projects/torus/torus/schema.py", line 77, in store
    self.timeseries.insert(stat, val, timestamp, intervals=self._rolling)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/kairos/timeseries.py", line 364, in insert
    self._insert( name, value, timestamp, intervals )
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/kairos/sql_backend.py", line 96, in _insert
    self._insert_data(name, value, timestamp, interval, config)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/kairos/sql_backend.py", line 261, in _insert_data
    if not self._update_data(name, value, timestamp, interval, config, conn):
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/kairos/sql_backend.py", line 294, in _update_data
    rval = conn.execute( stmt )
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 662, in execute
    params)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
    context)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
    exc_info
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
    context)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 324, in do_execute
    cursor.execute(statement, parameters)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/pymysql/cursors.py", line 102, in execute
    result = self._query(query)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/pymysql/cursors.py", line 202, in _query
    conn.query(q)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/pymysql/connections.py", line 734, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/pymysql/connections.py", line 845, in _read_query_result
    result.read()
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/pymysql/connections.py", line 1049, in read
    first_packet = self.connection._read_packet()
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/pymysql/connections.py", line 826, in _read_packet
    packet.check_error()
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/pymysql/connections.py", line 373, in check_error
    raise_mysql_exception(self.__data)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/pymysql/err.py", line 117, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "/home/aaron/.virtualenvs/torus/local/lib/python2.7/site-packages/pymysql/err.py", line 113, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
InternalError: (InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE histogram SET count=(histogram.count + %s) WHERE histogram.name = %s AND histogram.`interval` = %s AND histogram.i_time = %s AND histogram.r_time IS NULL AND histogram.value = %s'  ($ARGUMENTS)