Improve circuit breaker error msg

Answered

Follow

Håkon Mellbye

• January 31, 2023 12:11

It´s a bit hard to read subsequent pump-failed logs after a circuit-breaker.
It would be useful to easier see how many entities are in the queue, and this should be preserved in the subsequent pump-failed logs. Further on it's a bit unclear what the difference to commit dataset and rollback dataset is. Maybe add some more information in the warning that appears after pressing the buttons in the output pane. (possible also add the link to commit/roll back in the execution log) 

Example of pump-failed-entity after initial failure
{
  "end_time": "2023-01-31T11:57:51.463999Z",
  "event_type": "pump-failed",
"reason_why_stopped": "Pipe is not ready: Circuit breaker is tripped.",
  ......
    "lake.task.exceptions.DatasyncReadException: Pipe is not ready: Circuit breaker is tripped."
  ]
}

Example of pump-failed-entity on initial failure
{
  "end_time": "2023-01-31T11:32:46.022396Z",
  "event_type": "pump-failed",
  
  "original_error_message": "",
  "original_traceback": "",
  "pipe": "meterpoint-msim-share",
  "pipe_offset": "98143",
  "pump_definition": "pump:meterpoint-msim-share",
  "pump_started_location": 10563,
  "reason_why_stopped": "Circuit breaker tripped: count: 100, limit: 100",
  "retry_entities_exist": false,
  "start_time": "2023-01-31T11:32:45.451625Z",
  "sync_type": "incremental",
  "total_time": 0.570771,
  "traceback": [
    "lake.task.exceptions.CircuitBreakerTrippedException: Circuit breaker tripped: count: 100, limit: 100"
  ]
}

1

• Official comment

  

  Geir Ove Grønmo

  • March 03, 2023 12:11
  • Edited

  The "Pipe is not ready: " part of the message is just to indicate that the pipe wasn't able to run. I guess using "The pipe cannot run now:" might be better?
  
  It is possible to see the pipe queue size in the pipe API and it will grow over time. We could probably add the queue size at the time to the pump-failed event. That is possible, but do you actually need to know the queue size at the time – or is it enough to see the current pipe queue size?
  
  In the example you've posted there are 100 uncommited entities and the circuit breaker limit is 100. If you commit then those 100 entities are committed and they become visible. If rolled back then they are physically deleted from the dataset.

  Comment actions Permalink
• 

  Maximilien Sauban

  • January 31, 2023 12:25

  The "pipe not ready" comment also through me off a little. I read it as the pipe is not ready so the circuit breaker is tripped.

  0

  Comment actions Permalink

Please sign in to leave a comment.