Uploaded image for project: 'onedata'
  1. onedata
  2. VFS-11095

Add backoffs and limit logging when failing to subscribe for events

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Fixed
    • Icon: Minor Minor
    • 21.02.4
    • None
    • None
    • None
    • Sprint 239, Sprint 240, Sprint 241, Sprint 242

      The problem is related to proxy sessions (e.g. when mounting a Oneclient). If there is a subscription request and it should be proxied to a different provider, and the target provider is in an incompatible version (or probably offline - check this), sequencer will retry infinitely, without backoffs, and log a lot of error logs. We should:
      1. Implement backoffs so that it does not try every 5 seconds. Longest backoff should be e.g. an hour.
      2. Limit the logging - if the error is known (e.g. no_connection_to_peer_provider), use a debug log or a throttled warning log.

      Below is an excerpt from the logs:

      [E 09:43:17.037 <0.22274.235>] Cannot process request

      {client_message, {message_id, <<"e42ac1a1e7163b3cc56ab9f9feb33de3ch6cbe">>, <<"573505587337518118">>, <<131,88,100,0,34,111,112,95,119,111,114,107, 101,114,64,102,97,97,115,45,116,101,115,116, 46,100,97,116,97,104,117,98,46,101,103,105, 46,101,117,0,0,86,241,0,0,0,235,100,127,58, 171>>}

      ,
      undefined,
      <<"51994af8ea9515fa1bb64868e9056796ch9515">>,

      {client_tokens, <<"MDAxY2xvY2F00aW9uIGRhdGFodWIuZWdpLmV1CjAwNjdpZGVudGlmaWVyIDIvdG1wLTEvdXNyLWY5Njg2NzRjNDEwMjA4MDc3NTcyYzczYjAwMmI2NmRlL2FjdC8zYzc00NjdkZDM3NDk4MmE3NDE5MmZjOTk1ODhiOThiN2NoZjE2MQowMDFhY2lkIHRpbWUgPCAxNjg5NDEzMjgxCjAwM2RjaWQgc2VydmljZSA9IG9wdy1lNDJhYzFhMWU3MTYzYjNjYzU2YWI5ZjlmZWIzM2RlM2NoNmNiZQowMDNlY2lkIGNvbnN1bWVyID00gcHJ2LWU00MmFjMWExZTcxNjNiM2NjNTZhYjlmOWZlYjMzZGUzY2g2Y2JlCjAwODNjaWQgYXBpID00gb3B3L2FsbC8qLiouKjoqfG96dy9nZXQvdXNlci4qLio6KnxvencvY3JlYXRlL3VzZXIuKi5pZHBfYWNjZXNzX3Rva2VuLCo6KnxvencvZ2V00L3NwYWNlLiouKjoqfG96dy9nZXQvcHJvdmlkZXIuKi4qOioKMDAxYWNpZCB00aW1lIDwgMTY4NjgyMjQ4MgowMDJmc2lnbmF00dXJlICac4G01juIwSC028DJb8o701XOqBFr302oe79pCiZ3mDMLoCg">>, undefined}

      ,
      normal,

      {message_stream,707259639,undefined}

      ,

      {subscription,<<"-5">>, {file_attr_changed_subscription, <<"Z3VpZCNzcGFjZV9hMmZhYjc2YzYwOGEzNjhkZTU0OGE0YTk5ZDczNzE0NyNhMmZhYjc2YzYwOGEzNjhkZTU0OGE0YTk5ZDczNzE0Nw">>, undefined,500}

      ,
      undefined}} due to: {badmatch,
      {error,
      no_connection_to_peer_oneprovider}}. There has been 153 unsuccessful attempts so far. Retrying in 5000 milliseconds...

            plgwrzeszcz Michał Wrzeszcz (Inactive)
            plglopiola Łukasz Opioła
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: