Uploaded image for project: 'onedata'
  1. onedata
  2. VFS-11095

Add backoffs and limit logging when failing to subscribe for events

XMLWordPrintable

      • Icon: Task Task
      • Resolution: Fixed
      • Icon: Minor Minor
      • 21.02.4
      • None
      • None
      • None
      • Sprint 239, Sprint 240, Sprint 241, Sprint 242

        The problem is related to proxy sessions (e.g. when mounting a Oneclient). If there is a subscription request and it should be proxied to a different provider, and the target provider is in an incompatible version (or probably offline - check this), sequencer will retry infinitely, without backoffs, and log a lot of error logs. We should:
        1. Implement backoffs so that it does not try every 5 seconds. Longest backoff should be e.g. an hour.
        2. Limit the logging - if the error is known (e.g. no_connection_to_peer_provider), use a debug log or a throttled warning log.

        Below is an excerpt from the logs:

        [E 09:43:17.037 <0.22274.235>] Cannot process request

        {client_message, {message_id, <<"e42ac1a1e7163b3cc56ab9f9feb33de3ch6cbe">>, <<"573505587337518118">>, <<131,88,100,0,34,111,112,95,119,111,114,107, 101,114,64,102,97,97,115,45,116,101,115,116, 46,100,97,116,97,104,117,98,46,101,103,105, 46,101,117,0,0,86,241,0,0,0,235,100,127,58, 171>>}

        ,
        undefined,
        <<"51994af8ea9515fa1bb64868e9056796ch9515">>,

        {client_tokens, <<"MDAxY2xvY2F00aW9uIGRhdGFodWIuZWdpLmV1CjAwNjdpZGVudGlmaWVyIDIvdG1wLTEvdXNyLWY5Njg2NzRjNDEwMjA4MDc3NTcyYzczYjAwMmI2NmRlL2FjdC8zYzc00NjdkZDM3NDk4MmE3NDE5MmZjOTk1ODhiOThiN2NoZjE2MQowMDFhY2lkIHRpbWUgPCAxNjg5NDEzMjgxCjAwM2RjaWQgc2VydmljZSA9IG9wdy1lNDJhYzFhMWU3MTYzYjNjYzU2YWI5ZjlmZWIzM2RlM2NoNmNiZQowMDNlY2lkIGNvbnN1bWVyID00gcHJ2LWU00MmFjMWExZTcxNjNiM2NjNTZhYjlmOWZlYjMzZGUzY2g2Y2JlCjAwODNjaWQgYXBpID00gb3B3L2FsbC8qLiouKjoqfG96dy9nZXQvdXNlci4qLio6KnxvencvY3JlYXRlL3VzZXIuKi5pZHBfYWNjZXNzX3Rva2VuLCo6KnxvencvZ2V00L3NwYWNlLiouKjoqfG96dy9nZXQvcHJvdmlkZXIuKi4qOioKMDAxYWNpZCB00aW1lIDwgMTY4NjgyMjQ4MgowMDJmc2lnbmF00dXJlICac4G01juIwSC028DJb8o701XOqBFr302oe79pCiZ3mDMLoCg">>, undefined}

        ,
        normal,

        {message_stream,707259639,undefined}

        ,

        {subscription,<<"-5">>, {file_attr_changed_subscription, <<"Z3VpZCNzcGFjZV9hMmZhYjc2YzYwOGEzNjhkZTU0OGE0YTk5ZDczNzE0NyNhMmZhYjc2YzYwOGEzNjhkZTU0OGE0YTk5ZDczNzE0Nw">>, undefined,500}

        ,
        undefined}} due to: {badmatch,
        {error,
        no_connection_to_peer_oneprovider}}. There has been 153 unsuccessful attempts so far. Retrying in 5000 milliseconds...

                  plgwrzeszcz Michał Wrzeszcz (Inactive)
                  plglopiola Łukasz Opioła
                  Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                    Created:
                    Updated:
                    Resolved: