oneprovider_1 | 2023-07-13T10:43:40.780224643Z [op_worker] [I 2023-07-13 09:26:52.651 <0.3134.2>] Fixing task description after restart:
oneprovider_1 | 2023-07-13T10:43:40.780230569Z [op_worker] Pool = <<"storage_sync_traverse">>
oneprovider_1 | 2023-07-13T10:43:40.780236456Z [op_worker] TaskId = <<"storage_sync###b1578549109c03998dafd32149e722c6che29e###a5e1fd31dd4b62b0383d2679202d3f47cheb65###304">>
oneprovider_1 | 2023-07-13T10:43:40.780242765Z [op_worker] Task = {traverse_task,storage_sync_traverse,<<"default_executor">>,
oneprovider_1 | 2023-07-13T10:43:40.780248687Z [op_worker] <<"default_executor">>,<<"main_group">>,1688800021,
oneprovider_1 | 2023-07-13T10:43:40.780255744Z [op_worker] 1688800066,1688800066,
oneprovider_1 | 2023-07-13T10:43:40.780264632Z [op_worker] <<"63f28a759d2722cb6b81c125416b09bdch2138">>,false,
oneprovider_1 | 2023-07-13T10:43:40.780271446Z [op_worker] false,'op_worker@ceitec1.onedata.e-infra.cz',
oneprovider_1 | 2023-07-13T10:43:40.780279922Z [op_worker] finished,
oneprovider_1 | 2023-07-13T10:43:40.780288673Z [op_worker] #

{master_jobs_delegated => 3,master_jobs_done => 3, oneprovider_1 | 2023-07-13T10:43:40.780294948Z [op_worker] slave_jobs_delegated => 0,slave_jobs_done => 0, oneprovider_1 | 2023-07-13T10:43:40.780320368Z [op_worker] slave_jobs_failed => 0}

,
oneprovider_1 | 2023-07-13T10:43:40.780333189Z [op_worker] #{},all}
oneprovider_1 | 2023-07-13T10:43:40.780339004Z [op_worker] [I 2023-07-13 09:26:52.652 <0.4023.2>] Fixing task description after restart:
oneprovider_1 | 2023-07-13T10:43:40.780345483Z [op_worker] Pool = <<"storage_sync_traverse">>
oneprovider_1 | 2023-07-13T10:43:40.780353391Z [op_worker] TaskId = <<"storage_sync###ce23f50234243a6ca0164e4980edba6ccha386###71ef9b8dd99535bb9dca2194288c6db3ch931a###291">>
oneprovider_1 | 2023-07-13T10:43:40.780361743Z [op_worker] Task = {traverse_task,storage_sync_traverse,<<"default_executor">>,
oneprovider_1 | 2023-07-13T10:43:40.780371054Z [op_worker] <<"default_executor">>,<<"main_group">>,1688641003,
oneprovider_1 | 2023-07-13T10:43:40.780378069Z [op_worker] 1688641003,1688641012,
oneprovider_1 | 2023-07-13T10:43:40.780383664Z [op_worker] <<"3d07951d52071cf5449abf6ff8905190ch4090">>,false,
oneprovider_1 | 2023-07-13T10:43:40.780389943Z [op_worker] false,'op_worker@ceitec1.onedata.e-infra.cz',
oneprovider_1 | 2023-07-13T10:43:40.780395896Z [op_worker] finished,
oneprovider_1 | 2023-07-13T10:43:40.780401713Z [op_worker] #

{master_jobs_delegated => 5,master_jobs_done => 5, oneprovider_1 | 2023-07-13T10:43:40.780409458Z [op_worker] slave_jobs_delegated => 0,slave_jobs_done => 0, oneprovider_1 | 2023-07-13T10:43:40.780418053Z [op_worker] slave_jobs_failed => 0}

,
oneprovider_1 | 2023-07-13T10:43:40.780441092Z [op_worker] #{},all}
oneprovider_1 | 2023-07-13T10:43:40.780449220Z [op_worker] [I 2023-07-13 09:26:52.653 <0.10541.2>] Fixing task description after restart:
oneprovider_1 | 2023-07-13T10:43:40.780458589Z [op_worker] Pool = <<"storage_sync_traverse">>
oneprovider_1 | 2023-07-13T10:43:40.780466920Z [op_worker] TaskId = <<"storage_sync###9a11c80223470701b8deac2a1d9dfa15cha44e###9b442820499bb2390f6ac3e1c3796c96ch3f17###305">>
oneprovider_1 | 2023-07-13T10:43:40.780476217Z [op_worker] Task = {traverse_task,storage_sync_traverse,<<"default_executor">>,
oneprovider_1 | 2023-07-13T10:43:40.780484880Z [op_worker] <<"default_executor">>,<<"main_group">>,1688597775,
oneprovider_1 | 2023-07-13T10:43:40.780491390Z [op_worker] 1688597775,1688597778,
oneprovider_1 | 2023-07-13T10:43:40.780497198Z [op_worker] <<"28cb95456808f09122bcb4a4bd1299c2ch3c30">>,false,
oneprovider_1 | 2023-07-13T10:43:40.780503011Z [op_worker] false,'op_worker@ceitec1.onedata.e-infra.cz',
oneprovider_1 | 2023-07-13T10:43:40.780508588Z [op_worker] finished,
oneprovider_1 | 2023-07-13T10:43:40.780514493Z [op_worker] #

{master_jobs_delegated => 1,master_jobs_done => 1, oneprovider_1 | 2023-07-13T10:43:40.780520630Z [op_worker] slave_jobs_delegated => 0,slave_jobs_done => 0, oneprovider_1 | 2023-07-13T10:43:40.780526626Z [op_worker] slave_jobs_failed => 0}

,
oneprovider_1 | 2023-07-13T10:43:40.780532488Z [op_worker] #{},all}
oneprovider_1 | 2023-07-13T10:43:40.780538586Z [op_worker] [I 2023-07-13 09:26:52.666 <0.1203.0>] 0 jobs of task <<"storage_sync###9a11c80223470701b8deac2a1d9dfa15cha44e###9b442820499bb2390f6ac3e1c3796c96ch3f17###305">> cleaned - restart impossible
oneprovider_1 | 2023-07-13T10:43:40.780546315Z [op_worker] [I 2023-07-13 09:26:52.667 <0.1203.0>] 0 jobs of task <<"storage_sync###b1578549109c03998dafd32149e722c6che29e###a5e1fd31dd4b62b0383d2679202d3f47cheb65###304">> cleaned - restart impossible
oneprovider_1 | 2023-07-13T10:43:40.780552401Z [op_worker] [I 2023-07-13 09:26:52.667 <0.1203.0>] 0 jobs of task <<"storage_sync###ce23f50234243a6ca0164e4980edba6ccha386###71ef9b8dd99535bb9dca2194288c6db3ch931a###291">> cleaned - restart impossible
oneprovider_1 | 2023-07-13T10:43:40.780558693Z [op_worker] [I 2023-07-13 09:26:52.685 <0.1203.0>] Task <<"storage_sync###812c4fce89d6a3caf86329809487ae01chd959###284bae3bf73606bb8e88f167936b9835ch99aa###1">> started on restart of node for pool <<"storage_sync_traverse">> and executor <<"default_executor">>
oneprovider_1 | 2023-07-13T10:43:40.780565562Z [op_worker] [E 2023-07-13 09:26:52.688 <0.1203.0>] An unexpected exception occurred in internal_service:apply_with_retry/5 line 183
oneprovider_1 | 2023-07-13T10:43:40.780571862Z [op_worker] > Stacktrace:
oneprovider_1 | 2023-07-13T10:43:40.780577726Z [op_worker] auto_storage_import_worker:ensure_pool_initialized/0 line 195
oneprovider_1 | 2023-07-13T10:43:40.780583607Z [op_worker] internal_services_manager:start_service/4 line 66
oneprovider_1 | 2023-07-13T10:43:40.780589159Z [op_worker] rpc:call/5 line 401
oneprovider_1 | 2023-07-13T10:43:40.780594837Z [op_worker] erpc:call/5 line 126
oneprovider_1 | 2023-07-13T10:43:40.780600308Z [op_worker] erpc:execute_call/3 line 607
oneprovider_1 | 2023-07-13T10:43:40.780605811Z [op_worker] internal_services_manager:init_service/4 line 230
oneprovider_1 | 2023-07-13T10:43:40.780611692Z [op_worker] internal_service:apply_with_retry/5 line 180
oneprovider_1 | 2023-07-13T10:43:40.780617589Z [op_worker] traverse:schedule_task_and_check_other_waiting/4 line 1133
oneprovider_1 | 2023-07-13T10:43:40.780623534Z [op_worker] > Details: Error while applying fun traverse:init_pool_service with args [<<"storage_sync_traverse">>,
oneprovider_1 | 2023-07-13T10:43:40.780630032Z [op_worker] 10,
oneprovider_1 | 2023-07-13T10:43:40.780635692Z [op_worker] 10,
oneprovider_1 | 2023-07-13T10:43:40.780641270Z [op_worker] 10,
oneprovider_1 |0m 2023-07-13T10:43:40.780646987Z [op_worker] #{}: error:{case_clause,
oneprovider_1 | 2023-07-13T10:43:40.780652710Z [op_worker] {badrpc,
oneprovider_1 | 2023-07-13T10:43:40.780659442Z [op_worker] {'EXIT',
oneprovider_1 | 2023-07-13T10:43:40.780665522Z [op_worker] {{badmatch,
oneprovider_1 | 2023-07-13T10:43:40.780671448Z [op_worker] [

{error, oneprovider_1 | 2023-07-13T10:43:40.780677418Z [op_worker] already_exists}

]},
oneprovider_1 | 2023-07-13T10:43:40.780683087Z [op_worker] [{traverse_task_list,
oneprovider_1 | 2023-07-13T10:43:40.780688693Z [op_worker] add_link_with_timestamp,
oneprovider_1 | 2023-07-13T10:43:40.780694359Z [op_worker] 5,
oneprovider_1 | 2023-07-13T10:43:40.780699973Z [op_worker] [

{file, oneprovider_1 | 2023-07-13T10:43:40.780705983Z [op_worker] "/build/op-worker-thfkZf/op-worker-21.02.2/_build/default/lib/cluster_worker/src/modules/datastore/models/traverse/traverse_task_list.erl"}

,
oneprovider_1 | 2023-07-13T10:43:40.780712332Z [op_worker]

{line, oneprovider_1 | 2023-07-13T10:43:40.780718206Z [op_worker] 334}

]},
oneprovider_1 | 2023-07-13T10:43:40.780724134Z [op_worker] {traverse_task,
oneprovider_1 | 2023-07-13T10:43:40.780729785Z [op_worker] start,
oneprovider_1 | 2023-07-13T10:43:40.780735539Z [op_worker] 5,
oneprovider_1 | 2023-07-13T10:43:40.780747473Z [op_worker] [

{file, oneprovider_1 | 2023-07-13T10:43:40.780757023Z [op_worker] "/build/op-worker-thfkZf/op-worker-21.02.2/_build/default/lib/cluster_worker/src/modules/datastore/models/traverse/traverse_task.erl"}

,
oneprovider_1 | 2023-07-13T10:43:40.780764179Z [op_worker]

{line, oneprovider_1 | 2023-07-13T10:43:40.780770306Z [op_worker] 146}

]},
oneprovider_1 | 2023-07-13T10:43:40.780776170Z [op_worker] {traverse,
oneprovider_1 | 2023-07-13T10:43:40.780782811Z [op_worker] run_task,
oneprovider_1 | 2023-07-13T10:43:40.780790630Z [op_worker] 3,
oneprovider_1 | 2023-07-13T10:43:40.780811833Z [op_worker] [

{file, oneprovider_1 | 2023-07-13T10:43:40.780821574Z [op_worker] "/build/op-worker-thfkZf/op-worker-21.02.2/_build/default/lib/cluster_worker/src/traverse/traverse.erl"}

,
oneprovider_1 | 2023-07-13T10:43:40.780830612Z [op_worker]

{line, oneprovider_1 | 2023-07-13T10:43:40.780838041Z [op_worker] 804}

]}]}}}}
oneprovider_1 | 2023-07-13T10:43:40.780845807Z [op_worker] next retry in 100 ms
oneprovider_1 | 2023-07-13T10:43:40.780853895Z [op_worker] [E 2023-07-13 09:26:52.789 <0.1203.0>] An unexpected exception occurred in internal_service:apply_with_retry/5 line 183
oneprovider_1 | 2023-07-13T10:43:40.780862971Z [op_worker] > Stacktrace:
oneprovider_1 | 2023-07-13T10:43:40.780871519Z [op_worker] auto_storage_import_worker:ensure_pool_initialized/0 line 195
oneprovider_1 | 2023-07-13T10:43:40.780880224Z [op_worker] internal_services_manager:start_service/4 line 66
oneprovider_1 | 2023-07-13T10:43:40.780889480Z [op_worker] rpc:call/5 line 401
oneprovider_1 | 2023-07-13T10:43:40.780897816Z [op_worker] erpc:call/5 line 126
oneprovider_1 | 2023-07-13T10:43:40.780906733Z [op_worker] erpc:execute_call/3 line 607
oneprovider_1 | 2023-07-13T10:43:40.780928438Z [op_worker] internal_services_manager:init_service/4 line 230
oneprovider_1 | 2023-07-13T10:43:40.780937769Z [op_worker] internal_service:apply_with_retry/5 line 180
oneprovider_1 | 2023-07-13T10:43:40.780945903Z [op_worker] traverse:init_pool_service/5 line 225

oneprovider_1 | 2023-07-13T10:43:40.780954749Z [op_worker] > Details: Error while applying fun traverse:init_pool_service with args [<<"storage_sync_traverse">>,
oneprovider_1 | 2023-07-13T10:43:40.780964760Z [op_worker] 10,
oneprovider_1 | 2023-07-13T10:43:40.780973581Z [op_worker] 10,
oneprovider_1 | 2023-07-13T10:43:40.780983659Z [op_worker] 10,
oneprovider_1 |0m 2023-07-13T10:43:40.780992167Z [op_worker] #{}: throw:

{error, oneprovider_1 | 2023-07-13T10:43:40.781001170Z [op_worker] already_exists}

oneprovider_1 | 2023-07-13T10:43:40.781010518Z [op_worker] next retry in 200 ms
oneprovider_1 | 2023-07-13T10:43:40.781019462Z [op_worker] [E 2023-07-13 09:26:52.990 <0.1203.0>] An unexpected exception occurred in internal_service:apply_with_retry/5 line 183
oneprovider_1 | 2023-07-13T10:43:40.781028783Z [op_worker] > Stacktrace:
oneprovider_1 | 2023-07-13T10:43:40.781038211Z [op_worker] auto_storage_import_worker:ensure_pool_initialized/0 line 195
oneprovider_1 | 2023-07-13T10:43:40.781047799Z [op_worker] internal_services_manager:start_service/4 line 66
oneprovider_1 | 2023-07-13T10:43:40.781078749Z [op_worker] rpc:call/5 line 401
oneprovider_1 | 2023-07-13T10:43:40.781085825Z [op_worker] erpc:call/5 line 126
oneprovider_1 | 2023-07-13T10:43:40.781091604Z [op_worker] erpc:execute_call/3 line 607
oneprovider_1 | 2023-07-13T10:43:40.781097175Z [op_worker] internal_services_manager:init_service/4 line 230
oneprovider_1 | 2023-07-13T10:43:40.781104981Z [op_worker] internal_service:apply_with_retry/5 line 180
oneprovider_1 | 2023-07-13T10:43:40.781114366Z [op_worker] traverse:init_pool_service/5 line 225
oneprovider_1 | 2023-07-13T10:43:40.781123348Z [op_worker] > Details: Error while applying fun traverse:init_pool_service with args [<<"storage_sync_traverse">>,
oneprovider_1 | 2023-07-13T10:43:40.781132631Z [op_worker] 10,
oneprovider_1 | 2023-07-13T10:43:40.781141198Z [op_worker] 10,
oneprovider_1 | 2023-07-13T10:43:40.781159627Z [op_worker] 10,
oneprovider_1 |0m 2023-07-13T10:43:40.781170135Z [op_worker] #{}: throw:

{error, oneprovider_1 | 2023-07-13T10:43:40.781178999Z [op_worker] already_exists}

oneprovider_1 | 2023-07-13T10:43:40.781187066Z [op_worker] next retry in 400 ms
oneprovider_1 | 2023-07-13T10:43:40.781194638Z [op_worker] [E 2023-07-13 09:26:53.391 <0.1203.0>] An unexpected exception occurred in internal_service:apply_with_retry/5 line 183
oneprovider_1 | 2023-07-13T10:43:40.781202871Z [op_worker] > Stacktrace:
oneprovider_1 | 2023-07-13T10:43:40.781211235Z [op_worker] auto_storage_import_worker:ensure_pool_initialized/0 line 195
oneprovider_1 | 2023-07-13T10:43:40.781219839Z [op_worker] internal_services_manager:start_service/4 line 66
oneprovider_1 | 2023-07-13T10:43:40.781227949Z [op_worker] rpc:call/5 line 401
oneprovider_1 | 2023-07-13T10:43:40.781235753Z [op_worker] erpc:call/5 line 126
oneprovider_1 | 2023-07-13T10:43:40.781244085Z [op_worker] erpc:execute_call/3 line 607
oneprovider_1 | 2023-07-13T10:43:40.781253916Z [op_worker] internal_services_manager:init_service/4 line 230
oneprovider_1 | 2023-07-13T10:43:40.781262629Z [op_worker] internal_service:apply_with_retry/5 line 180
oneprovider_1 | 2023-07-13T10:43:40.781287722Z [op_worker] traverse:init_pool_service/5 line 225
oneprovider_1 | 2023-07-13T10:43:40.781296552Z [op_worker] > Details: Error while applying fun traverse:init_pool_service with args [<<"storage_sync_traverse">>,
oneprovider_1 | 2023-07-13T10:43:40.781306546Z [op_worker] 10,
oneprovider_1 | 2023-07-13T10:43:40.781315378Z [op_worker] 10,
oneprovider_1 | 2023-07-13T10:43:40.781324558Z [op_worker] 10,
oneprovider_1 |0m 2023-07-13T10:43:40.781333627Z [op_worker] #{}: throw:

{error, oneprovider_1 | 2023-07-13T10:43:40.781342100Z [op_worker] already_exists}

oneprovider_1 | 2023-07-13T10:43:40.781350714Z [op_worker] next retry in 800 ms