Improvements to data rebuild

[iwanbk]

Improve the periodic SweepObject:

• replace object_metas (func to get all metadata) which returns Vector to something which returns Stream or using Redis scan (improve sweep object to use scan. #140)
• Avoid duplicated backend checking
• cache unhealthy backend check
• Make the period longer, 10 mins is too fast, especially for big storage

Previous discussion: #131

Code reference:

0-stor_v2/zstor/src/actors/repairer.rs

Lines 59 to 117 in cd24f42

	impl Handler<SweepObjects> for RepairActor {
	type Result = ResponseFuture<()>;
	fn handle(&mut self, _: SweepObjects, _: &mut Self::Context) -> Self::Result {
	let meta = self.meta.clone();
	let backend_manager = self.backend_manager.clone();
	let zstor = self.zstor.clone();
	Box::pin(async move {
	let obj_metas = match meta.send(ObjectMetas).await {
	Err(e) => {
	error!("Could not request object metas from metastore: {}", e);
	return;
	}
	Ok(om) => match om {
	Err(e) => {
	error!("Could not get object metas from metastore: {}", e);
	return;
	}
	Ok(om) => om,
	},
	};
	for (key, metadata) in obj_metas.into_iter() {
	let backend_requests = metadata
	.shards()
	.iter()
	.map(|shard_info| shard_info.zdb())
	.cloned()
	.collect::<Vec<_>>();
	let backends = match backend_manager
	.send(RequestBackends {
	backend_requests,
	interest: StateInterest::Readable,
	})
	.await
	{
	Err(e) => {
	error!("Failed to request backends: {}", e);
	return;
	}
	Ok(backends) => backends,
	};
	let must_rebuild = backends.into_iter().any(|b| !matches!(b, Ok(Some(_))));
	if must_rebuild {
	if let Err(e) = zstor
	.send(Rebuild {
	file: None,
	key: Some(key),
	})
	.await
	{
	warn!("Failed to rebuild data: {}", e);
	}
	}
	}
	})
	}
	}

[iwanbk]

Avoid duplicated backend checking

I found that 0-stor already have this mechanism for the healthy backend.
For the unhealthy backend, we don't have it.
Added cache unhealthy backend check to the task list