SGD with mini-batches

[MargaretDuff]

We want to be able to do completely random mini-batch stochastic gradient descent ( or maybe other flavours...)

We could consider something like:
https://github.com/epapoutsellis/StochasticCIL/blob/b177e46dfb22c5305f52b6d61f16f42e03ddb2f0/Wrappers/Python/cil/optimisation/functions/SGFunction.py#L61

Example usage:

rs = RandomSampling.uniform(len(f_subsets), batch_size=5) 

F = SGFunction(f_subsets, selection=rs)
proxSGD = ISTA(initial = initial, f=F, g=G, 
            update_objective_interval = rs.num_batches, 
            max_iteration = num_epochs*rs.num_batches)
proxSGD.run(verbose=1)

Points to consider

• Is this part of the currently proposed ApproximateGradientSumFunction or a different class, perhaps a more general class
• We would need to think carefully about how to adjust the stepsize (wrt Lipschitz constant and batch size)
• We would also need to consider data passes ]

Discussed with @zeljkozeljko

[epapoutsellis]

Please check also #1345 (comment)
Walnut comparison for TV recon using SGD and MB.