Reduce runtime

[bennahugo]

Reduce fractional DFT computation time by reducing rows and channels where possible. Halves the runtime of the testing on the PP suite.

110.17s user 2.38s system 105% cpu 1:46.49 total

[bennahugo]

fixes #226

[bennahugo]

I've further reduced this runtime to a minute locally. However it appears the runtime does not really decrease on the workflow testing server. If this becomes a real headache I can set something up on Jenkins.

[sjperkins]

I've further reduced this runtime to a minute locally. However it appears the runtime does not really decrease on the workflow testing server. If this becomes a real headache I can set something up on Jenkins.

Cool, I think this is very good for the moment. I'll pick this up again next week as I'm on leave for the next 5 days.

[sjperkins]

In terms of functionality, the PP Gridder is in 0.2.7, released today.

[sjperkins]

@bennahugo I've gone and redone one of the test cases with a fixture that caches the DFT. Would you be OK to take it the rest of the way?

There's a slightly simpler example below that demonstrates the idea. While there are three cases, im_to_vis is only called twice because two of the inputs are the same.

Note well the use of the indirect kwarg to parametrize the fixtures. (docstrings here)

import numpy as np
import pytest


@pytest.fixture(scope="module", params=[100])
def uvw(request):
    return np.random.random((request.param, 3)).astype(np.float64)


@pytest.fixture(scope="module", params=[16])
def freq(request):
    return np.linspace(.856e9, 2*.856e9, request.param)


@pytest.fixture(scope="module")
def im_to_vis(request, uvw, freq):
    nrow = uvw.shape[0]
    nchan = freq.shape[0]
    print("im_to_vis", (nrow, nchan))

    return np.zeros((nrow, nchan), dtype=np.complex64)



@pytest.mark.parametrize("npix", [128])
@pytest.mark.parametrize("uvw", [128], indirect=True)
@pytest.mark.parametrize("freq", [32], indirect=True)
def test_stuff(npix, uvw, freq, im_to_vis):
    image = np.ones((npix, npix))

    vis = im_to_vis

@pytest.mark.parametrize("npix", [128])
@pytest.mark.parametrize("uvw", [128], indirect=True)
@pytest.mark.parametrize("freq", [32], indirect=True)
def test_stuff2(npix, uvw, freq, im_to_vis):
    image = np.ones((npix, npix))

    vis = im_to_vis

@pytest.mark.parametrize("npix", [128])
@pytest.mark.parametrize("uvw", [100], indirect=True)
@pytest.mark.parametrize("freq", [16], indirect=True)
def test_stuff3(npix, uvw, freq, im_to_vis):
    image = np.ones((npix, npix))

    vis = im_to_vis

[sjperkins]

Also, it would be preferable to create a new branch off master in future, rather than re-using old branches.

[sjperkins]

Note that the following is also possible!

@pytest.fixture(scope="module", params=[np.array([1.4e9])])
def frequency(request):
    return request.param

@pytest.fixture(scope="module")
def wavelength(frequency):
    return lightspeed / frequency

# Override default frequency fixture value, and by implication wavelenth
@pytest.mark.parameterize("frequency", [np.array([1.2e9])], indirect=True)
def test_this_thing(wavelength):
   assert lightspeed / np.array([1.2e9]) == wavelength