Debug ETL on managed infrastructure, i.e. [new] staging

[mattfullerton]

Because ideally this should work here too

[mattfullerton]

@davidmihalyi Have you had any recent experience with this? I.e. can we do a full CH load and larger GS loads on staging.resourceprojects.org?

[davidmihalyi]

Still not able to get full CH import through on staging... :(

[mattfullerton]

OK, thanks for trying!

[davidmihalyi]

also no error message either...

[mattfullerton]

It will be a crash/app being killed due to too many operation. In this case reports are not saved because the process doesn't get to finish and return a report. I will look in the logs.

[mattfullerton]

The CH import works on a version of staging with more memory (2G) and no LB that would kill it in the event of failing healthchecks (cc @iprunache that I am getting this right).

We need to make a final decision on whether this is a tenable solution for the future or whether we want to live with giving our instances more memory just because of infrequent imports.

Certainly I'm sure some optimization can be done, for example tuning what data is loaded from the DB (i.e. only certain fields, or tuning queries) when doing checks for existing items but we will probably not get away from the fact that importing is a) infrequent and b) more memory intensive than most site requests.

Once these decisions are made we can close this issue.

[iprunache]

@mattfullerton, you described the issue accurately.

[davidmihalyi]

Understood. We will have to push solving this issue (one way or the other) next year.

For now I would like one round of clean import this week (with extra memory) to use for a last data refresh for the year.
Thanks