Instructions for "Updating activities after changing import code" in readme don't work

[odscjames]

Describe the bug

While working on #296 I followed the instructions in the readme "Updating activities after changing import code" and they don't work

To Reproduce

• Check out my branch
• Follow instructions
• Wait
• On DB run

    select count(*) from activity where raw_json is null;
    select count(*) from activity where raw_json is not null;

The first one will be a non-zero number

Expected behaviour

"select count(*) from activity where raw_json is null;" will say 0

Screenshots

n/a

Additional context

n/a

ps. It is possible something in my changes to the import code is not right ....

[andylolz]

Okay so I think the problem here is that I updated the crawler code, but failed to update the related docs 😬 Sorry about that.

it looks like that bit of the README was written 7 years ago (!) Rolling back to then, it was certainly correct – if last_succ was None, the dataset would be queued for parsing:

iati-datastore/iati_datastore/iatilib/crawler.py

Lines 320 to 323 in 4b1851b

	need_update = [r for r in dataset.resources
	if not r.last_succ or r.last_succ < dataset.last_modified]
	for resource in need_update:
	rq.enqueue(update_resource, args=(resource.url,), result_ttl=0)

However, it’s definitely not true now. I think there are now two options: You can either run UPDATE resource SET last_parsed = NULL;, or alternatively, trigger an import with:

iati crawler download-and-update --ignore-hashes

ignore-hashes should parse everything regardless of whether the hashed data appears unchanged.

I’ve updated the README accordingly in 79fcb30.