run Gittip 101

[chadwhitacre]

100

[chadwhitacre]

Reviewed 62 accounts. Left 3 unreviewed (see #2341).

[chadwhitacre]

Backup taken and verified.

[chadwhitacre]

Droplet spun up and updated.

[chadwhitacre]

Script started.

[chadwhitacre]

Traceback (most recent call last):
  File "./env/bin/payday", line 9, in <module>
    load_entry_point('gittip==10.2.0-dev', 'console_scripts', 'payday')()
  File "/home/whit537/www.gittip.com/gittip/cli.py", line 17, in payday
    db = wireup.db()
  File "/home/whit537/www.gittip.com/gittip/wireup.py", line 33, in db
    psycopg2.extras.register_hstore(cursor, globally=True, unicode=True)
  File "/home/whit537/www.gittip.com/env/local/lib/python2.7/site-packages/psycopg2/extras.py", line 775, in register_hstore
    "hstore type not found in the database. "
psycopg2.ProgrammingError: hstore type not found in the database. please install it from your 'contrib/hstore.sql' file

[chadwhitacre]

I updated the droplet improperly. I forgot to pull before making.

[chadwhitacre]

Script restarted.

[chadwhitacre]

pid-1657 thread-140589118981888 (MainThread) Traceback (most recent call last):
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/cli.py", line 32, in payday
pid-1657 thread-140589118981888 (MainThread)     Payday(db).run()
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 140, in run
pid-1657 thread-140589118981888 (MainThread)     self.pachinko(ts_start, self.genparticipants(ts_start, ts_start))
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 264, in pachinko
pid-1657 thread-140589118981888 (MainThread)     tip(take['member'], amount)
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 258, in tip
pid-1657 thread-140589118981888 (MainThread)     , pachinko=True
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 424, in tip
pid-1657 thread-140589118981888 (MainThread)     tip['amount'], pachinko=pachinko):
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 456, in transfer
pid-1657 thread-140589118981888 (MainThread)     self.credit_participant(cursor, tippee, amount)
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 507, in credit_participant
pid-1657 thread-140589118981888 (MainThread)     assert rec is not None, (participant, amount)  # sanity check
pid-1657 thread-140589118981888 (MainThread) AssertionError: (u'aaronwinborn', Decimal('0.01'))

[chadwhitacre]

pending is NULL for @aaronwinborn.

[chadwhitacre]

Why is pending NULL?

[chadwhitacre]

Pending is NULL for 43972 participants.

[chadwhitacre]

There are 73659 participants total.

[chadwhitacre]

Maybe @aaronwinborn was added to the Drupal team during payday?

[chadwhitacre]

That's the sort of thing that happened the last time we had issues around pending, way back (will have to dig for that).

[chadwhitacre]

ctime is 2014-05-02 05:57:04.323376+00. Is that during last week's payday?

[chadwhitacre]

Last payday (#2340) was 2014-05-01.

[chadwhitacre]

The only place where pending is explicitly set to NULL is in clear_pending_to_balance, which happens right after pachinko.

[chadwhitacre]

But I seem to recall that NULL + 0 results in a NULL, and that we've had that happen before.

[chadwhitacre]

=> select (0 + null);                                                                     ┌──────────┐
│ ?column? │
├──────────┤
│        ¤ │
└──────────┘
(1 row)

[chadwhitacre]

Sanity check:

=> select (null + 0);
┌──────────┐
│ ?column? │
├──────────┤
│        ¤ │
└──────────┘
(1 row)

[Changaco]

Why is pending allowed to be NULL? 0 seems like a sane default to me.

[chadwhitacre]

@Changaco We use NULL as a sentinel value during payday.

[chadwhitacre]

Okay! @aaronwinborn hasn't joined Gittip. :-)

[chadwhitacre]

That is, https://www.gittip.com/aaronwinborn/ is redirecting to https://www.gittip.com/on/github/aaronwinborn/.

[chadwhitacre]

So the question is, how is aaronwinborn part of the DrupalMedia team without being a member of the site?

[chadwhitacre]

Okay! So, aaronwinborn's participant record was created at 2012-07-11 14:48:30.679927+00, which is before we started using randomized usernames for stub participants. The bug is that stub participants can be added as members of a team. A quick query of username and ctime shows that 2012-09-13 is the cutoff. That was the date we started using randomized usernames for stub participants.

Precluding adding stub participants as team members ticketed as #2364.

Randomizing usernames for old stub participants reticketed as #2365.

[chadwhitacre]

So what's the fix here? We should delete from takes where the user is a stub participant.

[chadwhitacre]

That's out of scope here, anyway. That's not a bug we need to fix or repair here.

[chadwhitacre]

We've so far identified two causes of duplicate transfers:

• A participant takes from a team that also gives to them.
• Payday crashed and we reran it with a bug.

Does that account for all of the duplicates we're seeing?

[chadwhitacre]

Let's assume for the moment that all of the duplicate transfers before last payday are due to taking from a team that gives to you. Let's look at just the duplicates from last payday. Let's assume there are up to three amongst the 66 that are also legitimate. So we have 63-66 bad duplicates. Here's the question (IRC): What are the dollar amounts involved?

Turns out there are only five duplicates that are $10 or more. They're all on the Gittip team. The sixth-highest (at $4) is also from Gittip. Here's a breakdown:

41   0.01
20   0.02 -  9.99
 5  10.00 - 50.00

There are 23 teams affected by this. It's really only the Gittip team where the dollar amounts are at all problematic. One option is to simply say that this was a mistake and let the teams eat it. Right now the teams are eating it. Some team members received more than their fair share. Except for Gittip, the amounts are so low and the teams are so barely active that it's really not a big deal.

Since we are Gittip :-) we can ask ourselves: are we okay with 12 of our members getting a bonus last week? Here are the people in question and the amounts of their bonuses:

=> select tippee, array_agg[1] from bar where count > 1 and date = '2014-05-08'::date and tipper='Gittip' order by tipper, array_agg;
┌─────────────┬───────────┐
│   tippee    │ array_agg │
├─────────────┼───────────┤
│ thefoxis    │      0.01 │
│ mtrythall   │      0.01 │
│ Aaron1011   │      0.01 │
│ galuszkak   │      0.02 │
│ dominic     │      1.00 │
│ pjf         │      2.00 │
│ ESWAT       │      4.00 │
│ sim6        │     10.00 │
│ duckinator  │     16.00 │
│ seanlinsley │     18.01 │
│ Changaco    │     20.00 │
│ patcon      │     50.00 │
└─────────────┴───────────┘
(12 rows)

Personally I'm fine giving these folks a bonus. I think that makes more sense than spending more time trying to reverse this. In which case, we can call it case closed on the assessment and db repair, and move on to fixing the bugs that landed us here in the first place. Any objections?

[chadwhitacre]

I'm going to proceed on that basis.

[chadwhitacre]

Object if you want to! :-)

[patcon]

Cool. I'd like to do a one-time split with @rummik, if that's cool. I've just adjusted Kim's amount accordingly, and I'll adjust back after payday :)

[chadwhitacre]

Now I'm looking at this constrained by as_team_member. I know we added that fairly recently. I don't know if we tried to set it to true for transfers in the past. It's at least useful for this week.

I'm writing up a blog post about this, somewhere to point people if/when they notice.

select id, timestamp::date as date, tipper, tippee, amount 
into temp table foo from transfers where as_team_member;

select date, tipper, tippee, count(id), array_agg(amount)
into temp bar from foo group by date, tipper, tippee order by count;

select * from bar where count > 1 order by date;

select date, count(date) as nduplicates from bar
where count > 1 group by date order by date;

[chadwhitacre]

Awesome, thanks @patcon! :-)

[chadwhitacre]

=> select tipper, sum(array_agg[1]) from bar where count > 1 group by tipper order by sum; 
┌────────────────┬────────┐
│     tipper     │  sum   │
├────────────────┼────────┤
│ 3v4l.org       │   0.01 │
│ thechangelog   │   0.01 │
│ saidozcan      │   0.01 │
│ BrightworkCoRe │   0.01 │
│ NuvolaPlayer   │   0.01 │
│ goodcode       │   0.01 │
│ esdiscuss      │   0.01 │
│ by_codebar     │   0.01 │
│ promisejs.org  │   0.01 │
│ meteorjs       │   0.01 │
│ teampopong     │   0.02 │
│ GetStorage     │   0.02 │
│ Innov_Africa   │   0.02 │
│ Bountysource   │   0.04 │
│ IPvoices       │   0.04 │
│ Coackme        │   0.09 │
│ Shields        │   1.58 │
│ Aspen          │   3.50 │
│ UnissonCo      │   4.04 │
│ rubygems       │  10.83 │
│ Gittip         │ 143.05 │
└────────────────┴────────┘
(21 rows)

[chadwhitacre]

That's $163.33 total.

[chadwhitacre]

Not sure why my result differs from @Changaco's: #2362 (comment).

[chadwhitacre]

The array_agg[1] is not stable. I was pulling @seanlinsley's 40.00 transfer instead of the 18.01 one.

[chadwhitacre]

Now I'm at 141.34, which fits with @Changaco's earlier result.

[chadwhitacre]

Now to determine which teams never got paid at all.

[chadwhitacre]

This would be any teams after the first failure.

[chadwhitacre]

Actually, any later members of the team we were processing when we crashed would also have not gotten a payout.

[chadwhitacre]

Using select * from takes where team='DrupalMedia' order by ctime desc; on the backup (where I can still see aaronwinborn), there are two affected users on the DrupalMedia team.

[chadwhitacre]

This is also inferrable from their Members page:

[chadwhitacre]

Okay, so how many teams are after the DrupalMedia team?

[chadwhitacre]

=> select count(id) from participants where claimed_time >= \
    (select claimed_time from participants where username='DrupalMedia');
┌───────┐
│ count │
├───────┤
│   561 │
└───────┘
(1 row)

[chadwhitacre]

=> select username from participants p join takes t on p.username = t.team 
where claimed_time >= (select claimed_time from participants where username='DrupalMedia') 
and number='plural' group by username;
┌───────────────┐
│   username    │
├───────────────┤
│ DrupalMedia   │
│ PerlDancer    │
│ PhongRobotics │
└───────────────┘
(3 rows)

[chadwhitacre]

Blog post published: https://twitter.com/Gittip/status/466684829040656387.

[chadwhitacre]

Case closed.