Skip to content
This repository has been archived by the owner on Feb 8, 2018. It is now read-only.

Receive package claim link-back #4397

Merged
merged 5 commits into from
Apr 26, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions gratipay/models/package/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

import uuid

from gratipay.models.team import Team
from postgres.orm import Model


Expand All @@ -10,6 +13,14 @@

class Package(Model):
"""Represent a gratipackage. :-)

Packages are entities on open source package managers; `npm
<https://www.npmjs.com/>`_ is the only one we support so far. Each package
on npm has a page on Gratipay with an URL of the form ``/on/npm/foo/``.
Packages can be claimed by Gratipay participants, at which point we create
a :py:class:`~gratipay.models.team.Team` for them under the hood so they
can start accepting payments.

"""

typname = 'packages'
Expand Down Expand Up @@ -40,3 +51,38 @@ def from_names(cls, package_manager, name):
"""
return cls.db.one("SELECT packages.*::packages FROM packages "
"WHERE package_manager=%s and name=%s", (package_manager, name))


@property
def team(self):
"""A computed attribute, the :py:class:`~gratipay.models.team.Team`
linked to this package if there is one, otherwise ``None``. Makes a
database call.
"""
return self._load_team(self.db)


def ensure_team(self, cursor, owner):
"""Given a db cursor and :py:class:`Participant`, insert into ``teams`` if need be.
"""
team = self._load_team(cursor)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an implicit transaction here? What if this method is called concurrently? Seems like you could wind up with multiple entries with different UUIDs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

" ... called concurrently with the same input."

Copy link
Contributor Author

@chadwhitacre chadwhitacre Apr 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If cursors is actually self.db then yes, there is an implicit transaction under here. However, the purpose of exposing the cursor parameter is precisely to allow passing in a proper cursor, so that an entire call chain can operate within one transaction to preserve integrity. In some methods cursor is optional (e.g.), but cursor is required and front-loaded here because integrity is an issue here.

The transaction of which ensure_teams is a part begins in finish_email_verification. If finish_email_verification is called concurrently with the same input, then ... I don't know. I've been working most of the day on trying to sort this out. What I have so far is in 8da4f9c.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the output I'm seeing:

pid-10149 thread-4563636224 (Thread-2) 1492636220.45 loading team
pid-10149 thread-4563636224 (Thread-2) 1492636220.46 old team: n/a
pid-10149 thread-4563636224 (Thread-2) 1492636220.46 slug: 0d8f713b-fc8b-43f8-ad78-638a8d0808c3
pid-10149 thread-4563636224 (Thread-2) 1492636220.47 new team: 0d8f713b-fc8b-43f8-ad78-638a8d0808c3
pid-10149 thread-4563636224 (Thread-2) 1492636220.47 updated, sleeping ...
pid-10149 thread-4563636224 (Thread-2) 1492636222.47 slept
pid-10149 thread-4567842816 (Thread-3) 1492636222.48 loading team
pid-10149 thread-4567842816 (Thread-3) 1492636222.49 old team: n/a
pid-10149 thread-4567842816 (Thread-3) 1492636222.49 slug: ef264dc3-7d62-44d4-a907-ab18c86afa32
pid-10149 thread-4567842816 (Thread-3) 1492636222.49 new team: ef264dc3-7d62-44d4-a907-ab18c86afa32
pid-10149 thread-4567842816 (Thread-3) 1492636222.49 updated, sleeping ...
pid-10149 thread-4567842816 (Thread-3) 1492636224.49 slept
F

================================================== FAILURES ==================================================
______________________ PackageLinking.test_finishing_email_verification_is_thread_safe _______________________
Traceback (most recent call last):
  File "/Users/whit537/personal/gratipay/gratipay.com/tests/py/test_email.py", line 809, in test_finishing_email_verification_is_thread_safe
    assert results[b.ident] == _email.VERIFICATION_FAILED
AssertionError: assert 5 == 1
 +  where 1 = _email.VERIFICATION_FAILED

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks to me like there's something hidden from view that is synchronizing the threads. Even though I start them in parallel, they appear to run in sequence. Hmm ... 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! I think it's because the ... well, wait ...

I was gonna say because the calls synchronize on the cursor.run('DELETE ...') call (why that one instead of the SELECTs? Is psycopg2/Postgres being fancy?), so that the second cursor sees a certain state following the lock release, ... but wouldn't a second insert on teams_to_packages still fail?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we go: team is not None the second time around.

tests/py/test_email.py pid-36042 thread-4463992832 (Thread-2) have cursor, will travel
pid-36042 thread-4463992832 (Thread-2) have record:Record(id=118, address=u'[email protected]', verified=
None, nonce=u'fccf6dec-7559-4cb2-ba77-046f2aa0efe7', verification_start=datetime.datetime(2017, 4, 24, 13, 44, 6, 588594, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)), verification_end=None, participant_id=1L)
pid-36042 thread-4463992832 (Thread-2) have packages:[<gratipay.models.package.Package object at 0x109d09250>]
pid-36042 thread-4463992832 (Thread-2) let's finish up ...
pid-36042 thread-4468199424 (Thread-3) have cursor, will travel
pid-36042 thread-4463992832 (Thread-2) finishing up ...
pid-36042 thread-4463992832 (Thread-2) team: None
pid-36042 thread-4468199424 (Thread-3) have record:Record(id=118, address=u'[email protected]', verified=None, nonce=u'fccf6dec-7559-4cb2-ba77-046f2aa0efe7', verification_start=datetime.datetime(2017, 4, 24, 13, 44, 6, 588594, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)), verification_end=None, participant_id=1L)
pid-36042 thread-4468199424 (Thread-3) have packages:[<gratipay.models.package.Package object at 0x109d12250>]
pid-36042 thread-4463992832 (Thread-2) inserting into teams_to_packages: 45 95
pid-36042 thread-4468199424 (Thread-3) let's finish up ...
pid-36042 thread-4463992832 (Thread-2) inserted into teams
pid-36042 thread-4463992832 (Thread-2) [Record(team_id=45L, package_id=95L)]
pid-36042 thread-4468199424 (Thread-3) finishing up ...
pid-36042 thread-4468199424 (Thread-3) team: 80e34bf6-324a-4b63-a700-c3fe3b624001
diff --git a/gratipay/models/package/__init__.py b/gratipay/models/package/__init__.py
index a0591a1..9a69037 100644
--- a/gratipay/models/package/__init__.py
+++ b/gratipay/models/package/__init__.py
@@ -65,7 +65,10 @@ class Package(Model):
     def ensure_team(self, cursor, owner):
         """Given a db cursor and :py:class:`Participant`, insert into ``teams`` if need be.
         """
+        from aspen import log
+        log('finishing up ...')
         team = self._load_team(cursor)
+        log('team: {}'.format(team.name if team else 'None'))
         if not team:
             slug = str(uuid.uuid4()).lower()
             team = Team.insert( slug=slug
@@ -76,6 +79,7 @@ class Package(Model):
                               , owner=owner
                               , _cursor=cursor
                                )
+            log('inserting into teams_to_packages: {} {}'.format(team.id, self.id))
             cursor.run('INSERT INTO teams_to_packages (team_id, package_id) '
                        'VALUES (%s, %s)', (team.id, self.id))
             from aspen import log
diff --git a/gratipay/models/participant/email.py b/gratipay/models/participant/email.py
index e47cafe..5ae2528 100644
--- a/gratipay/models/participant/email.py
+++ b/gratipay/models/participant/email.py
@@ -228,13 +228,17 @@ class Email(object):
 
 
     def finish_email_verification(self, email, nonce):
+        from aspen import log
         if '' in (email.strip(), nonce.strip()):
             return VERIFICATION_MISSING
         with self.db.get_cursor() as cursor:
+            log('have cursor, will travel')
             record = self.get_email(email, cursor)
+            log('have record:' + str(record))
             if record is None:
                 return VERIFICATION_FAILED
             packages = self.get_packages_claiming(cursor, nonce)
+            log('have packages:' + str(packages))
             if record.verified and not packages:
                 assert record.nonce is None  # and therefore, order of conditions matters
                 return VERIFICATION_REDUNDANT
@@ -243,6 +247,7 @@ class Email(object):
             if (utcnow() - record.verification_start) > EMAIL_HASH_TIMEOUT:
                 return VERIFICATION_EXPIRED
             try:
+                log('let\'s finish up ...')
                 self.finish_package_claims(cursor, nonce, *packages)
                 self.save_email_address(cursor, email)
                 cursor.connection.commit()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gosh. Is there a different possible behavior for each cursor call? 😳

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woo-hoo! 💃 ✊

screen shot 2017-04-24 at 10 07 02 am

tests/py/test_email.py pid-40671 thread-4572102656 (Thread-2) have cursor, will travel
pid-40671 thread-4572102656 (Thread-2) have record:Record(id=121, address=u'[email protected]', verified=None, nonce=u'3193076d-96ce-4864-b927-d5dedb9ff75c', verification_start=datetime.datetime(2017, 4, 24, 14, 6, 58, 165862, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)), verification_end=None, participant_id=1L)
pid-40671 thread-4572102656 (Thread-2) have packages:[<gratipay.models.package.Package object at 0x110421210>]
pid-40671 thread-4572102656 (Thread-2) let's finish up ...
pid-40671 thread-4576309248 (Thread-3) have cursor, will travel
pid-40671 thread-4572102656 (Thread-2) finishing up ...
pid-40671 thread-4572102656 (Thread-2) team: None
pid-40671 thread-4572102656 (Thread-2) inserting into teams_to_packages: 47 98
pid-40671 thread-4572102656 (Thread-2) inserted into teams
pid-40671 thread-4572102656 (Thread-2) [Record(team_id=47L, package_id=98L)]
pid-40671 thread-4576309248 (Thread-3) have record:Record(id=121, address=u'[email protected]', verified=True, nonce=None, verification_start=datetime.datetime(2017, 4, 24, 14, 6, 58, 165862, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)), verification_end=datetime.datetime(2017, 4, 24, 14, 6, 58, 192552, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)), participant_id=1L)
pid-40671 thread-4576309248 (Thread-3) have packages:[]
.

================================ 86 tests deselected by '-kthread_safe' =================================
================================ 1 passed, 86 deselected in 3.54 seconds ================================
diff --git a/gratipay/models/package/__init__.py b/gratipay/models/package/__init__.py
index a0591a1..9a69037 100644
--- a/gratipay/models/package/__init__.py
+++ b/gratipay/models/package/__init__.py
@@ -65,7 +65,10 @@ class Package(Model):
     def ensure_team(self, cursor, owner):
         """Given a db cursor and :py:class:`Participant`, insert into ``teams`` if need be.
         """
+        from aspen import log
+        log('finishing up ...')
         team = self._load_team(cursor)
+        log('team: {}'.format(team.name if team else 'None'))
         if not team:
             slug = str(uuid.uuid4()).lower()
             team = Team.insert( slug=slug
@@ -76,6 +79,7 @@ class Package(Model):
                               , owner=owner
                               , _cursor=cursor
                                )
+            log('inserting into teams_to_packages: {} {}'.format(team.id, self.id))
             cursor.run('INSERT INTO teams_to_packages (team_id, package_id) '
                        'VALUES (%s, %s)', (team.id, self.id))
             from aspen import log
diff --git a/gratipay/models/participant/email.py b/gratipay/models/participant/email.py
index e47cafe..bd6260d 100644
--- a/gratipay/models/participant/email.py
+++ b/gratipay/models/participant/email.py
@@ -228,13 +228,17 @@ class Email(object):
 
 
     def finish_email_verification(self, email, nonce):
+        from aspen import log
         if '' in (email.strip(), nonce.strip()):
             return VERIFICATION_MISSING
         with self.db.get_cursor() as cursor:
-            record = self.get_email(email, cursor)
+            log('have cursor, will travel')
+            record = self.get_email(email, cursor, and_lock=True)
+            log('have record:' + str(record))
             if record is None:
                 return VERIFICATION_FAILED
             packages = self.get_packages_claiming(cursor, nonce)
+            log('have packages:' + str(packages))
             if record.verified and not packages:
                 assert record.nonce is None  # and therefore, order of conditions matters
                 return VERIFICATION_REDUNDANT
@@ -243,6 +247,7 @@ class Email(object):
             if (utcnow() - record.verification_start) > EMAIL_HASH_TIMEOUT:
                 return VERIFICATION_EXPIRED
             try:
+                log('let\'s finish up ...')
                 self.finish_package_claims(cursor, nonce, *packages)
                 self.save_email_address(cursor, email)
                 cursor.connection.commit()
@@ -295,15 +300,19 @@ class Email(object):
                                )
 
 
-    def get_email(self, address, cursor=None):
+    def get_email(self, address, cursor=None, and_lock=False):
         """Return a record for a single email address on file for this participant.
+
+        :param unicode address: the email address for which to get a record
+        :param Cursor cursor: a database cursor; if ``None``, we'll use ``self.db``
+        :param and_lock: if True, we will acquire a write-lock for you on the row
+        :returns: a database record (named tuple)
+
         """
-        return (cursor or self.db).one("""
-            SELECT *
-              FROM emails
-             WHERE participant_id=%s
-               AND address=%s
-        """, (self.id, address))
+        sql = 'SELECT * FROM emails WHERE participant_id=%s AND address=%s'
+        if and_lock:
+            sql += ' FOR UPDATE'
+        return (cursor or self.db).one(sql, (self.id, address))
 
 
     def get_emails(self):

if not team:
slug = str(uuid.uuid4()).lower()
team = Team.insert( slug=slug
, slug_lower=slug
, name=slug
, homepage=''
, product_or_service=''
, owner=owner
, _cursor=cursor
)
cursor.run('INSERT INTO teams_to_packages (team_id, package_id) '
'VALUES (%s, %s)', (team.id, self.id))
return team


def _load_team(self, cursor):
return cursor.one( 'SELECT t.*::teams FROM teams t WHERE t.id='
'(SELECT team_id FROM teams_to_packages tp WHERE tp.package_id=%s)'
, (self.id,)
)
143 changes: 96 additions & 47 deletions gratipay/models/participant/email.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@ def start_email_verification(self, email, *packages):
verification email for an unverified email address.

:param unicode email: the email address to add
:param Package packages: packages to optionally also verify ownership of
:param gratipay.models.package.Package packages: packages to optionally
also verify ownership of

:returns: ``None``

Expand Down Expand Up @@ -201,63 +202,111 @@ def start_package_claims(self, c, nonce, *packages):
)


def update_email(self, email):
"""Set the email address for the participant.
def set_primary_email(self, email, cursor=None):
"""Set the primary email address for the participant.
"""
if not getattr(self.get_email(email), 'verified', False):
raise EmailNotVerified()
username = self.username
with self.db.get_cursor() as c:
self.app.add_event( c
, 'participant'
, dict(id=self.id, action='set', values=dict(primary_email=email))
)
c.run("""
UPDATE participants
SET email_address=%(email)s
WHERE username=%(username)s
""", locals())
if cursor:
self._set_primary_email(email, cursor)
else:
with self.db.get_cursor() as cursor:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the place where I'm talking about the get_cursor call.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is a bit of a goofy construction. liberapay/postgres.py#40 would help.

self._set_primary_email(email, cursor)
self.set_attributes(email_address=email)


def verify_email(self, email, nonce):
if '' in (email, nonce):
def _set_primary_email(self, email, cursor):
if not getattr(self.get_email(email, cursor), 'verified', False):
raise EmailNotVerified()
self.app.add_event( cursor
, 'participant'
, dict(id=self.id, action='set', values=dict(primary_email=email))
)
cursor.run("""
UPDATE participants
SET email_address=%(email)s
WHERE username=%(username)s
""", dict(email=email, username=self.username))


def finish_email_verification(self, email, nonce):
if '' in (email.strip(), nonce.strip()):
return VERIFICATION_MISSING
r = self.get_email(email)
if r is None:
return VERIFICATION_FAILED
if r.verified:
assert r.nonce is None # and therefore, order of conditions matters
return VERIFICATION_REDUNDANT
if not constant_time_compare(r.nonce, nonce):
return VERIFICATION_FAILED
if (utcnow() - r.verification_start) > EMAIL_HASH_TIMEOUT:
return VERIFICATION_EXPIRED
try:
self.db.run("""
UPDATE emails
SET verified=true, verification_end=now(), nonce=NULL
WHERE participant_id=%s
AND address=%s
AND verified IS NULL
""", (self.id, email))
except IntegrityError:
return VERIFICATION_STYMIED
with self.db.get_cursor() as cursor:
record = self.get_email(email, cursor, and_lock=True)
if record is None:
return VERIFICATION_FAILED
packages = self.get_packages_claiming(cursor, nonce)
if record.verified and not packages:
assert record.nonce is None # and therefore, order of conditions matters
return VERIFICATION_REDUNDANT
if not constant_time_compare(record.nonce, nonce):
return VERIFICATION_FAILED
if (utcnow() - record.verification_start) > EMAIL_HASH_TIMEOUT:
return VERIFICATION_EXPIRED
try:
self.finish_package_claims(cursor, nonce, *packages)
self.save_email_address(cursor, email)
except IntegrityError:
return VERIFICATION_STYMIED
return VERIFICATION_SUCCEEDED


def get_packages_claiming(self, cursor, nonce):
"""Given a nonce, return :py:class:`~gratipay.models.package.Package`
objects associated with it.
"""
return cursor.all("""
SELECT p.*::packages
FROM packages p
JOIN claims c
ON p.id = c.package_id
WHERE c.nonce=%s
""", (nonce,))


def save_email_address(self, cursor, address):
"""Given an email address, modify the database.
"""
cursor.run("""
UPDATE emails
SET verified=true, verification_end=now(), nonce=NULL
WHERE participant_id=%s
AND address=%s
AND verified IS NULL
""", (self.id, address))
if not self.email_address:
self.update_email(email)
return VERIFICATION_SUCCEEDED
self.set_primary_email(address, cursor)


def finish_package_claims(self, cursor, nonce, *packages):
"""Create teams if needed and associate them with the packages.
"""
cursor.run('DELETE FROM claims WHERE nonce=%s', (nonce,))
package_ids = []
for package in packages:
package.ensure_team(cursor, self)
package_ids.append(package.id)
self.app.add_event( cursor
, 'participant'
, dict( id=self.id
, action='finish-claim'
, values=dict(package_ids=package_ids)
)
)


def get_email(self, email):
def get_email(self, address, cursor=None, and_lock=False):
"""Return a record for a single email address on file for this participant.

:param unicode address: the email address for which to get a record
:param Cursor cursor: a database cursor; if ``None``, we'll use ``self.db``
:param and_lock: if True, we will acquire a write-lock on the email record before returning
:returns: a database record (a named tuple)

"""
return self.db.one("""
SELECT *
FROM emails
WHERE participant_id=%s
AND address=%s
""", (self.id, email))
sql = 'SELECT * FROM emails WHERE participant_id=%s AND address=%s'
if and_lock:
sql += ' FOR UPDATE'
return (cursor or self.db).one(sql, (self.id, address))


def get_emails(self):
Expand Down
6 changes: 4 additions & 2 deletions gratipay/models/team/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from .available import Available
from .closing import Closing
from .membership import Membership
from .package import Package
from .takes import Takes
from .tip_migration import TipMigration

Expand All @@ -36,7 +37,7 @@ def slugize(name):
return slug


class Team(Model, Available, Closing, Membership, Takes, TipMigration):
class Team(Model, Available, Closing, Membership, Package, Takes, TipMigration):
"""Represent a Gratipay team.
"""

Expand Down Expand Up @@ -98,9 +99,10 @@ def _from_thing(cls, thing, value):

@classmethod
def insert(cls, owner, **fields):
cursor = fields.pop('_cursor') if '_cursor' in fields else None
fields['slug_lower'] = fields['slug'].lower()
fields['owner'] = owner.username
return cls.db.one("""
return (cursor or cls.db).one("""
INSERT INTO teams
(slug, slug_lower, name, homepage,
Expand Down
22 changes: 22 additions & 0 deletions gratipay/models/team/package.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals


class Package(object):
"""A :py:class:`~gratipay.models.team.Team` can be associated with :py:class:`Package`.
"""

@property
def package(self):
"""A computed attribute, the
:py:class:`~gratipay.models.package.Package` linked to this team if
there is one, otherwise ``None``. Makes a database call.
"""
return self._load_package(self.db)


def _load_package(self, cursor):
return cursor.one( 'SELECT p.*::packages FROM packages p WHERE p.id='
'(SELECT package_id FROM teams_to_packages tp WHERE tp.team_id=%s)'
, (self.id,)
)
5 changes: 5 additions & 0 deletions sql/branch.sql
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,9 @@ BEGIN;
, UNIQUE(nonce, package_id)
);

CREATE TABLE teams_to_packages
( team_id bigint UNIQUE REFERENCES teams(id) ON DELETE RESTRICT
, package_id bigint UNIQUE REFERENCES packages(id) ON DELETE RESTRICT
);

END;
Loading