Skip to content
This repository has been archived by the owner on Feb 8, 2018. It is now read-only.

Rewrite of the dashboard reconciliation #3975

Merged
merged 11 commits into from
Nov 23, 2016
Merged

Rewrite of the dashboard reconciliation #3975

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8096f29
Rewrite of the reconciliation for the finance
kaguillera Mar 31, 2016
a1d4d8a
Resolving rebase again
kaguillera Sep 9, 2016
a639c27
Trim trailing whitespace
chadwhitacre Apr 13, 2016
7acc5be
Reformat SQL query
chadwhitacre Apr 13, 2016
0410b2d
Rewrite of the reconciliation for the finance
kaguillera Mar 31, 2016
d52f936
Altered a exchanges and exchane_route to handle NULL values fir recon…
kaguillera Apr 15, 2016
ecfa273
Move new sptfile over the old one
chadwhitacre Apr 13, 2016
0843b82
Rewrite of the reconciliation for the finance
kaguillera Mar 31, 2016
0eb41a4
Removed and unnecessary file.
kaguillera Sep 22, 2016
bac13de
Minor cleanups
chadwhitacre Nov 23, 2016
fea4533
More comment
chadwhitacre Nov 23, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions sql/branch.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
-- Alter the enums to cater for missing data.
ALTER TYPE payment_net ADD VALUE 'unknown';
ALTER TYPE exchange_status ADD VALUE 'unknown';

-- Update the field status in the exchanges table from NULL to 'unknown'
UPDATE exchanges SET status = 'unknown' WHERE status IS NULL;

-- Alter the exchanges table to ensure that no more NULL values are entered
ALTER TABLE exchanges ALTER COLUMN status SET NOT NULL;

-- Insert records for ‘unknown’ (previously NULL in exchanges table
-- network in exchange_route table
INSERT INTO exchange_routes (participant, network, address, error)
(
SELECT DISTINCT participants.id, 'unknown'::payment_net, 'None', 'None'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Goofy to use the string "None" here. I think the empty string might be a better choice?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a uniqueness constraint on (participant, network, address), which is why non-unique (network, address) is fine.

gratipay-bak=# \d exchange_routes
                                Table "public.exchange_routes"
┌─────────────┬───────────────┬──────────────────────────────────────────────────────────────┐
│   Column    │     Type      │                          Modifiers                           │
├─────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ id          │ integer       │ not null default nextval('exchange_routes_id_seq'::regclass) │
│ participant │ bigint        │ not null                                                     │
│ network     │ payment_net   │ not null                                                     │
│ address     │ text          │ not null                                                     │
│ error       │ text          │ not null                                                     │
│ fee_cap     │ numeric(35,2) │                                                              │
└─────────────┴───────────────┴──────────────────────────────────────────────────────────────┘
Indexes:
    "exchange_routes_pkey" PRIMARY KEY, btree (id)
    "exchange_routes_participant_network_address_key" UNIQUE CONSTRAINT, btree (participant, network, address)
Check constraints:
    "exchange_routes_address_check" CHECK (address <> ''::text)
Foreign-key constraints:
    "exchange_routes_participant_fkey" FOREIGN KEY (participant) REFERENCES participants(id)
Referenced by:
    TABLE "exchanges" CONSTRAINT "exchanges_route_fkey" FOREIGN KEY (route) REFERENCES exchange_routes(id)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Soooo...if we replace the None for address and error will that be sufficient or do you still have some concerns

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, let's switch to n/a for address and the empty string for error.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in bac13de.

FROM exchanges, participants
WHERE exchanges.participant = participants.username
AND route IS NULL
);

-- Update exchanges records with exchange_route ids pointing to ‘unknown’ network records for that participants
UPDATE exchanges
SET route = exchange_routes.id
FROM exchange_routes, participants
WHERE exchange_routes.participant = participants.id
AND participants.username = exchanges.participant;

-- Alter exchanges table and set route to not null
ALTER TABLE exchanges ALTER COLUMN route SET NOT NULL;
188 changes: 122 additions & 66 deletions www/dashboard/reconciliation.spt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,46 +5,95 @@ from decimal import Decimal as D
[---]
_by_month = website.db.all("""

SELECT date_trunc('month', "timestamp") as month
, sum(case when amount > 0 then amount end) as payins
, -(sum(case when amount < 0 then amount end)) as payouts
, sum(fee) as income

FROM exchanges
GROUP BY month
ORDER BY month ASC
SELECT date_trunc('month', "timestamp") AS month
, COALESCE(sum(case when amount > 0 then amount end), 0) as payins
, COALESCE(-(sum(case when amount < 0 then amount end)), 0) as payouts
, COALESCE(sum(case when amount > 0 then fee end), 0) as payins_income
, COALESCE(sum(case when amount < 0 then fee end), 0) as payouts_income
, status
, network
FROM exchanges, exchange_routes
WHERE exchanges.route = exchange_routes.id
GROUP BY month, network, status
ORDER BY month ASC;

""")

def by_month():
balance = D(0)
for month, payins, payouts, income in _by_month:
if payins is None:
payins = D(0)
if payouts is None:
payouts = D(0)
by_month = {}
for month, payins, payouts, payins_income, payouts_income, status, network in _by_month:
month = str(month)[:7]
if month not in by_month.keys():
by_month[month] = { network: { 's_payin': 0, 'p_payin': 0, 'f_payin': 0, \
'u_payin': 0, 'total_payin': 0, 'payin_income': 0, \
's_payout': 0, 'p_payout': 0, 'f_payout': 0,\
'u_payout': 0, 'total_payout': 0, 'payout_income': 0 } }
if network not in by_month[month].keys():
by_month[month][network] = { 's_payin': 0, 'p_payin': 0, 'f_payin': 0, \
'u_payin': 0, 'total_payin': 0, 'payin_income': 0, \
's_payout': 0, 'p_payout': 0, 'f_payout': 0,\
'u_payout': 0,'total_payout': 0, 'payout_income': 0 }
if status == 'succeeded':
by_month[month][network]['s_payin'] = payins
by_month[month][network]['s_payout'] = payouts
by_month[month][network]['payin_income'] = payins_income
by_month[month][network]['payout_income'] = payouts_income
if status == 'pending':
by_month[month][network]['p_payin'] = payins
by_month[month][network]['p_payout'] = payouts
if status == 'failed':
by_month[month][network]['f_payin'] = payins
by_month[month][network]['f_payout'] = payouts
if status == 'unknown':
by_month[month][network]['u_payin'] = payins
by_month[month][network]['u_payout'] = payouts
by_month[month][network]['total_payin'] += payins
by_month[month][network]['total_payout'] += payouts

balance = balance + payins - payouts
yield month, payins, payouts, balance, income
by_month = by_month()

if user.ADMIN:
by_day = website.db.all("""
SELECT date_trunc('day', timestamp)::date as day
, sum(case when amount > 0 then amount + fee else 0 end) as charge_amount
, sum(case when amount < 0 then -(amount - fee) else 0 end) as payout_amount
, sum(amount) as liability_delta
, sum(fee) as income
return by_month

FROM exchanges
GROUP BY day
ORDER BY day ASC;
""")

def by_month_csv(by_month):
output = [ ["Month", "Network", "Succeeded Payins", "Pending Payins",
"Failed Payins", "Unknown Payin", "Total Payins", "Payin Fee Income",
"Succeeded Payouts", "Pending Payouts", "Failed Payouts",
"Unknown Payout", "Total Payouts", "Payout Fee Income"] ]
for month, recs in sorted(by_month.items()):
for network,transfers in recs.items():
row = []
row.append( str(month)[:7] )
row.append( network )
row.append( transfers['s_payin'] )
row.append( transfers['p_payin'] )
row.append( transfers['f_payin'] )
row.append( transfers['u_payin'] )
row.append( transfers['total_payin'] )
row.append( transfers['payin_income'] )
row.append( transfers['s_payout'] )
row.append( transfers['p_payout'] )
row.append( transfers['f_payout'] )
row.append( transfers['u_payout'] )
row.append( transfers['total_payout'] )
row.append( transfers['payout_income'] )
output.append(row)
return output

by_month = by_month()
by_month_csv = by_month_csv(by_month)
fmt = lambda x: "{:,.2f}".format(x)

[---] text/csv via csv_dump
by_month_csv

[---] application/json via json_dump
by_month

[---] text/html
<html>
<head>
<title>Escrow Reconciliation Report</title>
<title>Escrow Reconciliation Report by Month and Network</title>
<style>
body {
font-family: monospace;
Expand All @@ -58,6 +107,7 @@ fmt = lambda x: "{:,.2f}".format(x)
th {
text-align: left;
font-family: monospace;
padding-right: 30px;
}
td {
text-align: right;
Expand All @@ -73,45 +123,51 @@ fmt = lambda x: "{:,.2f}".format(x)
process.</p>

<h2>Escrow and Income by Month</h2>
<table>
<tr>
<th>Month</th>
<th>Payins ($)</th>
<th>Payouts ($)</th>
<th>Balance ($)</th>
<th>Fee Income ($)</th>
</tr>
{% for month, payins, payouts, balance, fees in by_month %}
<tr>
<td>{{ str(month)[:7] }}</td>
<td>{{ fmt(payins) }}</td>
<td>{{ fmt(payouts) }}</td>
<td>{{ fmt(balance) }}</td>
<td>{{ fmt(fees) }}</td>
</tr>
{% for month, recs in by_month|dictsort %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Is this a Jinja2 thing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

<h3>{{ str(month)[:7] }}</h3>
<table>
<tr>
<th rowspan="2">Network</th>
<th colspan="5">Payins ($)</th>
<th>&nbsp;</th>
<th colspan="5">Payouts ($)</th>
</tr>
<tr>
<th>Succeeded</th>
<th>Pending</th>
<th>Failed</th>
<th>Unknown</th>
<th>Total</th>
<th>Fee Income</th>
<th>&nbsp;</th>
<th>Succeeded</th>
<th>Pending</th>
<th>Failed</th>
<th>Unknown</th>
<th>Total</th>
<th>Fee Income</th>
</tr>
{% for network, transfers in recs.items() %}
<tr>
<td>{{ network }}</td>
<td>{{ fmt( transfers['s_payin'] ) }}</td>
<td>{{ fmt( transfers['p_payin'] ) }}</td>
<td>{{ fmt( transfers['f_payin'] ) }}</td>
<td>{{ fmt( transfers['u_payin'] ) }}</td>
<td>{{ fmt( transfers['total_payin'] ) }}</td>
<td>{{ fmt( transfers['payin_income'] ) }}</td>
<td>&nbsp;</td>
<td>{{ fmt( transfers['s_payout'] ) }}</td>
<td>{{ fmt( transfers['p_payout'] ) }}</td>
<td>{{ fmt( transfers['f_payout'] ) }}</td>
<td>{{ fmt( transfers['u_payout'] ) }}</td>
<td>{{ fmt( transfers['total_payout'] ) }}</td>
<td>{{ fmt( transfers['payout_income'] ) }}</td>
</tr>
{% endfor %}
<tr><td colspan="14"><hr></td></tr>
</table>
{% endfor %}
</table>

{% if user.ADMIN %}
<h2>By Day</h2>
<table>
<tr>
<th>Date</th>
<th>Charge Amount ($)</th>
<th>Payout Amount ($)</th>
<th>Fee Income ($)</th>
<th>Liability &Delta; ($)</th>
</tr>
{% for day, charge_amount, payout_amount, liability_delta, fee in by_day %}
<tr>
<td>{{ day }}</td>
<td>{{ fmt(charge_amount) }}</td>
<td>{{ fmt(payout_amount) }}</td>
<td>{{ fmt(fee) }}</td>
<td>{{ fmt(liability_delta) }}</td>
</tr>
{% endfor %}
</table>
{% endif %}
</body>
</html>