Connector to pull Plaid financial data into the Firefly finance tool.
Inspired by firefly-plaid-connector.
Please note this version of the connector requires at least version v6.1.2 of Firefly.
These are basic instructions for installing and running the connector. See further topics below for more details.
- Ensure you have a JRE or JDK for at least Java 17.
- Download the latest JAR from the releases page.
- Move the JAR to your desired working directory.
- Make a
persistence/
subdirectory in your working directory for the connector to persist data to that's writeable by the user running the connector. - Set up a configuration file. I suggest copying the existing
application.yml
and modifying as needed. See below for details on configuration. - Run the connector, for instance with
java -jar connector.jar --spring.config.location=application.yml
New versions of the Docker image are pushed to GHCR with each release.
The latest version is available at ghcr.io/dvankley/firefly-plaid-connector-2:latest
.
- Pull down the docker-compose-polled.yml and/or
docker-compose-batch.yml files.
- Ensure you copy the raw file with the exact same whitespace, otherwise you'll have issues. YAML is very picky about whitespace.
- The two different compose files correspond to different run modes and contain suggested configuration for a given mode.
- Set the
HOST_APPLICATION_CONFIG_FILE_LOCATION
environment variable to point to your config file (or just manually insert your path in the compose file). - If running in polled mode (and thus using
docker-compose-polled.yml
), create a directory on your host machine that is writable by anyone, then pass its location in asHOST_PERSISTENCE_DIRECTORY_LOCATION
.- This directory will store the polled mode cursor file, which is basically the polled mode's last synced state.
- Using a bind mount for this is kind of crappy, but I couldn't find a good way to make the Spring Boot Gradle bootBuildImage plugin set perms on a named volume cleanly. Suggestions welcome.
- Run
docker compose up
.
The application container requires the following:
- An application configuration file to read from.
2. See below for details on configuration.
2. The
SPRING_CONFIG_LOCATION
environment variable can be used to set the location (in the container) of the application configuration file. - A directory to write the sync cursor file to (if run in polled mode).
3. The
FIREFLYPLAIDCONNECTOR2_POLLED_CURSORFILEDIRECTORYPATH
environment variable can be used to set the location (in the container) of the directory that will contain the sync cursor file.
The example below uses bind mounts for these purposes for
simplicity, but you can also use volumes if you want.
If using volumes you will need to use an intermediate container
to write the application configuration file to a volume, as well as setting permissions to allow the
application user cnb
to write to the cursor file directory's volume.
docker run \
--mount type=bind,source=/host/machine/application/config/file/directory,destination=/opt/fpc-config,readonly \
--mount type=bind,source=/host/machine/writeable/directory,destination=/opt/fpc-cursors \
-e SPRING_CONFIG_LOCATION=/opt/fpc-config/application.yml \
-e FIREFLYPLAIDCONNECTOR2_POLLED_CURSORFILEDIRECTORYPATH=/opt/fpc-cursors \
-t firefly-plaid-connector-2
The connector can be run in either batch
or polled
mode.
Uses the Plaid transactions get endpoint to pull historical Plaid transaction data, convert it to Firefly transactions, and write them to Firefly.
This is typically used when you're first setting up your Firefly instance and you want to backfill existing transactions
before running indefinitely in polled
mode.
Batch
mode can be memory intensive if you're pulling a large volume of transactions, so if your Firefly server is
running on a low-spec VPS (like mine), then I recommend running your large batch
mode pull on a higher spec machine
(like your home computer or whatever). It can still be pointed at your existing Firefly server with no problem.
Uses the Plaid sync endpoint to periodically pull down transaction data, convert it to Firefly transactions, and write them to Firefly.
When started in this mode, the connector will not pull any past transactions, but will check for new transactions every
fireflyPlaidConnector2.polled.syncFrequencyMinutes
.
The last sync position of each account will be stored in persistence/plaid_sync_cursors.txt
.
The most complex part of the connector by far is the transfer matching logic. Firefly has a notion of transfers, which are special transactions that instead of the usual flow of money from an asset account into an expense account or from a revenue account into an asset account, represent the flow of money from one asset account into another.
For me this typically happens when I'm making payments on credit cards or the like (because I have credit cards set up as asset accounts instead of liability accounts because that's how Firefly works).
Plaid has no notion of inter-account transactions; transaction data only references the account that it's on.
We try to bridge these by matching two Plaid transactions on different asset accounts and converting them into a single
Firefly transfer. You can see the full logic in TransactionConverter.sortByPairs
, but the summary is that the
connector searches for pairs of Plaid transfer-type transactions with inverse amounts (i.e. $100 and -$100)
that are within fireflyPlaidConnector2.transferMatchWindowDays
of each other. This isn't flawless, but
is sufficient in most cases.
Note that if the connector attempts to convert an existing non-transfer Firefly transaction and an incoming Plaid transaction to a Firefly transfer, the existing Firefly transaction will be deleted and the new transfer transaction will be created because the Firefly API does not support converting existing transaction types.
If fireflyPlaidConnector2.batch.setInitialBalance
is set to true
, the connector will try to create "initial balance"
transactions for each Firefly account that result in the current Firefly balance for each account equalling the
current balance that Plaid reports for that account.
This is determined by summing up all transactions pulled during
this run for that account, then subtracting that from the Plaid-reported current balance to get the amount for the
"initial balance" transaction.
For Firefly asset accounts that are of the "credit card" type, the Plaid balance is interpreted as a negative rather than a positive value. This is because Plaid always reports balances as positive (as far as I can tell), and it's assumed that a Firefly account set as a credit card is linked to a Plaid account that's also a credit card.
Ensure that the current balance in the target Firefly accounts are 0 before using this feature.
Because the initial balance transaction amount is determined from the transactions pulled in this batch, I do not recommend enabling this feature unless you're pulling all of the transactions you intend to backfill for a given account (as opposed to just filling in some gaps).
Note that the Plaid balance endpoint (or the underlying institution data source) is kind of crappy. To wit:
- There is a
lastUpdatedDatetime
field in the response schema, but according to the Plaid documentation it's only populated for Capital One for whatever reason, and due to that it's impossible to tell exactly what point in time the Plaid balance represents.- The upshot of this is that if the Plaid balance is out of date by a transaction or two, the "initial balance" transaction amount will also be a bit off. Plan accordingly.
- According to the docs, the Plaid balance endpoint forces a synchronous update from the underlying institution data source, which can cause this part of the connector's run to take a while. The Plaid API timeout has been adjusted to account for this, but it still fails with surprising regularity.
As you have probably guessed from the name, Plaid is an important part of using this connector. To use Plaid, you will need an account. As of this writing, you can sign up for a developer account for free, which should be sufficient for the purposes of this connector.
Once you've signed up for Plaid, you should be provided a client id and secret, which go in the application config
file where you'd expect (fireflyPlaidConnector2.plaid.clientId
and fireflyPlaidConnector2.plaid.secret
).
Next up, you need to connect Plaid to your various financial institutions. The easiest way to do this is to run
Plaid Quickstart locally. I found the Docker experience to be fairly painless.
I recommend copying the .env.example
file to a new .env
file, filling in your credentials, setting PLAID_ENV
to
development
(unless you're using production
for some reason), and setting PLAID_PRODUCTS
to transactions
. Note that
if you leave PLAID_PRODUCTS
set to the default auth,transactions
, you won't be able to connect to some of the
financial institutions you might expect because they don't support the auth
product.
Once you have Quickstart running, just follow the UI prompts to connect to your financial institutions.
For each institution you connect to, Plaid should give you an item_id
and an access_token
. Make a note of both.
Each institution can contain multiple accounts (i.e. your bank has both your savings and checking account), so you
will need to make an additional call to the Plaid API to get individual account ids. Example callout using httpie
:
http POST https://development.plaid.com/accounts/get \
client_id=yourclientid \
secret=yoursecret \
access_token=access-development-your-items-access-token
This will give you a list of account ids that belong to that item/financial institution. Make a note of these, as you will need to enter them into the connector's configuration file.
The configuration file is unsurprisingly where most of the configuration of the connector takes place.
I recommend copying the application.yml
file
and customizing it to your preferences. Each property should be well documented in the comments of that file.
This is my workflow for using the connector with Firefly. Of course, you don't have to use it exactly like this, but you may find this useful for reference in thinking through your own workflow.
- Set up your Plaid developer account, connect to your various financial institutions, and set up the connector application configuration file as described in Configuration.
- Plan your category mapping
- Determine what Firefly budgets
and categories you want to use.
- Note the functional difference between the two.
- In my case, as recommended by the Firefly documentation, we used budgets for expenses that are optional and should have a limit (or target) amount each month. We used categories for expenses that are not optional or should not have a limited amount each month.
- You may want to just put everything in budgets (and not set amounts for categories that don't need them) as it makes the charts and reports a bit easier to read.
- Determine the mapping between your Firefly budgets/categories and Plaid categories.
- The Plaid category taxonomy includes "primary" categories and "detailed" subcategories.
- See the official reference CSV
- Or if you prefer, see the connector's parsed category list.
- Assign Plaid primary or detailed categories to your Firefly budgets and categories.
- Each Plaid primary or detailed category may only be assigned to one Firefly budget or category.
- Multiple different Plaid primary or detailed categories may be assigned to a single Firefly budget or category.
- For example, you might assign both
FOOD_AND_DRINK.BEER_WINE_AND_LIQUOR
andENTERTAINMENT
to a "Spending" budget.
- For example, you might assign both
- You can assign both a Plaid primary category and its detailed subcategories to different Firefly budgets or categories; just make a careful note in these cases as you have to be careful when building the corresponding Firefly rules later.
- The Plaid category taxonomy includes "primary" categories and "detailed" subcategories.
- Determine what Firefly budgets
and categories you want to use.
- Implement your Firefly budgets/categories and corresponding rules
- Creating budgets and categories is straightforward. You do not need amounts for budgets at this time.
- Rules
- I created a rule group for Plaid category processing, then another rule group below it (and thus overriding it) to handle case-by-case overrides of specific transactions that Plaid didn't categorize the way I wanted.
- For Plaid category processing, my basic rule template is:
- Trigger: when a transaction is created
- Stop processing:
true
- Strict mode:
false
- Rule Triggers: "Any tag is...":
plaid-detailed-cat-coffee
etc.- Add additional tag triggers if multiple Plaid categories are assigned to a single Firefly budget or category
- Action: "Set Budget (or Category) to...": corresponding Firefly budget or category name
- Save a snapshot of your Firefly database state in case the next steps don't do what you want
- This step is optional but recommended.
- If Firefly's using a Sqlite database, all you have to do is copy the database file. If using Mysql or Postgres, you will need to use the appropriate backup and restore tooling.
- Run the connector in
batch
mode- I recommend doing this connector run on a higher-spec machine (i.e. your laptop or whatever) to give it all the memory it needs. It doesn't matter where you run it as long as the connector can make a network connection to your Firefly server.
- The
fireflyPlaidConnector2.batch.maxSyncDays
property is up to you. I used 2 years for my initial backfill, but your value will depend on your needs and patience for the process.- For what it's worth, the performance bottleneck is Firefly handling transaction inserts. This isn't a system resource thing either, as it took about as long on my M1 Max as it did on a $5 VPS.
- Keep an eye on the connector's logs to ensure nothing's gone wrong.
- Go through transaction reports and add additional rules for any categorization gaps
- I went through transactions without budgets for each month and added new override rules as needed.
- Transactions without budgets in Firefly can be viewed by navigating to Budgets in the sidebar, selecting the desired month in the Period Navigator at the top, then clicking the "Expenses without budget" link in the very bottom left of the window.
- The "Apply rule X to a selection of your transactions" feature in the Firefly Automations UI is very useful for filling in budget/category gaps after you've already run a transaction import.
- I went through transactions without budgets for each month and added new override rules as needed.
- If anything's gone wrong, restore your database snapshot from before and try again.
- Set up the connector to run in
polled
mode.- I created a
systemd
unit to run the connector on my Debian VPS, your mileage may vary.- You will of course need to set up users, permissions, directories, etc. to support this.
- I created a
[Unit]
Description=Firefly Plaid Connector 2
[Service]
WorkingDirectory=/opt/firefly-plaid-connector
ExecStart=/bin/java -Xms128m -Xmx1024m -jar firefly-plaid-connector.jar --spring.profiles.active=prod
User=firefly-plaid-connector
Type=simple
Restart=on-failure
RestartSec=10
Environment=“SPRING_CONFIG_LOCATION=/opt/firefly-plaid-connector/application-prod.yml”
[Install]
WantedBy=multi-user.target
On occasion you will get an ITEM_LOGIN_REQUIRED
error in the connector logs. This typically happens when the credentials
for one of the institutional accounts you've linked Plaid to have changed. You can find the access token for the account in question on the log line above the exception log.
There are two methods for resolving the error, described below.
This is the recommended method of resolving this issue, although it's a bit more complex than create mode.
- Check out https://github.com/dvankley/quickstart
- Start the frontend and the java backend per the instructions in the README and verify that the frontend loads correctly at
https://localhost:3000
. - Find the access token value for the account with the
ITEM_LOGIN_REQUIRED
error (it should be in the connector logs just above the exception). - Navigate to
https://localhost:3000?input_access_token=$yourAccessTokenHere
in your browser. - Complete the Link flow in the UI.
After completing these steps, your account credentials should be fixed and the connector should resolve itself on its next run (assuming you're running in polled mode).
This is basically just going through the Connecting Accounts workflow again for that account, replacing the access token and account id in your configuration file, and restarting the connector. Unfortunately, as discussed in #39, this method permanently chews through your Item quota and is NOT RECOMMENDED for that reason. At this point it does have the advantage of being simpler than update mode, which is why it's listed here as an option.
If you're getting this error frequently, check if you have MFA enabled on your account with the corresponding financial institution. MFA can cause high freqency invalidation of Plaid account credentials, so consider disabling it. Obviously compromising your security posture to use this connector isn't great, so hopefully your institution provides limited permission accounts for service access.
Also note that CIBC currently has this issue.
-
When setting up access to a provider from the Plaid Quick Start I'm getting a message "Something went wrong".
- Several institutions are restricting accesss to development access accounts. An approved paid production account will need to be setup with Plaid to gain access to these accounts.
-
I'm getting an
ITEM_LOGIN_REQUIRED
error when running the connector.
The logging
key in the application configuration file controls the logging level of various packages and classes in
the application. Setting logging.level.net.djvk
to DEBUG
or TRACE
is recommended if you're having problems,
as it will log additional info that should help diagnose the issue.
If you have an issue, feel free to report it via the Github issue tracker. I am actively maintaining this project
but my available time is finite, so the odds of your issue being addressed will be increased if you include relevant
logs at the TRACE
level with your issue report.
Writing a test case (i.e. in net.djvk.fireflyPlaidConnector2.transactions.TransactionConverterTest
) to demonstrate
your issue greatly increases the chances of me fixing it quickly. If test infrastructure is missing to test the
element of the code you see an issue with, let me know and I can work on improving that.
Setup should be identical to any other Spring Boot/Gradle application.
I recommend adding an additional configuration file (i.e. application-dev.yml
) and enabling the corresponding Spring
profile (i.e. dev
) to allow you to persist and iterate on your local configuration.
Connecting to the Plaid development
environment as usual should be fine for development.
I recommend setting up a local copy of Firefly for development purposes, especially one that you can easily backup
and restore the database for to minimize your feedback loop on testing things.
I don't currently have firm guidelines yet, but I will adopt some if I ever get contributions. For now the main guideline is to have a test covering the changes you make.
- Why did you make a new program rather than contributing to firefly-plaid-connector?
- I initially tried firefly-plaid-connector, but I had a few issues with it, and it didn't fully support Plaid categories. I tried to set it up for development locally, but after about an hour trying to get the right version of the .NET SDK to work, I decided I was better off making my own gravy. So here we are.
Do you, like me, run Home Assistant and also want real-ish time budget notifications? Then check out this Node-RED flow.