Add Salesforce Connector

[cwarden]

Checklist

• Manual verification
• Unit test coverage
• E2E test coverage
• Needs manual QA?

Summary

Issue addressed:

Use Salesforce Bulk API as a source.

Details:

Add Salesforce connector supporting the FileIterator interface using the Bulk API with CSV jobs.

Each source requires a SOQL Query and SObject name.

Credentials can be provided for the source or the connector.

Each bulk job batch is retrieved as a single CSV file for rill to ingest.

Steps to Verify

1. Add a new Source
2. Select Salesforce
3. Enter Username, Password (with token), SOQL Query, and SObject Name

[k-anshul]

Hey @cwarden

Thanks a lot for working on this.
We are both very happy and impressed to see such important contributions from community members.
I understand this is still a draft but thought to pass some high level feedback on this :
There seems to be some boiler plate code around authentication and the bulk API call and I wonder if some library already achieves the same. We already use libraries for s3/gcs/bigquery/snowflake etc to do something similar.
I took a quick look and found this library but need to evaluate if it supports the features we need to build a MVP connector here. Please let me know what are your thoughts on this and any other library suggestions you might have.

Thank you and looking forward to contribute on this with you.

[cwarden]

Thanks a lot for working on this. We are both very happy and impressed to see such important contributions from community members. I understand this is still a draft but thought to pass some high level feedback on this : There seems to be some boiler plate code around authentication and the bulk API call and I wonder if some library already achieves the same. We already use libraries for s3/gcs/bigquery/snowflake etc to do something similar. I took a quick look and found this library but need to evaluate if it supports the features we need to build a MVP connector here. Please let me know what are your thoughts on this and any other library suggestions you might have.

I'm using the force cli library, which I maintain. It's grown out of the command-line tool and some of the code dates back to Go 1.1 so there are some rough edges, but it has been heavily used.

I plan to clean up the authentication in this PR so it doesn't use a credential store.

I'm not sure if it makes sense to move some of the Bulk API handling into the library, but I'm open to suggestions.

[cwarden]

Thanks for the thorough code review, @k-anshul. You're seeing some of the artifacts of this code being extracted from an existing application, such as the interfaces that were used to facilitate testing.

[k-anshul]

Thanks for the PR @cwarden LGTM
I will also let @begelundmuller take a look at this.
Two things:

1. I discovered that whenever I am submitting an incorrect query, the application gets stuck. Could you please take a look at this ? Sample incorrect query : SELECT Id, IsDeleted, Name, Type, ParentId, Rating, Site FROM Account WHERE Rating is not null
2. Would you also be interested in adding docs for the connector ? It can also be taken up in a separate PR. PR for reference : https://github.com/rilldata/rill/pull/3147/files

[cwarden]

Thanks for the PR @cwarden LGTM I will also let @begelundmuller take a look at this. Two things:

1. I discovered that whenever I am submitting an incorrect query, the application gets stuck. Could you please take a look at this ? Sample incorrect query : SELECT Id, IsDeleted, Name, Type, ParentId, Rating, Site FROM Account WHERE Rating is not null

Fixed in d82fb1f.

2. Would you also be interested in adding docs for the connector ? It can also be taken up in a separate PR. PR for reference : https://github.com/rilldata/rill/pull/3147/files

Added in 8951aec.

[k-anshul]

Thanks