Add CSV file import function

[KumaJie]

#27148

[mergify]

@KumaJie E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@KumaJie E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[wayblink]

@KumaJie
Hi, thanks for your contribution.

I tried your code. One problem I found: CSV file start with a Byte Order Mark \ufeffi will fail with error message the field '\ufeffid' is not defined in collection schema. I think we need to make it compatible.

Some discussion about BOM: https://stackoverflow.com/questions/21371673/reading-files-with-a-bom-in-go

[KumaJie]

@KumaJie Hi, thanks for your contribution.

I tried your code. One problem I found: CSV file start with a Byte Order Mark \ufeffi will fail with error message the field '\ufeffid' is not defined in collection schema. I think we need to make it compatible.

Some discussion about BOM: https://stackoverflow.com/questions/21371673/reading-files-with-a-bom-in-go

Thanks for your feedback and I will fix that. Your suggestion is very helpful😊.

// csv parser
reader := bufio.NewReader(file)
// discard bom in the file
r, _, err := reader.ReadRune()
if err != nil {
    return err
}
if r != '\ufeff' {
    reader.UnreadRune()
}
// ....

[xiaofan-luan]

rerun the ut

[mergify]

@KumaJie ut workflow job failed, comment rerun ut can trigger the job again.

[codecov]

Codecov Report

Merging #27149 (42154f1) into master (bf7f32b) will increase coverage by 0.03%.
Report is 12 commits behind head on master.
The diff coverage is 86.67%.

@@            Coverage Diff             @@
##           master   #27149      +/-   ##
==========================================
+ Coverage   81.74%   81.78%   +0.03%     
==========================================
  Files         821      823       +2     
  Lines      116328   116911     +583     
==========================================
+ Hits        95098    95614     +516     
- Misses      18071    18116      +45     
- Partials     3159     3181      +22     

Files	Coverage Δ
internal/rootcoord/import_manager.go	75.70% <100.00%> (ø)
internal/util/importutil/import_util.go	97.77% <ø> (ø)
internal/util/importutil/json_parser.go	96.77% <100.00%> (+0.03%)	⬆️
internal/util/importutil/import_wrapper.go	91.14% <76.00%> (-2.07%)	⬇️
internal/util/importutil/csv_handler.go	91.90% <91.90%> (ø)
internal/util/importutil/csv_parser.go	81.41% <81.41%> (ø)

... and 11 files with indirect coverage changes

[wayblink]

csv InputOffset() is only supported after go 1.19. This will break milvus's go >=1.18 dependency restrictions. @xiaofan-luan Is that OK or we have to work around it?

[wayblink]

@KumaJie Generally LGTM. I prefer to put BOM handle logic into CSVParser. Could you please adjust it and fix code checks in https://github.com/milvus-io/milvus/actions/runs/6298875222/job/17382624374?pr=27149?

[KumaJie]

rerun the ut

[KumaJie]

rerun ut

[github-actions]

Hello KumaJie, you are not in the organization, so you do not have the permission to rerun the workflow, please contact @milvus-io/milvus-maintainers for help.

[xiaofan-luan]

rerun it

[czs007]

/approve
/lgtm

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, KumaJie

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment