Road to v0.2

[matthewmturner]

This issue for setting the specific issues we would like completed before a v0.2 release. My hope is that there can be a release by end of Sept / Beginning of October.

• Querys that return large # of results can freeze / crash the app #137
• To investigate: Is DDL working correctly? #138
• Add --flightsql paramter for -c and -f #128
• (REQUIRED) Add tests for configuration system #144
• (REQUIRED) Implement select * from 'foo.parquet' support #125
• Add back DDL mode #153
• (REQUIRED) Create / add duckdb metadata functions #148
• (REQUIRED) Update to DataFusion 42.0.0 #158
• Add syntax highlighting to config #159
• Update from datafusionrc to ddl in config dir #166
• (REQUIRED) Figure out ddl mode executed / error UI #167
• (REQUIRED) DDL should optionally be available from CLI #170
• Add serve subcommand for flightsql feature #168
• Setup localstack for testing s3 #173
• Add ui to flightsql tab when unable to connect #175
• Test flightsql error cases #177
• (REQUIRED) Benchmark from CLI
• (REQUIRED) Analyze from CLI
• DDL mode isnt loading DDL file correctly now #213
• Add select / copy / paste / run selection to tui editor #224

[matthewmturner]

@alamb let me know if there is anything you would like to add

[alamb]

I think we should consider adding #148 and #158 as well

[matthewmturner]

So I think it turns out that the DDL issue I have is a bug with datafusion registering external tables to a schema.

I have this in my ~/.datafusion/.datafusionrc`

CREATE SCHEMA staging;

CREATE TABLE staging.foo AS VALUES (1);
CREATE EXTERNAL TABLE staging.min_aggs STORED AS PARQUET LOCATION 'ny2://atlas/sip/aggregates/minute_by_ticker_monthly_v2/year_month=2015-08/data.01.parquet';

But when I run SHOW TABLES I get the following:

Notice how the schema isnt picked up for the the external table. I dont see anything in the docs for CREATE EXTERNAL TABLE that would suggest this shouldnt work but maybe im misunderstanding.

@alamb do you have a view on this? If you agree its a bug in datafusion then i can create a ticket there.

[alamb]

I agree it is a datafusion bug. I will file a ticket

[alamb]

Filed apache/datafusion#12607

[Kinrany]

Is there anything that the current trunk version does worse than v0.1?

[matthewmturner]

@Kinrany i dont think so although i havent used v0.1 in a long time (years). i took a brief pause for december to work on some fun not work related project but im about to get back into dft in the beginning of the new year. im going to review where things are again and potentially do a release early on before proceeding to the next round of features i have in mind.