Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support compression in read_cmdstan_csv #1027

Open
jsks opened this issue Oct 1, 2024 · 0 comments
Open

Support compression in read_cmdstan_csv #1027

jsks opened this issue Oct 1, 2024 · 0 comments
Labels
feature New feature or request

Comments

@jsks
Copy link

jsks commented Oct 1, 2024

Cmdstan output files can get pretty large. data.table::fread used in read_cmdstan_csv and read_csv_metadata supports gzip and bzip2 compressed files if the R.utils package is installed.

On the cmdstanr side this would require at least modifying the suffix assertions (assert_file_exists) in both functions. And, switching out grep in favor of zgrep to handle the commented headers with the caveat (at least on linux) that the argument order be switched so that --color=never comes before the regex pattern.

So in read_cmdstan_csv:

fread_cmd <- paste0("zgrep --color=never -v '^#' '", path.expand(output_file), "'")

read_metadata_csv:

fread_cmd <- paste0("zgrep --color=never '^[#a-zA-Z]' '", path.expand(csv_file), "'")

and similarly for the WSL specific code.

@jsks jsks added the feature New feature or request label Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant