Change base read function

[nealhaddaway]

The app was built using read.csv, should change to readr::read_delim

[nealhaddaway]

Having a major headache doing this. Switched the code out fine, coerced it into a dataframe rather than a tibble for an initial debug, but there are some rogue 'if' statements causing it to fail despite reading in OK. Can't work out which one, though. Help MUCH appreciated, @DrMattG!

[DrMattG]

This is a weird one! If we strip it back to utils::read.delim it works - so it has to be something to do with what readr::read_delim does as opposed to read.delim - readr::read_delim gives us a tibble (https://readr.tidyverse.org/reference/read_delim.html) and utils::read.delim gives us a dataframe but I am not sure if that is the route of the problem.... gahhhhhhhhhhhhhhhhhhhhhhh (this is annoying!!!!!!)

[DrMattG]

@nealhaddaway
okay - so just run some debugging and the problem is in

locale = readr::locale(decimal_mark = input$dec,
                              encoding = input$upload_encoding)

There is no input$dec - if I add one in then I get an error about the default encoding. So if I delete the default option and select UTF-8 then it will upload- I am not sure of what effects this has further down the line though

## ui.R ##
......
fluidRow(
                       column(12,
                              wellPanel(
                              h5(strong("CSV Properties")),
                              # Input: Checkbox if file has header ----
                              checkboxInput("header", "Header row?", TRUE),
                              
                              selectInput("upload_encoding",
                                          label = "Select File Encoding",
                                          choices = list("UTF-8", 
                                                         "latin1",
                                                         "mac"),
                                          selected = "UTF-8"
                              ),
                              # Input: Select separator ----
                              selectInput("sep",
                                           "Field Separator",
                                           choices = c(
                                             ",",
                                             ";",
                                             Tab = "\t",
                                             '|'
                                           ),
                                           selected = ","
                              ),
                              # Input: Select quotes ----
                              selectInput(
                                "quote",
                                "Quote Delimiter",
                                choices = c(
                                  None = "",
                                  '"',
                                  "'"
                                ),
                                selected = '"'
                              ),
                              selectInput(
                                "dec",
                                "Decimal marker",
                                choices = c(".", ","),
                                selected = "."
                              ))))
                   )),
`
.....

## server.R ##
.....
# if CSV data are supplied, add them to data_internal
   observeEvent(input$sysmapdata_upload, {
       raw <- data.frame(readr::read_delim(
       file = input$sysmapdata_upload$datapath,
       delim = input$sep,
       quote = input$quote,
       locale = readr::locale(decimal_mark = input$dec,
                             encoding = input$upload_encoding)
       ))
      data_internal$raw <- data.frame(lapply(raw, function(x) iconv(x, "UTF-8", "UTF-8",sub=' ')))
     data_internal$filtered <- data_internal$raw #instantiate filtered table with raw values
     
  
   })

[nealhaddaway]

Hmm - do you mean locale doesn't allow decimal specification? Or that it's missing from the UI?

[DrMattG]

pull request #88 and #86 (Dont ask what happened to 87!) should fix this now

[nealhaddaway]

I guess this is fixed now - it's working, but now no longer properly allows one dodgy file I have to be encoded in Mac to work. Jacqui sent it to me the other day. So read.csv correctly (?) dealt with non-standard characters in mac encoding, but this now stops that workaround... not ideal... It also isn't reading in äå etc. properly...