Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

input UTF-16LE file creates garbled output #15977

Open
leron-l opened this issue Mar 2, 2024 · 0 comments
Open

input UTF-16LE file creates garbled output #15977

leron-l opened this issue Mar 2, 2024 · 0 comments

Comments

@leron-l
Copy link

leron-l commented Mar 2, 2024

Logstash information:

  1. Logstash version : 8.12
  2. Logstash installation source : zip
  3. How is Logstash being run Via command line

OS version : windows11

input file: test.sql: text/plain; charset=utf-16le

this file encode UTF-16LE, With this config, the output is garbled.

config

input {
  file {
    path => ["/test.sql"]
    tags => ["test"]
    start_position => "beginning"
    codec => plain {
      charset => "UTF-16LE"
    }
  }
}

filter {
}

output {
  stdout {
    codec => rubydebug
  }
}

debug log

Received line {:path=>"/test.sql", :text=>"\xFF\xFEU\x00S\x00E\x00 \x00[\x00B\x00i\x00g\x00D\x00a\x00t\x00a\x00]\x00"}
config LogStash::Codecs::Plain/@charset = "UTF-16LE"
config LogStash::Codecs::Plain/@id = "f0a2ff23-f6f4-4a06-bf48-4e5cd4e3f1a1"
config LogStash::Codecs::Plain/@enable_metric = true
inputs/LogStash::Inputs::File: adding tag {"tag"=>"test"}
Received line {:path=>"/test.sql", :text=>"\x00G\x00O\x00"}
inputs/LogStash::Inputs::File: adding tag {"tag"=>"test"}
Received line {:path=>"/test.sql", :text=>"\x00/\x00*\x00*\x00*\x00*\x00*\x00*\x00 \x00O\x00b\x00j\x00e\x00c\x00t\x00:\x00 \x00 \x00T\
inputs/LogStash::Inputs::File: adding tag {"tag"=>"test"}
Received line {:path=>"/test.sql", :text=>"\x00S\x00E\x00T\x00 \x00A\x00N\x00S\x00I\x00_\x00N\x00U\x00L\x00L\x00S\x00 \x00O\x00N\x00"}

I've noticed that the charset configuration(Codecs::Plain/@charset = "UTF-16LE") doesn't output until after the first row of data,
this reason cause garbled , or is there a problem with config

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant