Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use security_protocol sasl_ssl, why the options jaas_path is not work and invalid? #136

Open
Anteoy opened this issue Jun 8, 2017 · 2 comments

Comments

@Anteoy
Copy link

Anteoy commented Jun 8, 2017

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

  • Version: logstash 5.4.0

  • Operating System: ubuntu 16.04

  • Config File (if you have sensitive info, please remove it):
    output {
    kafka {
    bootstrap_servers => "SASL_SSL://xxx"
    topic_id => ["xxx"]

      security_protocol => "SASL_SSL"
      sasl_mechanism => "ONS"
    
      jaas_path => "xx/agent/jaas.conf"
    
      ssl_truststore_password => "xx"
      ssl_truststore_location => "xx/kafka.client.truststore.jks"
      # sasl_kerberos_service_name => "kafka"
      codec => json
    

    }
    }

  • Sample Data:
    logstash -f [this config file]

  • Steps to Reproduce:

It worked at the same config in the node of input,but in the node of output ,It isn't valid and didn't work,if my file of ..logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.6/vendor/jar-dependencies/runtime-jars/ons-sasl-client-0.1.jar is not existed.it will log 'can not find the class in this jar' at the node of input kafka ,but at the node of output,It only log 'Failed to construct kafka producer' with anything else,it will not log like this 'can not find the class in this jar',what should I do for jaas_path in the config's node of output?

@Anteoy
Copy link
Author

Anteoy commented Jun 8, 2017

More logstash log (when the jaas_path file is not existed):
Sending Logstash's logs to /home/zhoudazhuang/bin-software/logstash-5.4.0/logs which is now configured via log4j2.properties
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.producer.ProducerConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2017-06-09T00:44:56,863][ERROR][logstash.outputs.kafka ] Unable to create Kafka producer from given configuration {:kafka_error_message=>org.apache.kafka.common.KafkaException: Failed to construct kafka producer}
[2017-06-09T00:44:56,867][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::OutputDelegator:0x3e8ccfe2 @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0xc091a31 @Metric=#<LogStash::Instrument::Metric:0x63068023 @collector=#<LogStash::Instrument::Collector:0x2f444f4a @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x77ef669f @store=#<Concurrent::Map:0x337322c1 @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x6423bb14, @fast_lookup=#<Concurrent::Map:0x691096c5 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"d53492fd674ac37554dc8251d2aef5de98461009-2"]>, @Metric=#<LogStash::Instrument::NamespacedMetric:0x4f089db5 @Metric=#<LogStash::Instrument::Metric:0x63068023 @collector=#<LogStash::Instrument::Collector:0x2f444f4a @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x77ef669f @store=#<Concurrent::Map:0x337322c1 @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x6423bb14, @fast_lookup=#<Concurrent::Map:0x691096c5 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#<LogStash::Logging::Logger:0x5bc16af1 @logger=#Java::OrgApacheLoggingLog4jCore::Logger:0x1ddbf1b0>, @strategy=#<LogStash::OutputDelegatorStrategies::Shared:0xbc46f14 @output=<LogStash::Outputs::Kafka bootstrap_servers=>"SASL_SSL://kafka-ons-internet.aliyun.com:8080", topic_id=>"class100LogKafka", security_protocol=>"SASL_SSL", sasl_mechanism=>"ONS", jaas_path=>"/home/zhoudazhuang/class100/docker-repo/seelog-logstash/agent/jaas.conf", ssl_truststore_password=>, ssl_truststore_location=>"/home/zhoudazhuang/class100/docker-repo/seelog-logstash/agent/kafka.client.truststore.jks", codec=><LogStash::Codecs::JSON id=>"json_75b5d496-8e82-42ef-adcb-f571fe51549f", enable_metric=>true, charset=>"UTF-8">, id=>"d53492fd674ac37554dc8251d2aef5de98461009-2", enable_metric=>true, workers=>1, acks=>"1", batch_size=>16384, block_on_buffer_full=>true, buffer_memory=>33554432, compression_type=>"none", key_serializer=>"org.apache.kafka.common.serialization.StringSerializer", linger_ms=>0, max_request_size=>1048576, metadata_fetch_timeout_ms=>60000, metadata_max_age_ms=>300000, receive_buffer_bytes=>32768, reconnect_backoff_ms=>10, retries=>0, retry_backoff_ms=>100, send_buffer_bytes=>131072, ssl=>false, timeout_ms=>30000, value_serializer=>"org.apache.kafka.common.serialization.StringSerializer">>, @id="d53492fd674ac37554dc8251d2aef5de98461009-2", @metric_events=#<LogStash::Instrument::NamespacedMetric:0x6e896a8b @Metric=#<LogStash::Instrument::Metric:0x63068023 @collector=#<LogStash::Instrument::Collector:0x2f444f4a @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x77ef669f @store=#<Concurrent::Map:0x337322c1 @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x6423bb14, @fast_lookup=#<Concurrent::Map:0x691096c5 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"d53492fd674ac37554dc8251d2aef5de98461009-2", :events]>, @output_class=LogStash::Outputs::Kafka>", :error=>"Failed to construct kafka producer"}
[2017-06-09T00:44:56,878][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>org.apache.kafka.common.KafkaException: Failed to construct kafka producer, :backtrace=>["org.apache.kafka.clients.producer.KafkaProducer.(org/apache/kafka/clients/producer/KafkaProducer.java:335)", "org.apache.kafka.clients.producer.KafkaProducer.(org/apache/kafka/clients/producer/KafkaProducer.java:188)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:423)", "RUBY.create_producer(/home/zhoudazhuang/bin-software/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.5/lib/logstash/outputs/kafka.rb:242)", "RUBY.register(/home/zhoudazhuang/bin-software/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.5/lib/logstash/outputs/kafka.rb:178)", "RUBY.register(/home/zhoudazhuang/bin-software/logstash-5.4.0/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9)", "RUBY.register(/home/zhoudazhuang/bin-software/logstash-5.4.0/logstash-core/lib/logstash/output_delegator.rb:41)", "RUBY.register_plugin(/home/zhoudazhuang/bin-software/logstash-5.4.0/logstash-core/lib/logstash/pipeline.rb:268)", "RUBY.register_plugins(/home/zhoudazhuang/bin-software/logstash-5.4.0/logstash-core/lib/logstash/pipeline.rb:279)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.register_plugins(/home/zhoudazhuang/bin-software/logstash-5.4.0/logstash-core/lib/logstash/pipeline.rb:279)", "RUBY.start_workers(/home/zhoudazhuang/bin-software/logstash-5.4.0/logstash-core/lib/logstash/pipeline.rb:288)", "RUBY.run(/home/zhoudazhuang/bin-software/logstash-5.4.0/logstash-core/lib/logstash/pipeline.rb:214)", "RUBY.start_pipeline(/home/zhoudazhuang/bin-software/logstash-5.4.0/logstash-core/lib/logstash/agent.rb:398)", "java.lang.Thread.run(java/lang/Thread.java:745)"]}
[2017-06-09T00:44:56,902][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-06-09T00:44:59,886][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}

this is use kafka for input log(when the jaas_path file is not existed):
[2017-06-09T00:45:22,655][ERROR][logstash.inputs.kafka ] Unable to create Kafka consumer from given configuration {:kafka_error_message=>org.apache.kafka.common.KafkaException: Failed to construct kafka consumer, :cause=>org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: can not find the LoginModule class: com.aliyun.openservices.ons.sasl.client.OnsLoginModule}

@Anteoy
Copy link
Author

Anteoy commented Jun 9, 2017

I guess the reason is the kafka construct producer faild, I only find the error '"java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:423)"' , Is there any detail logs for this? Is the options of sasl.kerberos.service.name necessary?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant