Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Meta] OpenSearch / OpenSearch Dashboards - Windows distribution (X64) #33

Closed
7 of 19 tasks
Tracked by #741
bbarani opened this issue May 18, 2021 · 49 comments
Closed
7 of 19 tasks
Tracked by #741
Assignees
Labels
campaign Parent issues of OpenSearch release campaigns. distributions enhancement New Enhancement Meta triaged This issue has been reviewed by the triage team v2.4.0 'Issues and PRs related to version v2.4.0' windows

Comments

@bbarani
Copy link
Member

bbarani commented May 18, 2021

Describe the solution you'd like
Provide signed (X64)Windows artifact for OpenSearch / OpenSearch Dashboards

Additional context
Currently there are no Windows artifacts for OpenSearch / OpenSearch Dashboards

Current State (2021/10/28)

You can build on Windows or build cross-platform for Windows on Linux or MacOS.

  1. Install pre-requisites (everything needed in OpenSearch Dev Guide + maven for Security Plugin).
  2. Build artifacts from source, ./build.sh manifests/1.1.0/opensearch-1.1.0.yml --platform windows
  3. Assemble a bundle, ./assemble.sh buids/manifest.yml.
  4. Produces opensearch-1.1.0-SNAPSHOT-windows-x64.zip.
  5. Extract .zip, e.g. into C:\Program Files\OpenSearch
  6. Run bash plugins/opensearch-security/tools/install_demo_configuration.sh and follow prompts to create demo certificate.
  7. mkdir logs ([BUG] Windows package doesn't contain a logs folder and doesn't start OpenSearch#1463)
  8. Run bin\opensearch.bat, server will start.
  9. curl https://localhost:9200 -u admin:admin --insecure
(Old Tasks)

Remaining

Done

Design Document

OpenSearch and OpenSearch Dashboards

Plugins Meta Issues

Milestones

MSI Only Issue:

@bbarani bbarani added the enhancement New Enhancement label May 18, 2021
@cashel1
Copy link

cashel1 commented Jul 20, 2021

The greatest value and the least effort would be a ZIP deliverable. This would be sufficient for windows desktop developers and sysadmins installing on a server. Thanks

@spaceghostcowboy
Copy link

Hello, we would also be interested in an installable Windows version or instructions on how to get this working on Windows. We were about to implement this tech into a project and then saw the lic change and fork. We are eager to get onto the forked version and get something into prod. Thank you for reading this.

@dblock
Copy link
Member

dblock commented Sep 12, 2021

If anyone on this issue wants to contribute I'd start with opensearch-project/dashboards-observability#11. The python build workflow will need a bit of work to execute properly on Windows, it currently assumes a lot of *nix paths.

@peternied peternied added the triaged This issue has been reviewed by the triage team label Sep 21, 2021
@cjcjameson
Copy link

We are looking at https://discuss.opendistrocommunity.dev/t/windows-installation/6555/6 which talks about workarounds, but also hoping to get this supported natively with a distribution released.

Sad that it moved to target 1.3 (https://github.com/orgs/opensearch-project/projects/1?card_filter_query=windows) but looking for ways to help.

@tommyongsf
Copy link

+1

@dblock
Copy link
Member

dblock commented Oct 5, 2021

The easiest way to help is to get bundle workflow to succeed on Windows. It think it would require a little bit of python work (we join paths with / all over the place) and small stuff stuff like that. Basically I want to be able to run the equivalent of build.sh ../manifests/1.1.0/openseach-1.1.0.yml on Windows.

I can help merging PRs and such and maybe we can get this in earlier?

@marianotn
Copy link

+1 too

@jcgraybill
Copy link
Contributor

For reference, Elasticsearch's windows installation offered admins a set of options for how to run/manage it, including as a service. Let's be sure to understand which of these are important to users on Windows and implement them for the Windows version of OpenSearch. https://www.elastic.co/guide/en/elasticsearch/reference/current/windows.html#msi-installer-windows-service

@dblock
Copy link
Member

dblock commented Oct 11, 2021

Beyond building on Windows, I would be happy to help anyone to write a wix-based installer for the OpenSearch distribution for windows, possibly made faster by using this library of custom actions. Definitely a project that could then be moved and maintained as part of opensearch-project org. It would make it super easy for anyone running Windows to get started with OpenSearch.

@peterzhuamazon
Copy link
Member

After testing on Windows 2019 server with GUI, I am able to start the Linux x64 Tarball by removing linux jdk, install Windows jdk, remove knn, remove PA.


# ./opensearch-tar-install.sh
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /opensearch/opensearch-1.1.0
OpenSearch install type: .tar.gz on NAME=MSYS2
OpenSearch config dir: /opensearch/opensearch-1.1.0/config
OpenSearch config file: /opensearch/opensearch-1.1.0/config/opensearch.yml
OpenSearch bin dir: /opensearch/opensearch-1.1.0/bin
OpenSearch plugins dir: /opensearch/opensearch-1.1.0/plugins
OpenSearch lib dir: /opensearch/opensearch-1.1.0/lib
Detected OpenSearch Version: x-content-1.1.0
Detected OpenSearch Security Version: 1.1.0.0
/opensearch/opensearch-1.1.0/config/opensearch.yml seems to be already configured for Security. Quit.
chmod: cannot access '/opensearch/opensearch-1.1.0/plugins/opensearch-performance-analyzer/pa_bin/performance-analyzer-agent': No such file or directory
done security
done plugins
Checking kNN library
ls: cannot access '/opensearch/opensearch-1.1.0/plugins/opensearch-knn/knnlib/libKNNIndex*.so': No such file or directory
TEST FAILED OR FILE NOT EXIST
KNN lib path not found, set new path
:/opensearch/opensearch-1.1.0/plugins/opensearch-knn/knnlib
[2021-10-11T22:20:04,066][INFO ][o.o.n.Node               ] [EC2AMAZ-0ODBHTT] version[1.1.0], pid[4984], build[tar/15e9f137622d878b79103df8f82d78d782b686a1/2021-10-04T21:29:03.079792Z], OS[Windows Server 2019/10.0/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.2/14.0.2+12]
[2021-10-11T22:20:04,066][INFO ][o.o.n.Node               ] [EC2AMAZ-0ODBHTT] JVM home [C:\msys64\opensearch\opensearch-1.1.0\jdk], using bundled JDK [true]
[2021-10-11T22:20:04,066][INFO ][o.o.n.Node               ] [EC2AMAZ-0ODBHTT] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=C:\msys64\tmp\opensearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=1000, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=C:/msys64/opensearch/opensearch-1.1.0/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=C:/msys64/opensearch/opensearch-1.1.0, -Dopensearch.path.conf=C:/msys64/opensearch/opensearch-1.1.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2021-10-11T22:20:05,176][INFO ][o.o.s.s.t.SSLConfig      ] [EC2AMAZ-0ODBHTT] SSL dual mode is disabled
[2021-10-11T22:20:05,176][INFO ][o.o.s.OpenSearchSecurityPlugin] [EC2AMAZ-0ODBHTT] OpenSearch Config path is C:\msys64\opensearch\opensearch-1.1.0\config
[2021-10-11T22:20:05,441][INFO ][o.o.s.s.DefaultSecurityKeyStore] [EC2AMAZ-0ODBHTT] JVM supports TLSv1.3
[2021-10-11T22:20:05,441][INFO ][o.o.s.s.DefaultSecurityKeyStore] [EC2AMAZ-0ODBHTT] Config directory is C:\msys64\opensearch\opensearch-1.1.0\config/, from there the key- and truststore files are resolved relatively
[2021-10-11T22:20:06,004][INFO ][o.o.s.s.DefaultSecurityKeyStore] [EC2AMAZ-0ODBHTT] TLS Transport Client Provider : JDK
[2021-10-11T22:20:06,004][INFO ][o.o.s.s.DefaultSecurityKeyStore] [EC2AMAZ-0ODBHTT] TLS Transport Server Provider : JDK
[2021-10-11T22:20:06,004][INFO ][o.o.s.s.DefaultSecurityKeyStore] [EC2AMAZ-0ODBHTT] TLS HTTP Provider             : JDK
[2021-10-11T22:20:06,004][INFO ][o.o.s.s.DefaultSecurityKeyStore] [EC2AMAZ-0ODBHTT] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2021-10-11T22:20:06,004][INFO ][o.o.s.s.DefaultSecurityKeyStore] [EC2AMAZ-0ODBHTT] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2, TLSv1.1]
[2021-10-11T22:20:06,207][INFO ][o.o.s.OpenSearchSecurityPlugin] [EC2AMAZ-0ODBHTT] Clustername: opensearch
[2021-10-11T22:20:09,394][INFO ][o.o.i.r.ReindexPlugin    ] [EC2AMAZ-0ODBHTT] ReindexPlugin reloadSPI called
[2021-10-11T22:20:09,394][INFO ][o.o.i.r.ReindexPlugin    ] [EC2AMAZ-0ODBHTT] Unable to find any implementation for RemoteReindexExtension
[2021-10-11T22:20:09,410][INFO ][o.o.j.JobSchedulerPlugin ] [EC2AMAZ-0ODBHTT] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2021-10-11T22:20:09,426][INFO ][o.o.j.JobSchedulerPlugin ] [EC2AMAZ-0ODBHTT] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2021-10-11T22:20:09,504][INFO ][o.o.j.JobSchedulerPlugin ] [EC2AMAZ-0ODBHTT] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [aggs-matrix-stats]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [analysis-common]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [geo]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [ingest-common]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [ingest-geoip]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [ingest-user-agent]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [lang-expression]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [lang-mustache]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [lang-painless]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [mapper-extras]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [opensearch-dashboards]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [parent-join]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [percolator]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [rank-eval]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [reindex]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [repository-url]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded module [transport-netty4]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-alerting]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-anomaly-detection]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-asynchronous-search]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-cross-cluster-replication]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-index-management]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-job-scheduler]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-notebooks]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-reports-scheduler]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-security]
[2021-10-11T22:20:09,504][INFO ][o.o.p.PluginsService     ] [EC2AMAZ-0ODBHTT] loaded plugin [opensearch-sql]
[2021-10-11T22:20:09,519][INFO ][o.o.s.OpenSearchSecurityPlugin] [EC2AMAZ-0ODBHTT] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2021-10-11T22:20:09,535][INFO ][o.o.e.NodeEnvironment    ] [EC2AMAZ-0ODBHTT] using [1] data paths, mounts [[(C:)]], net usable_space [77gb], net total_space [99.9gb], types [NTFS]
[2021-10-11T22:20:09,535][INFO ][o.o.e.NodeEnvironment    ] [EC2AMAZ-0ODBHTT] heap size [1gb], compressed ordinary object pointers [true]
[2021-10-11T22:20:09,598][INFO ][o.o.n.Node               ] [EC2AMAZ-0ODBHTT] node name [EC2AMAZ-0ODBHTT], node ID [DRoyIbNJQc6vfJ8KlwRD7w], cluster name [opensearch], roles [master, remote_cluster_client, data, ingest]
[2021-10-11T22:20:13,331][WARN ][o.o.s.c.Salt             ] [EC2AMAZ-0ODBHTT] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2021-10-11T22:20:13,363][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Message routing enabled: true
[2021-10-11T22:20:13,472][INFO ][o.o.s.f.SecurityFilter   ] [EC2AMAZ-0ODBHTT] <NONE> indices are made immutable.
[2021-10-11T22:20:13,925][INFO ][o.o.a.b.ADCircuitBreakerService] [EC2AMAZ-0ODBHTT] Registered memory breaker.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/C:/msys64/opensearch/opensearch-1.1.0/plugins/opensearch-anomaly-detection/protostuff-runtime-1.7.4.jar) to field java.lang.Throwable.cause
WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2021-10-11T22:20:14,753][INFO ][o.o.t.NettyAllocator     ] [EC2AMAZ-0ODBHTT] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2021-10-11T22:20:14,831][INFO ][o.o.d.DiscoveryModule    ] [EC2AMAZ-0ODBHTT] using discovery type [zen] and seed hosts providers [settings]
[2021-10-11T22:20:15,222][WARN ][o.o.g.DanglingIndicesState] [EC2AMAZ-0ODBHTT] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-10-11T22:20:15,722][INFO ][o.o.n.Node               ] [EC2AMAZ-0ODBHTT] initialized
[2021-10-11T22:20:15,722][INFO ][o.o.n.Node               ] [EC2AMAZ-0ODBHTT] starting ...
[2021-10-11T22:20:15,935][INFO ][o.o.t.TransportService   ] [EC2AMAZ-0ODBHTT] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2021-10-11T22:20:16,138][WARN ][o.o.b.BootstrapChecks    ] [EC2AMAZ-0ODBHTT] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-10-11T22:20:16,157][INFO ][o.o.c.c.ClusterBootstrapService] [EC2AMAZ-0ODBHTT] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-10-11T22:20:19,164][INFO ][o.o.c.c.Coordinator      ] [EC2AMAZ-0ODBHTT] setting initial configuration to VotingConfiguration{DRoyIbNJQc6vfJ8KlwRD7w}
[2021-10-11T22:20:19,368][INFO ][o.o.c.s.MasterService    ] [EC2AMAZ-0ODBHTT] elected-as-master ([1] nodes joined)[{EC2AMAZ-0ODBHTT}{DRoyIbNJQc6vfJ8KlwRD7w}{9Cy-ux1cTZWwIh6ss72SrQ}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{EC2AMAZ-0ODBHTT}{DRoyIbNJQc6vfJ8KlwRD7w}{9Cy-ux1cTZWwIh6ss72SrQ}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2021-10-11T22:20:19,430][INFO ][o.o.c.c.CoordinationState] [EC2AMAZ-0ODBHTT] cluster UUID set to [0IWTz4_WTqa5PjQQEWz5dw]
[2021-10-11T22:20:19,493][INFO ][o.o.c.s.ClusterApplierService] [EC2AMAZ-0ODBHTT] master node changed {previous [], current [{EC2AMAZ-0ODBHTT}{DRoyIbNJQc6vfJ8KlwRD7w}{9Cy-ux1cTZWwIh6ss72SrQ}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2021-10-11T22:20:19,508][INFO ][o.o.a.c.ADClusterEventListener] [EC2AMAZ-0ODBHTT] Cluster is not recovered yet.
[2021-10-11T22:20:19,540][INFO ][o.o.h.AbstractHttpServerTransport] [EC2AMAZ-0ODBHTT] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2021-10-11T22:20:19,540][INFO ][o.o.n.Node               ] [EC2AMAZ-0ODBHTT] started
[2021-10-11T22:20:19,540][INFO ][o.o.s.OpenSearchSecurityPlugin] [EC2AMAZ-0ODBHTT] Node started
[2021-10-11T22:20:19,540][INFO ][o.o.s.c.ConfigurationRepository] [EC2AMAZ-0ODBHTT] Will attempt to create index .opendistro_security and default configs if they are absent
[2021-10-11T22:20:19,540][INFO ][o.o.s.c.ConfigurationRepository] [EC2AMAZ-0ODBHTT] Background init thread started. Install default config?: true
[2021-10-11T22:20:19,540][INFO ][o.o.s.OpenSearchSecurityPlugin] [EC2AMAZ-0ODBHTT] 0 OpenSearch Security modules loaded so far: []
[2021-10-11T22:20:19,594][INFO ][o.o.a.c.HashRing         ] [EC2AMAZ-0ODBHTT] Node added: [DRoyIbNJQc6vfJ8KlwRD7w]
[2021-10-11T22:20:19,594][INFO ][o.o.g.GatewayService     ] [EC2AMAZ-0ODBHTT] recovered [0] indices into cluster_state
[2021-10-11T22:20:19,594][INFO ][o.o.a.c.HashRing         ] [EC2AMAZ-0ODBHTT] Add data node to AD version hash ring: DRoyIbNJQc6vfJ8KlwRD7w
[2021-10-11T22:20:19,609][INFO ][o.o.a.c.HashRing         ] [EC2AMAZ-0ODBHTT] All nodes with known AD version: {DRoyIbNJQc6vfJ8KlwRD7w=ADNodeInfo{version=1.1.0, isEligibleDataNode=true}}
[2021-10-11T22:20:19,609][INFO ][o.o.a.c.HashRing         ] [EC2AMAZ-0ODBHTT] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2021-10-11T22:20:19,609][INFO ][o.o.a.c.HashRing         ] [EC2AMAZ-0ODBHTT] Build AD version hash ring successfully
[2021-10-11T22:20:19,609][INFO ][o.o.a.c.ADDataMigrator   ] [EC2AMAZ-0ODBHTT] Start migrating AD data
[2021-10-11T22:20:19,609][INFO ][o.o.a.c.ADDataMigrator   ] [EC2AMAZ-0ODBHTT] AD job index doesn't exist, no need to migrate
[2021-10-11T22:20:19,609][INFO ][o.o.a.c.ADClusterEventListener] [EC2AMAZ-0ODBHTT] Init AD version hash ring successfully
[2021-10-11T22:20:19,719][INFO ][o.o.c.m.MetadataCreateIndexService] [EC2AMAZ-0ODBHTT] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1]
[2021-10-11T22:20:19,734][INFO ][o.o.c.r.a.AllocationService] [EC2AMAZ-0ODBHTT] updating number_of_replicas to [0] for indices [.opendistro_security]
[2021-10-11T22:20:20,058][INFO ][o.o.c.r.a.AllocationService] [EC2AMAZ-0ODBHTT] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
[2021-10-11T22:20:20,137][INFO ][o.o.s.c.ConfigurationRepository] [EC2AMAZ-0ODBHTT] Index .opendistro_security created?: true
[2021-10-11T22:20:20,137][INFO ][o.o.s.c.ConfigurationRepository] [EC2AMAZ-0ODBHTT] Node started, try to initialize it. Wait for at least yellow cluster state....
[2021-10-11T22:20:20,137][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Will update 'config' with C:\msys64\opensearch\opensearch-1.1.0\plugins/opensearch-security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-10-11T22:20:20,230][INFO ][o.o.c.m.MetadataMappingService] [EC2AMAZ-0ODBHTT] [.opendistro_security/QYxJjeRMTVqMX8xEttANYg] create_mapping [_doc]
[2021-10-11T22:20:20,465][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Doc with id 'config' and version 2 is updated in .opendistro_security index.
[2021-10-11T22:20:20,465][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Will update 'roles' with C:\msys64\opensearch\opensearch-1.1.0\plugins/opensearch-security/securityconfig/roles.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-10-11T22:20:20,480][INFO ][o.o.c.m.MetadataMappingService] [EC2AMAZ-0ODBHTT] [.opendistro_security/QYxJjeRMTVqMX8xEttANYg] update_mapping [_doc]
[2021-10-11T22:20:20,668][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Doc with id 'roles' and version 2 is updated in .opendistro_security index.
[2021-10-11T22:20:20,668][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Will update 'rolesmapping' with C:\msys64\opensearch\opensearch-1.1.0\plugins/opensearch-security/securityconfig/roles_mapping.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-10-11T22:20:20,683][INFO ][o.o.c.m.MetadataMappingService] [EC2AMAZ-0ODBHTT] [.opendistro_security/QYxJjeRMTVqMX8xEttANYg] update_mapping [_doc]
[2021-10-11T22:20:20,855][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Doc with id 'rolesmapping' and version 2 is updated in .opendistro_security index.
[2021-10-11T22:20:20,855][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Will update 'internalusers' with C:\msys64\opensearch\opensearch-1.1.0\plugins/opensearch-security/securityconfig/internal_users.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-10-11T22:20:20,871][INFO ][o.o.c.m.MetadataMappingService] [EC2AMAZ-0ODBHTT] [.opendistro_security/QYxJjeRMTVqMX8xEttANYg] update_mapping [_doc]
[2021-10-11T22:20:21,043][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Doc with id 'internalusers' and version 2 is updated in .opendistro_security index.
[2021-10-11T22:20:21,043][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Will update 'actiongroups' with C:\msys64\opensearch\opensearch-1.1.0\plugins/opensearch-security/securityconfig/action_groups.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-10-11T22:20:21,058][INFO ][o.o.c.m.MetadataMappingService] [EC2AMAZ-0ODBHTT] [.opendistro_security/QYxJjeRMTVqMX8xEttANYg] update_mapping [_doc]
[2021-10-11T22:20:21,230][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Doc with id 'actiongroups' and version 2 is updated in .opendistro_security index.
[2021-10-11T22:20:21,230][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Will update 'tenants' with C:\msys64\opensearch\opensearch-1.1.0\plugins/opensearch-security/securityconfig/tenants.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-10-11T22:20:21,246][INFO ][o.o.c.m.MetadataMappingService] [EC2AMAZ-0ODBHTT] [.opendistro_security/QYxJjeRMTVqMX8xEttANYg] update_mapping [_doc]
[2021-10-11T22:20:21,402][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Doc with id 'tenants' and version 2 is updated in .opendistro_security index.
[2021-10-11T22:20:21,402][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Will update 'nodesdn' with C:\msys64\opensearch\opensearch-1.1.0\plugins/opensearch-security/securityconfig/nodes_dn.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2021-10-11T22:20:21,418][INFO ][o.o.c.m.MetadataMappingService] [EC2AMAZ-0ODBHTT] [.opendistro_security/QYxJjeRMTVqMX8xEttANYg] update_mapping [_doc]
[2021-10-11T22:20:21,590][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Doc with id 'nodesdn' and version 2 is updated in .opendistro_security index.
[2021-10-11T22:20:21,590][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Will update 'whitelist' with C:\msys64\opensearch\opensearch-1.1.0\plugins/opensearch-security/securityconfig/whitelist.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=true
[2021-10-11T22:20:21,590][INFO ][o.o.c.m.MetadataMappingService] [EC2AMAZ-0ODBHTT] [.opendistro_security/QYxJjeRMTVqMX8xEttANYg] update_mapping [_doc]
[2021-10-11T22:20:21,746][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Doc with id 'whitelist' and version 2 is updated in .opendistro_security index.
[2021-10-11T22:20:21,746][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Will update 'audit' with C:\msys64\opensearch\opensearch-1.1.0\plugins/opensearch-security/securityconfig/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-10-11T22:20:21,777][INFO ][o.o.c.m.MetadataMappingService] [EC2AMAZ-0ODBHTT] [.opendistro_security/QYxJjeRMTVqMX8xEttANYg] update_mapping [_doc]
[2021-10-11T22:20:21,933][INFO ][o.o.s.s.ConfigHelper     ] [EC2AMAZ-0ODBHTT] Doc with id 'audit' and version 2 is updated in .opendistro_security index.
[2021-10-11T22:20:22,077][INFO ][stdout                   ] [EC2AMAZ-0ODBHTT] [FINE] No subscribers registered for event class org.opensearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2021-10-11T22:20:22,077][INFO ][stdout                   ] [EC2AMAZ-0ODBHTT] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2021-10-11T22:20:22,077][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing on REST API is enabled.
[2021-10-11T22:20:22,077][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2021-10-11T22:20:22,077][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing on Transport API is enabled.
[2021-10-11T22:20:22,077][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2021-10-11T22:20:22,077][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing of request body is enabled.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Bulk requests resolution is disabled during request auditing.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Index resolution is enabled during request auditing.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Sensitive headers auditing is enabled.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing requests from kibanaserver users is disabled.
[2021-10-11T22:20:22,093][WARN ][o.o.s.a.r.AuditMessageRouter] [EC2AMAZ-0ODBHTT] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing of external configuration is disabled.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing of internal configuration is enabled.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing only metadata information for read request is enabled.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing will watch {} for read requests.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing read operation requests from kibanaserver users is disabled.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing only metadata information for write request is enabled.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing diffs for write requests is disabled.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing write operation requests from kibanaserver users is disabled.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Auditing will watch <NONE> for write requests.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] .opendistro_security is used as internal security index.
[2021-10-11T22:20:22,093][INFO ][o.o.s.a.i.AuditLogImpl   ] [EC2AMAZ-0ODBHTT] Internal index used for posting audit logs is null
[2021-10-11T22:20:22,093][INFO ][o.o.s.c.ConfigurationRepository] [EC2AMAZ-0ODBHTT] Hot-reloading of audit configuration is enabled
[2021-10-11T22:20:22,093][INFO ][o.o.s.c.ConfigurationRepository] [EC2AMAZ-0ODBHTT] Node 'EC2AMAZ-0ODBHTT' initialized


# curl https://localhost:9200 -u admin:admin --insecure
{
  "name" : "EC2AMAZ-0ODBHTT",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "0IWTz4_WTqa5PjQQEWz5dw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.1.0",
    "build_type" : "tar",
    "build_hash" : "15e9f137622d878b79103df8f82d78d782b686a1",
    "build_date" : "2021-10-04T21:29:03.079792Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}



# uname -a
MSYS_NT-10.0-17763 EC2AMAZ-0ODBHTT 3.2.0-340.x86_64 2021-08-02 16:30 UTC x86_64 Msys

@dblock
Copy link
Member

dblock commented Oct 12, 2021

Let's work this POC into our build process.

Looking at https://github.com/opensearch-project/opensearch-build/pull/733/files we can make a few minimal changes to produce a Windows zip during the assemble step.

The Windows release are based on the LINUX tarball. Here are the steps to produce a OpenSearch Windows Zip.

  1. Download the latest tarball from opensearch.org downloads page.

a) If we want to use released artifacts, then we should be using the build manifest that was produced with the released build, and remove the components that don't work on windows from that. If we want to be super clean, add support for exclusions to assemble, e.g. assemble.sh --exclude k-nn --exclude ....

b) If we want to build from source, create another manifest (e.g. opensearch-windows-1.1.0.yml) that doesn't have the components that are currently not supported on Windows and add it to manifests/1.1.0. Calling build.sh manifests/1.1.0/opensearch-windows-1.1.0.yml will produce the right set of components for windows, from source, with no code changes.

  1. Untar the tarball.

Nothing to do, (1).

  1. Download the latest release of Windows JDK14 from AdoptOpenJDK here

a) The code that extracts the min tarball needs to also run a custom post-install script that will delete the JDK and install a Windows JDK. We already have custom install scripts run for each plugin, we need to add this same code as a post-step to the min bundle.

b) Longer term, we'll want to support building a -min no-JDK bundle on Linux targeting Windows and include that in the manifest.

  1. Extract the archive, rename the folder to jdk and replace it with the bundled jdk folder from the OpenSearch LINUX tarball.

Done in (3).

  1. Go into the bin folder, and run ./opensearch-plugin remove <Plugin Folder Name> to remove opensearch-knn and opensearch-performance-analyzer.

Nothing to do, see (1).

  1. Remove knn and performance analyzer related configurations from opensearch-tar-install.sh, but it is optional, as it will not stop OpenSearch from launching on Windows.

Nothing to do, see (1).

  1. You can start OpenSearch on Windows either by running opensearch-tar-install.sh or by running bin/opensearch from Shell.

This works.

  1. If you want to start OpenSearch by running .bat files, you need to make some tweaks based on the OpenDistro for Elasticsearch Windows Zip here.
    You can unzip the archive, copy all the .bat files from ODFE bin folder to OpenSearch bin folder. Make corresponding tweaks, and make them run for OpenSearch.

Can bring any .bat files along the way in (3).

...

Modify README to say to run build.sh ... followed by assemble.sh with the right parameters to produce a Windows zip.

Even if we produce a Windows bundle we have no guarantees it works. So I think we want to build Windows from source more than we want what's proposed above, on Windows hardware. Run build.sh manifests/1.1.0/opensearch-1.1.0.yml on Windows and open/fix any bugs. This would produce a native Windows bundle. Each component should also be adding Windows in their CI and we should be adding native support in plugins such as k-nn for Windows. This is obviously more work, and we can do both.

@dblock
Copy link
Member

dblock commented Oct 12, 2021

@peternied Do we have a permalink to the public location for the released 1.1.0 build manifest.yml?

@jcgraybill
Copy link
Contributor

Both the OpenSearch and OpenSearch Dashboards repos have Windows build targets, that produce installable Windows artifacts. Rather than start with the Linux build and reconfigure it for Windows, we should be starting with those build artifacts and adding the plugins and enhancements that we know work in Windows.

@dblock dblock mentioned this issue Oct 14, 2021
1 task
@abhinavGupta16 abhinavGupta16 added the v1.2.0 OpenSearch 1.2 version label Oct 18, 2021
@dblock
Copy link
Member

dblock commented Oct 19, 2021

I'm working on fixing this repo to work on Windows in https://github.com/dblock/opensearch-build/tree/fix-windows

@dblock
Copy link
Member

dblock commented Oct 20, 2021

CI for this repo fixed for Windows in #767 (+ #772 and #768).

@CEHENKLE
Copy link
Member

CEHENKLE commented Sep 27, 2022

It looks like we're making progress on the Windows distribution, but I am a bit concerned that we're not adding Windows CI to OpenSearch core itself, which is opensearch-project/OpenSearch#1448. cc: @CEHENKLE

@bbarani / @peterzhuamazon -- What date would we need this by? (flagging for @minalsha)

@dblock
Copy link
Member

dblock commented Sep 27, 2022

It looks like we're making progress on the Windows distribution, but I am a bit concerned that we're not adding Windows CI to OpenSearch core itself, which is opensearch-project/OpenSearch#1448. cc: @CEHENKLE

@bbarani / @peterzhuamazon -- What date would we need this by? (flagging for @minalsha)

I think this is a must have if we're going to be making any official distribution for Windows. We have to ensure that tests pass.

@peterzhuamazon
Copy link
Member

I am a bit confused if you are talking about windows snapshot in build repo, it is currently commented out because of some bugs I am fixing, but it is able to run tho.

if you are talking about core repo then I am not doing anything there.

Thanks.

@dblock
Copy link
Member

dblock commented Sep 28, 2022

@peterzhuamazon This is exactly what I am raising - the distribution build for windows is working, but we aren't running tests on windows in OpenSearch core.

@dblock dblock added the v2.4.0 'Issues and PRs related to version v2.4.0' label Sep 29, 2022
@peterzhuamazon
Copy link
Member

peterzhuamazon commented Sep 29, 2022

Windows Issues Board:

Non-blocker (probably):
* opensearch-project/OpenSearch-Dashboards#2764

@peterzhuamazon peterzhuamazon added Meta campaign Parent issues of OpenSearch release campaigns. labels Sep 29, 2022
@rursprung
Copy link
Contributor

rursprung commented Oct 3, 2022

how do you run the jenkins builds on windows? this line here:

[ -z "$PLATFORM" ] && PLATFORM=$(uname -s | awk '{print tolower($0)}')

doesn't return windows for me when running it in mingw64 (part of Git for Windows):

$ uname -s
MINGW64_NT-10.0-19042

though it's probably not an issue as you're explicitly already passing it in here:

nevertheless, it might be worth it to explicitly handle windows in build.sh so that the script can be also be run manually w/o having to explicitly set the platform on windows?

PS: unrelated, but something i just learnt and wanted to share 🙂: you can avoid using awk here by using ${PLATFORM,,} to turn it to lowercase (see e.g. this article)

@dblock
Copy link
Member

dblock commented Oct 3, 2022

how do you run the jenkins builds on windows?

That build.sh is called from the build workflow which takes an input manifest and supports --platform.

@anirudha
Copy link
Contributor

do we have a Dashboards artifact for windows to test ?

@bbarani
Copy link
Member Author

bbarani commented Oct 17, 2022

do we have a Dashboards artifact for windows to test ?

Dashboards core artifact generation for Windows is currently blocked due to below issues

opensearch-project/OpenSearch-Dashboards#2554
opensearch-project/OpenSearch-Dashboards#2026

@peternied
Copy link
Member

@bbarani @CEHENKLE @davidlago We've got an parity support question for this upcoming release.

In the Security Plugin there is support for using OpenSSL instead of the Java's native SSLEngine, this features defaults to off and is only supported when running JDK11. It works with no issues on Linux and MacOS platforms running JDK11.

Windows is non-trivially different and we have yet to get the feature working on this platform (~2 dev days of effort). Based on its complexity, and redundancy; I am of the opinion that we should not consider this a release blocker windows platform. We instead will update the feature to be disabled on Windows and update the documentation accordingly. Post 2.4.0 if there is interest, support for OpenSSL could be added.

Alternatively, if we focus on supporting OpenSSL for windows on JDK 11 I have low confidence we will deliver this feature for the feature freeze date of 11/03, we can have a better completion estimate for the EOD of Monday 11/01.

What do you think of dropping OpenSSL support for the Windows platform in 2.4.0?

[1] opensearch-project/security#2195

@davidlago
Copy link

My $0.02 is to not block 2.4 on Windows for this support, as it is something that is not universally supported (across JDK versions) for our other distributions and there is an alternative SSL engine. Also, we never had a Windows release, supporting OpenSSL to begin with, so this is not technically "dropping" support potentially breaking some folks who were already relying on it.

@peternied
Copy link
Member

What do you think of dropping OpenSSL support for the Windows platform in 2.4.0?

Circling back, turns out OpenSSL does not work on linux platforms either and hasn't for an indeterminate amount of time, this is not a platform specific issue anymore and so should have no impact on the 2.4.0 release. Thanks!

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Nov 6, 2022

We have windows release candidate prepared for release on 2.4.0:

Thanks.

@peterzhuamazon
Copy link
Member

Official Windows 2.4.0 ZIP is released now:

Thanks.

@bbarani
Copy link
Member Author

bbarani commented Nov 17, 2022

@peterzhuamazon Please close this issue and create separate issues for pending action items.

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Dec 14, 2022

Our 1.3.7 Version start supporting Windows ZIP for 1.x line:

Thanks.

@bbarani
Copy link
Member Author

bbarani commented Jan 10, 2023

Closing this issue as we have added Windows support to both 1.x and 2.x versions.

@bbarani bbarani closed this as completed Jan 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
campaign Parent issues of OpenSearch release campaigns. distributions enhancement New Enhancement Meta triaged This issue has been reviewed by the triage team v2.4.0 'Issues and PRs related to version v2.4.0' windows
Projects
None yet
Development

No branches or pull requests