Skip to content

OpenStack SSL Reverse Proxy Findings

Jump to bottom Edit New page
Brett Campbell edited this page May 1, 2014 · 7 revisions

Securing OpenStack w/ SSL

This is a brain dump / action list about trying to shore up the issues given to us by the security folks around the API endpoints.

We are also in contact with the author of this blog post to see what they know, and coordinate some effort in Atlanta.

Here's the original gist.

HAProxy Changes

Regardless of how we decide to secure the APIs, we need to change HAP to do pass thru and use the ssl-hello checks:

mode tcps
option ssl-hello-chk

Native SSL

We have not (but should) try the native SSL in Icehouse. There are known issues with native ssl around cinder/glance in Havana, but these may very well be related to the hardcoded protocol issues we've found along the way. We should also try the native ssl just to get a fresh status of the issues and get that into this document; even if native SSL is slower.

Proxy Terminated SSL

We have prototyped much of what it takes to get nearly everything working via command line and horizon using proxy terminated ssl in front of each of the API endpoints. The endpoints are bound to localhost:port, and then we drop in a mod_proxy vhost config into apache sites-enabled for that service at ip:port using the following config:

# must match the CN openstack.pem is signed with
ServerName 172.16.0.125

# local eth0 IP
Listen 172.16.0.244:9292

<VirtualHost 172.16.0.244:9292>
    # same as ServerName above
    ServerName            172.16.0.125

    SSLEngine             On
    SSLProtocol           +SSLv3 +TLSv1
    SSLCipherSuite        HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
    SSLCACertificateFile  /etc/ssl/certs/openstack_ca.pem
    SSLCertificateFile    /etc/ssl/certs/openstack.pem
    SSLCertificateKeyFile /etc/ssl/private/openstack.key

    # custom header for Paste SSLMiddleware
    RequestHeader   set   X-Forwarded-Proto "https"

    ProxyRequests off
    ProxyPreserveHost on
    ProxyPass / http://127.0.0.1:9292/
    ProxyPassReverse / http://127.0.0.1:9292/

    LogLevel warn
    ErrorLog /var/log/glance/api_error.log
    CustomLog /var/log/glance/api_access.log combined
</VirtualHost>

The ServerName directive is only needed if the service certificate is signed with a different name than the actual hostname (for example, the admin may sign the cert with CN "os_prod_ep1" and then configure all clients to use https://os_prod_ep1:XXXX, which can be controlled through DNS).

The following apache modules are required in the recipes using the apache2 cookbook:

  • ssl
  • proxy
  • proxy_http
  • headers

At the same time, we disable the code to stop dropping in the mod_wsgi files, and ensure all of the api services and apache get restart notifications and such.

The following files contain the localhost binding changes:

  • ceilometer/ceilometer.conf: host = localhost

  • cinder/cinder.conf: osapi_volume_listen = localhost

  • glance/glance-registry.conf: bind_host = localhost

  • glance/glance-api.conf: bind_host = localhost

  • neutron/neutron.conf: bind_host = localhost

  • nova/nova.conf: osapi_compute_listen = localhost

  • nova/nova.conf: ec2_listen = localhost

The following files contain the mod_proxy vhost setups in /etc/apache2/sites-enabled mirroring the above sample config:

  • openstack-ceilometer-api
  • openstack-dashboard
  • openstack-glance-registry
  • openstack-nova-ec2api
  • openstack-cinder-api
  • openstack-glance-api
  • openstack-neutron-server
  • openstack-nova-osapi

A refactor of some of this generic work has been refactorered into the openstack-ssl cookbook.

Trusted SSL Connections

In an ideal installation, the common name of the ssl certs will match the vip, or the dns name assigned to the vip/api endpoints as configured in Keystone. In this case, all one has to do is drop the certs into the correct place, rehash them in to the list of trusted ca certs, and drop the appropriate attributes into the cookbooks.

/etc/ssl/certs/openstack_ca.pem
/etc/ssl/certs/openstack.pem
/etc/ssl/private/openstack.key

$ sudo c_rehash /etc/ssl/certs

Finally, add the CA cert to your openrc:

export OS_CACERT=/etc/ssl/certs/openstack_ca.pem

At this time, there are a ton of attributes in all of the different cookbooks for all of the different services. Those aren't listed here because I'm sure they're different in StackForge proper (where we are headed now).

These are some of the config file changes that have to be made to inform the services what ca file to use when talking directly to other services (they should add this attr to the endpoints attrs in keystone!):

  • glance-api.conf: registry_client_protocol = https, registry_client_ca_file = /etc/ssl/certs/openstack_ca.pem (and registry, cache)

  • Controller and Compute: nova.conf: glance_api_servers=https://ip:port (this originally has no protocol)

  • Compute only: nova.conf: neutron_ca_certificates_file = /etc/ssl/certs/openstack_ca.pem

Insecure SSL (Common Name Mismatch)

If you want/need to run SSL temrination with a certificate that is self signed w/o trusting it at the host level, or where the common name can't/doesn't match the vip/ip used for the API themselves, you will need to to set various *_insecure = True flags. These include:

  • glance-api.conf: registry_client_protocol=https, register_client_insecure=True, cinder_api_insecure=True

  • glance-cache.conf: registry_client_protocol=https, register_client_insecure=True, cinder_api_insecure=True

  • glance-registry.conf: registry_client_protocol=https, register_client_insecure=True, cinder_api_insecure=True

  • cinder.conf: nova_api_insecure=True, glance_api_insecure=True glance_host protocol option is hardcoded, needs upstream love as it has no glance_protocol, etc like nova does. :-(

  • nova.conf: cinder_api_insecure=True, glance_api_insecure=True, neutron_api_insecure=True glance_api_servers needs https:// in the template `

  • neutron.conf: nova_api_insecure=True This is Icehouse only.

This also means that when using the command line tools, you will have to also pass the --insecure flag:

nova list

ERROR: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

nova --insecure list
...

Upstream Issues

WGSI Middleware

Heat in Icehouse has added a SSL middleware class to ensure the X-Forwarded-Proto HTTP header passed in from proxies properly configures the wsgi-url_scheme environment setting. This ensures that all self referencing urls in outbound json responses contain the proper scheme.

As an exmaple, here is the middleware for the glance api in /usr/lib/python2.7/dist-packages/glance/api/middleware/ssl.py:

from oslo.config import cfg
import webob.dec
import webob.exc

from glance.common import wsgi
import glance.openstack.common.log as logging

LOG = logging.getLogger(__name__)

ssl_middleware_opts = [
    cfg.StrOpt('secure_proxy_ssl_header',
               default='X-Forwarded-Proto',
               help="The HTTP Header that will be used to determine which "
                    "the original request protocol scheme was, even if it was "
                    "removed by an SSL terminator proxy.")
]
cfg.CONF.register_opts(ssl_middleware_opts)


class SSLMiddleware(wsgi.Middleware):
    """A middleware that replaces the request wsgi.url_scheme environment
    variable with the value of HTTP header configured in
    secure_proxy_ssl_header if exists in the incoming request.
    This is useful if the server is behind a SSL termination proxy.
    """
    def __init__(self, application):
        LOG.info("Initialized ssl middleware")
        super(SSLMiddleware, self).__init__(application)
        self.secure_proxy_ssl_header = 'HTTP_{0}'.format(
            cfg.CONF.secure_proxy_ssl_header.upper().replace('-', '_'))

    @webob.dec.wsgify(RequestClass=wsgi.Request)
    def __call__(self, req):
        req.environ['wsgi.url_scheme'] = req.environ.get(
            self.secure_proxy_ssl_header, req.environ['wsgi.url_scheme'])
        return self.application

And the changes to the glance-api-paste.ini:

[pipeline:glance-api-keystone]
pipeline = ssl versionnegotiation authtoken context rootapp

[filter:ssl]
paste.filter_factory = glance.api.middleware.ssl:SSLMiddleware.factory

As an example, here's a request to the native non-ssl Nova API:

curl localhost:8774

{"versions": [{"status": "CURRENT", "updated": "2011-01-21T11:33:21Z", "id": "v2.0", "links": [{"href": "http://localhost:8774/v2/", "rel": "self"}]}]}r

Here is the same request to the ssl terminated reverse proxy, which in turn calls the native service without the middleware in place (note the returned URL is still http):

curl https://claco-controller1:8774

{"versions": [{"status": "CURRENT", "updated": "2011-01-21T11:33:21Z", "id": "v2.0", "links": [{"href": "http://claco-controller1:8774/v2/", "rel": "self"}]}]}

and with the middleware in place (the returned URL is now https):

curl https://claco-controller1:8774

{"versions": [{"status": "CURRENT", "updated": "2011-01-21T11:33:21Z", "id": "v2.0", "links": [{"href": "https://claco-controller1:8774/v2/", "rel": "self"}]}]}

The middleware has not been backported to Havana, nor has it been added to an other API services WSGI at the time of this writing. This will require us to get these into upstream proper like. However, we have a Chef definition in the openstack-ssl cookbook called ssl_middleware that can be used to generically drop the middleware code into any OpenStack service. Eg.:

ssl_middleware "glance" do
  services ['glance-api', 'glance-registry']
  wsgi_lib "glance.common"
end

SSL middleware code has worked "out of the box" for the following services, with the exception of Neutron API (which seems to ignore the wsgi.url_scheme setting entirely):

  • neutron (needs tweaking)
  • glance
  • cinder
  • nova

Upstream Bugs

Add a custom footer
Add a custom sidebar
Clone this wiki locally