Py2 and Py3 compatibility for metaclass interfaces

* ServerTLSUpdater and InstallerSpecificUpdater implementation

* Fixed tests and added disables for linter :/

* Added error logging for misconfigurationerror from plugin check

* Remove redundant parameter from interfaces

* Renaming the interfaces

* Finalize interface renaming and move tests to own file

* Refactored the runners

* Refactor the cli params

* Fix the interface args

* Fixed documentation

* Documentation and naming fixes

* Remove ServerTLSConfigurationUpdater

* Remove unnecessary linter disable

* Rename run_renewal_updaters to run_generic_updaters

* Do not raise exception, but make log message more informative and visible for the user

* Run renewal deployer before installer restart

.gitignore

@@ -38,6 +38,7 @@ tests/letstest/venv/
 
 # pytest cache
 .cache
+.mypy_cache/
 
 # docker files
 .docker

.travis.yml

@@ -13,7 +13,11 @@ before_script:
 matrix:
   include:
     - python: "2.7"
-      env: TOXENV=py27_install BOULDER_INTEGRATION=1
+      env: TOXENV=py27_install BOULDER_INTEGRATION=v1
+      sudo: required
+      services: docker
+    - python: "2.7"
+      env: TOXENV=py27_install BOULDER_INTEGRATION=v2
       sudo: required
       services: docker
     - python: "2.7"
@@ -25,16 +29,14 @@ matrix:
       addons:
     - python: "2.7"
       env: TOXENV=lint
-    - python: "2.6"
-      env: TOXENV=py26
-      sudo: required
-      services: docker
+    - python: "3.5"
+      env: TOXENV=mypy
     - python: "2.7"
-      env: TOXENV=py27-oldest
+      env: TOXENV='py27-{acme,apache,certbot,dns,nginx}-oldest'
       sudo: required
       services: docker
-    - python: "3.3"
-      env: TOXENV=py33
+    - python: "3.4"
+      env: TOXENV=py34
       sudo: required
       services: docker
     - python: "3.6"

CHANGELOG.md

@@ -2,6 +2,153 @@
 
 Certbot adheres to [Semantic Versioning](http://semver.org/).
 
+## 0.23.0 - 2018-04-04
+
+### Added
+
+* Support for OpenResty was added to the Nginx plugin.
+
+### Changed
+
+* The timestamps in Certbot's logfiles now use the system's local time zone
+  rather than UTC.
+* Certbot's DNS plugins that use Lexicon now rely on Lexicon>=2.2.1 to be able
+  to create and delete multiple TXT records on a single domain.
+* certbot-dns-google's test suite now works without an internet connection.
+
+### Fixed
+
+* Removed a small window that if during which an error occurred, Certbot
+  wouldn't clean up performed challenges.
+* The parameters `default` and `ipv6only` are now removed from `listen`
+  directives when creating a new server block in the Nginx plugin.
+* `server_name` directives enclosed in quotation marks in Nginx are now properly
+  supported.
+* Resolved an issue preventing the Apache plugin from starting Apache when it's
+  not currently running on RHEL and Gentoo based systems.
+
+Despite us having broken lockstep, we are continuing to release new versions of
+all Certbot components during releases for the time being, however, the only
+packages with changes other than their version number were:
+
+* certbot
+* certbot-apache
+* certbot-dns-cloudxns
+* certbot-dns-dnsimple
+* certbot-dns-dnsmadeeasy
+* certbot-dns-google
+* certbot-dns-luadns
+* certbot-dns-nsone
+* certbot-dns-rfc2136
+* certbot-nginx
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/50?closed=1
+
+## 0.22.2 - 2018-03-19
+
+### Fixed
+
+* A type error introduced in 0.22.1 that would occur during challenge cleanup
+  when a Certbot plugin raises an exception while trying to complete the
+  challenge was fixed.
+
+Despite us having broken lockstep, we are continuing to release new versions of
+all Certbot components during releases for the time being, however, the only
+packages with changes other than their version number were:
+
+* certbot
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/53?closed=1
+
+## 0.22.1 - 2018-03-19
+
+### Changed
+
+* The ACME server used with Certbot's --dry-run and --staging flags is now
+  Let's Encrypt's ACMEv2 staging server which allows people to also test ACMEv2
+  features with these flags.
+
+### Fixed
+
+* The HTTP Content-Type header is now set to the correct value during
+  certificate revocation with new versions of the ACME protocol.
+* When using Certbot with Let's Encrypt's ACMEv2 server, it would add a blank
+  line to the top of chain.pem and between the certificates in fullchain.pem
+  for each lineage. These blank lines have been removed.
+* Resolved a bug that caused Certbot's --allow-subset-of-names flag not to
+  work.
+* Fixed a regression in acme.client.Client that caused the class to not work
+  when it was initialized without a ClientNetwork which is done by some of the
+  other projects using our ACME library.
+
+Despite us having broken lockstep, we are continuing to release new versions of
+all Certbot components during releases for the time being, however, the only
+packages with changes other than their version number were:
+
+* acme
+* certbot
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/51?closed=1
+
+## 0.22.0 - 2018-03-07
+
+### Added
+
+* Support for obtaining wildcard certificates and a newer version of the ACME
+  protocol such as the one implemented by Let's Encrypt's upcoming ACMEv2
+  endpoint was added to Certbot and its ACME library. Certbot still works with
+  older ACME versions and will automatically change the version of the protocol
+  used based on the version the ACME CA implements.
+* The Apache and Nginx plugins are now able to automatically install a wildcard
+  certificate to multiple virtual hosts that you select from your server
+  configuration.
+* The `certbot install` command now accepts the `--cert-name` flag for
+  selecting a certificate.
+* `acme.client.BackwardsCompatibleClientV2` was added to Certbot's ACME library
+  which automatically handles most of the differences between new and old ACME
+  versions. `acme.client.ClientV2` is also available for people who only want
+  to support one version of the protocol or want to handle the differences
+  between versions themselves.
+* certbot-auto now supports the flag --install-only which has the script
+  install Certbot and its dependencies and exit without invoking Certbot.
+* Support for issuing a single certificate for a wildcard and base domain was
+  added to our Google Cloud DNS plugin. To do this, we now require your API
+  credentials have additional permissions, however, your credentials will
+  already have these permissions unless you defined a custom role with fewer
+  permissions than the standard DNS administrator role provided by Google.
+  These permissions are also only needed for the case described above so it
+  will continue to work for existing users. For more information about the
+  permissions changes, see the documentation in the plugin.
+
+### Changed
+
+* We have broken lockstep between our ACME library, Certbot, and its plugins.
+  This means that the different components do not need to be the same version
+  to work together like they did previously. This makes packaging easier
+  because not every piece of Certbot needs to be repackaged to ship a change to
+  a subset of its components.
+* Support for Python 2.6 and Python 3.3 has been removed from ACME, Certbot,
+  Certbot's plugins, and certbot-auto. If you are using certbot-auto on a RHEL
+  6 based system, it will walk you through the process of installing Certbot
+  with Python 3 and refuse to upgrade to a newer version of Certbot until you
+  have done so.
+* Certbot's components now work with older versions of setuptools to simplify
+  packaging for EPEL 7.
+
+### Fixed
+
+* Issues caused by Certbot's Nginx plugin adding multiple ipv6only directives
+  has been resolved.
+* A problem where Certbot's Apache plugin would add redundant include
+  directives for the TLS configuration managed by Certbot has been fixed.
+* Certbot's webroot plugin now properly deletes any directories it creates.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/48?closed=1
+
 ## 0.21.1 - 2018-01-25
 
 ### Fixed

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:2-alpine
+FROM python:2-alpine3.7
 
 ENTRYPOINT [ "certbot" ]
 EXPOSE 80 443

acme/acme/__init__.py

@@ -10,13 +10,3 @@ supported version: `draft-ietf-acme-01`_.
   https://github.com/ietf-wg-acme/acme/tree/draft-ietf-acme-acme-01
 
 """
-import sys
-import warnings
-
-for (major, minor) in [(2, 6), (3, 3)]:
-    if sys.version_info[:2] == (major, minor):
-        warnings.warn(
-                "Python {0}.{1} support will be dropped in the next release of "
-                "acme. Please upgrade your Python version.".format(major, minor),
-                DeprecationWarning,
-        ) #pragma: no cover

acme/acme/client.py

@@ -211,7 +211,7 @@ class ClientBase(object):  # pylint: disable=too-many-instance-attributes
             response, authzr.body.identifier, authzr.uri)
         return updated_authzr, response
 
-    def revoke(self, cert, rsn):
+    def _revoke(self, cert, rsn, url):
         """Revoke certificate.
 
         :param .ComparableX509 cert: `OpenSSL.crypto.X509` wrapped in
@@ -219,14 +219,15 @@ class ClientBase(object):  # pylint: disable=too-many-instance-attributes
 
         :param int rsn: Reason code for certificate revocation.
 
+        :param str url: ACME URL to post to
+
         :raises .ClientError: If revocation is unsuccessful.
 
         """
-        response = self._post(self.directory[messages.Revocation],
-                                 messages.Revocation(
-                                     certificate=cert,
-                                     reason=rsn),
-                                 content_type=None)
+        response = self._post(url,
+                              messages.Revocation(
+                                certificate=cert,
+                                reason=rsn))
         if response.status_code != http_client.OK:
             raise errors.ClientError(
                 'Successful revocation must return HTTP OK status')
@@ -258,11 +259,12 @@ class Client(ClientBase):
         """
         # pylint: disable=too-many-arguments
         self.key = key
-        self.net = ClientNetwork(key, alg=alg, verify_ssl=verify_ssl) if net is None else net
+        if net is None:
+            net = ClientNetwork(key, alg=alg, verify_ssl=verify_ssl)
 
         if isinstance(directory, six.string_types):
             directory = messages.Directory.from_json(
-                self.net.get(directory).json())
+                net.get(directory).json())
         super(Client, self).__init__(directory=directory,
             net=net, acme_version=1)
 
@@ -308,9 +310,17 @@ class Client(ClientBase):
         :returns: Authorization Resource.
         :rtype: `.AuthorizationResource`
 
+        :raises errors.WildcardUnsupportedError: if a wildcard is requested
+
         """
         if new_authzr_uri is not None:
             logger.debug("request_challenges with new_authzr_uri deprecated.")
+
+        if identifier.value.startswith("*"):
+            raise errors.WildcardUnsupportedError(
+                "Requesting an authorization for a wildcard name is"
+                " forbidden by this version of the ACME protocol.")
+
         new_authz = messages.NewAuthorization(identifier=identifier)
         response = self._post(self.directory.new_authz, new_authz)
         # TODO: handle errors
@@ -331,6 +341,8 @@ class Client(ClientBase):
         :returns: Authorization Resource.
         :rtype: `.AuthorizationResource`
 
+        :raises errors.WildcardUnsupportedError: if a wildcard is requested
+
         """
         return self.request_challenges(messages.Identifier(
             typ=messages.IDENTIFIER_FQDN, value=domain), new_authzr_uri)
@@ -528,6 +540,18 @@ class Client(ClientBase):
                 "Recursion limit reached. Didn't get {0}".format(uri))
         return chain
 
+    def revoke(self, cert, rsn):
+        """Revoke certificate.
+
+        :param .ComparableX509 cert: `OpenSSL.crypto.X509` wrapped in
+            `.ComparableX509`
+
+        :param int rsn: Reason code for certificate revocation.
+
+        :raises .ClientError: If revocation is unsuccessful.
+
+        """
+        return self._revoke(cert, rsn, self.directory[messages.Revocation])
 
 
 class ClientV2(ClientBase):
@@ -657,6 +681,19 @@ class ClientV2(ClientBase):
                 return orderr.update(body=body, fullchain_pem=certificate_response)
         raise errors.TimeoutError()
 
+    def revoke(self, cert, rsn):
+        """Revoke certificate.
+
+        :param .ComparableX509 cert: `OpenSSL.crypto.X509` wrapped in
+            `.ComparableX509`
+
+        :param int rsn: Reason code for certificate revocation.
+
+        :raises .ClientError: If revocation is unsuccessful.
+
+        """
+        return self._revoke(cert, rsn, self.directory['revokeCert'])
+
 
 class BackwardsCompatibleClientV2(object):
     """ACME client wrapper that tends towards V2-style calls, but
@@ -725,6 +762,10 @@ class BackwardsCompatibleClientV2(object):
 
         :returns: The newly created order.
         :rtype: OrderResource
+
+        :raises errors.WildcardUnsupportedError: if a wildcard domain is
+            requested but unsupported by the ACME version
+
         """
         if self.acme_version == 1:
             csr = OpenSSL.crypto.load_certificate_request(OpenSSL.crypto.FILETYPE_PEM, csr_pem)
@@ -768,13 +809,26 @@ class BackwardsCompatibleClientV2(object):
                     'certificate, please rerun the command for a new one.')
 
             cert = OpenSSL.crypto.dump_certificate(
-                    OpenSSL.crypto.FILETYPE_PEM, certr.body.wrapped)
-            chain = crypto_util.dump_pyopenssl_chain(chain)
+                    OpenSSL.crypto.FILETYPE_PEM, certr.body.wrapped).decode()
+            chain = crypto_util.dump_pyopenssl_chain(chain).decode()
 
             return orderr.update(fullchain_pem=(cert + chain))
         else:
             return self.client.finalize_order(orderr, deadline)
 
+    def revoke(self, cert, rsn):
+        """Revoke certificate.
+
+        :param .ComparableX509 cert: `OpenSSL.crypto.X509` wrapped in
+            `.ComparableX509`
+
+        :param int rsn: Reason code for certificate revocation.
+
+        :raises .ClientError: If revocation is unsuccessful.
+
+        """
+        return self.client.revoke(cert, rsn)
+
     def _acme_version_from_directory(self, directory):
         if hasattr(directory, 'newNonce'):
             return 2

acme/acme/client_test.py

@@ -40,6 +40,7 @@ DIRECTORY_V2 = messages.Directory({
     'newAccount': 'https://www.letsencrypt-demo.org/acme/new-account',
     'newNonce': 'https://www.letsencrypt-demo.org/acme/new-nonce',
     'newOrder': 'https://www.letsencrypt-demo.org/acme/new-order',
+    'revokeCert': 'https://www.letsencrypt-demo.org/acme/revoke-cert',
 })
 
 
@@ -79,6 +80,9 @@ class ClientTestBase(unittest.TestCase):
         self.authzr = messages.AuthorizationResource(
             body=self.authz, uri=authzr_uri)
 
+        # Reason code for revocation
+        self.rsn = 1
+
 
 class BackwardsCompatibleClientV2Test(ClientTestBase):
     """Tests for  acme.client.BackwardsCompatibleClientV2."""
@@ -95,10 +99,10 @@ class BackwardsCompatibleClientV2Test(ClientTestBase):
         self.chain = [wrapped, wrapped]
 
         self.cert_pem = OpenSSL.crypto.dump_certificate(
-            OpenSSL.crypto.FILETYPE_PEM, messages_test.CERT.wrapped)
+            OpenSSL.crypto.FILETYPE_PEM, messages_test.CERT.wrapped).decode()
 
         single_chain = OpenSSL.crypto.dump_certificate(
-            OpenSSL.crypto.FILETYPE_PEM, loaded)
+            OpenSSL.crypto.FILETYPE_PEM, loaded).decode()
         self.chain_pem = single_chain + single_chain
 
         self.fullchain_pem = self.cert_pem + self.chain_pem
@@ -251,6 +255,19 @@ class BackwardsCompatibleClientV2Test(ClientTestBase):
             client.finalize_order(mock_orderr, mock_deadline)
             mock_client().finalize_order.assert_called_once_with(mock_orderr, mock_deadline)
 
+    def test_revoke(self):
+        self.response.json.return_value = DIRECTORY_V1.to_json()
+        with mock.patch('acme.client.Client') as mock_client:
+            client = self._init()
+            client.revoke(messages_test.CERT, self.rsn)
+        mock_client().revoke.assert_called_once_with(messages_test.CERT, self.rsn)
+
+        self.response.json.return_value = DIRECTORY_V2.to_json()
+        with mock.patch('acme.client.ClientV2') as mock_client:
+            client = self._init()
+            client.revoke(messages_test.CERT, self.rsn)
+        mock_client().revoke.assert_called_once_with(messages_test.CERT, self.rsn)
+
 
 class ClientTest(ClientTestBase):
     """Tests for acme.client.Client."""
@@ -271,9 +288,6 @@ class ClientTest(ClientTestBase):
             uri='https://www.letsencrypt-demo.org/acme/cert/1',
             cert_chain_uri='https://www.letsencrypt-demo.org/ca')
 
-        # Reason code for revocation
-        self.rsn = 1
-
         from acme.client import Client
         self.client = Client(
             directory=self.directory, key=KEY, alg=jose.RS256, net=self.net)
@@ -285,6 +299,16 @@ class ClientTest(ClientTestBase):
             directory=uri, key=KEY, alg=jose.RS256, net=self.net)
         self.net.get.assert_called_once_with(uri)
 
+    @mock.patch('acme.client.ClientNetwork')
+    def test_init_without_net(self, mock_net):
+        mock_net.return_value = mock.sentinel.net
+        alg = jose.RS256
+        from acme.client import Client
+        self.client = Client(
+            directory=self.directory, key=KEY, alg=alg)
+        mock_net.called_once_with(KEY, alg=alg, verify_ssl=True)
+        self.assertEqual(self.client.net, mock.sentinel.net)
+
     def test_register(self):
         # "Instance of 'Field' has no to_json/update member" bug:
         # pylint: disable=no-member
@@ -362,6 +386,13 @@ class ClientTest(ClientTestBase):
             errors.UnexpectedUpdate, self.client.request_challenges,
             self.identifier)
 
+    def test_request_challenges_wildcard(self):
+        wildcard_identifier = messages.Identifier(
+            typ=messages.IDENTIFIER_FQDN, value='*.example.org')
+        self.assertRaises(
+            errors.WildcardUnsupportedError, self.client.request_challenges,
+            wildcard_identifier)
+
     def test_request_domain_challenges(self):
         self.client.request_challenges = mock.MagicMock()
         self.assertEqual(
@@ -614,8 +645,7 @@ class ClientTest(ClientTestBase):
     def test_revoke(self):
         self.client.revoke(self.certr.body, self.rsn)
         self.net.post.assert_called_once_with(
-            self.directory[messages.Revocation], mock.ANY, content_type=None,
-            acme_version=1)
+            self.directory[messages.Revocation], mock.ANY, acme_version=1)
 
     def test_revocation_payload(self):
         obj = messages.Revocation(certificate=self.certr.body, reason=self.rsn)
@@ -752,6 +782,11 @@ class ClientV2Test(ClientTestBase):
         deadline = datetime.datetime.now() - datetime.timedelta(seconds=60)
         self.assertRaises(errors.TimeoutError, self.client.finalize_order, self.orderr, deadline)
 
+    def test_revoke(self):
+        self.client.revoke(messages_test.CERT, self.rsn)
+        self.net.post.assert_called_once_with(
+            self.directory["revokeCert"], mock.ANY, acme_version=2)
+
 
 class MockJSONDeSerializable(jose.JSONDeSerializable):
     # pylint: disable=missing-docstring

acme/acme/crypto_util.py

@@ -5,7 +5,6 @@ import logging
 import os
 import re
 import socket
-import sys
 
 import OpenSSL
 import josepy as jose
@@ -132,8 +131,7 @@ def probe_sni(name, host, port=443, timeout=300,
     context = OpenSSL.SSL.Context(method)
     context.set_timeout(timeout)
 
-    socket_kwargs = {} if sys.version_info < (2, 7) else {
-        'source_address': source_address}
+    socket_kwargs = {'source_address': source_address}
 
     host_protocol_agnostic = None if host == '::' or host == '0' else host
 
@@ -289,6 +287,9 @@ def dump_pyopenssl_chain(chain, filetype=OpenSSL.crypto.FILETYPE_PEM):
     :param list chain: List of `OpenSSL.crypto.X509` (or wrapped in
         :class:`josepy.util.ComparableX509`).
 
+    :returns: certificate chain bundle
+    :rtype: bytes
+
     """
     # XXX: returns empty string when no chain is available, which
     # shuts up RenewableCert, but might not be the best solution...

acme/acme/crypto_util_test.py

@@ -194,9 +194,9 @@ class MakeCSRTest(unittest.TestCase):
         self.assertTrue(b'--END CERTIFICATE REQUEST--' in csr_pem)
         csr = OpenSSL.crypto.load_certificate_request(
             OpenSSL.crypto.FILETYPE_PEM, csr_pem)
-        # In pyopenssl 0.13 (used with TOXENV=py26-oldest and py27-oldest), csr
-        # objects don't have a get_extensions() method, so we skip this test if
-        # the method isn't available.
+        # In pyopenssl 0.13 (used with TOXENV=py27-oldest), csr objects don't
+        # have a get_extensions() method, so we skip this test if the method
+        # isn't available.
         if hasattr(csr, 'get_extensions'):
             self.assertEquals(len(csr.get_extensions()), 1)
             self.assertEquals(csr.get_extensions()[0].get_data(),
@@ -212,9 +212,9 @@ class MakeCSRTest(unittest.TestCase):
         csr = OpenSSL.crypto.load_certificate_request(
             OpenSSL.crypto.FILETYPE_PEM, csr_pem)
 
-        # In pyopenssl 0.13 (used with TOXENV=py26-oldest and py27-oldest), csr
-        # objects don't have a get_extensions() method, so we skip this test if
-        # the method isn't available.
+        # In pyopenssl 0.13 (used with TOXENV=py27-oldest), csr objects don't
+        # have a get_extensions() method, so we skip this test if the method
+        # isn't available.
         if hasattr(csr, 'get_extensions'):
             self.assertEquals(len(csr.get_extensions()), 2)
             # NOTE: Ideally we would filter by the TLS Feature OID, but

acme/acme/errors.py

@@ -115,3 +115,6 @@ class ConflictError(ClientError):
         self.location = location
         super(ConflictError, self).__init__()
 
+
+class WildcardUnsupportedError(Error):
+    """Error for when a wildcard is requested but is unsupported by ACME CA."""

acme/acme/messages.py

@@ -435,6 +435,7 @@ class Authorization(ResourceBody):
     # be absent'... then acme-spec gives example with 'expires'
     # present... That's confusing!
     expires = fields.RFC3339Field('expires', omitempty=True)
+    wildcard = jose.Field('wildcard', omitempty=True)
 
     @challenges.decoder
     def challenges(value):  # pylint: disable=missing-docstring,no-self-argument

acme/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [
@@ -19,19 +19,10 @@ install_requires = [
     'pyrfc3339',
     'pytz',
     'requests[security]>=2.4.1',  # security extras added in 2.4.1
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'six>=1.9.0',  # needed for python_2_unicode_compatible
 ]
 
-# env markers cause problems with older pip and setuptools
-if sys.version_info < (2, 7):
-    install_requires.extend([
-        'argparse',
-        'ordereddict',
-    ])
-
 dev_extras = [
     'pytest',
     'pytest-xdist',
@@ -52,16 +43,15 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Intended Audience :: Developers',
         'License :: OSI Approved :: Apache Software License',
         'Programming Language :: Python',
         'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-apache/certbot_apache/configurator.py

@@ -5,6 +5,7 @@ import logging
 import os
 import pkg_resources
 import re
+import six
 import socket
 import time
 
@@ -152,6 +153,9 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         self.assoc = dict()
         # Outstanding challenges
         self._chall_out = set()
+        # List of vhosts configured per wildcard domain on this run.
+        # used by deploy_cert() and enhance()
+        self._wildcard_vhosts = dict()
         # Maps enhancements to vhosts we've enabled the enhancement for
         self._enhanced_vhosts = defaultdict(set)
 
@@ -262,12 +266,27 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             self.aug, self.conf("server-root"), self.conf("vhost-root"),
             self.version, configurator=self)
 
+    def _wildcard_domain(self, domain):
+        """
+        Checks if domain is a wildcard domain
+
+        :param str domain: Domain to check
+
+        :returns: If the domain is wildcard domain
+        :rtype: bool
+        """
+        if isinstance(domain, six.text_type):
+            wildcard_marker = u"*."
+        else:
+            wildcard_marker = b"*."
+        return domain.startswith(wildcard_marker)
+
     def deploy_cert(self, domain, cert_path, key_path,
                     chain_path=None, fullchain_path=None):
         """Deploys certificate to specified virtual host.
 
-        Currently tries to find the last directives to deploy the cert in
-        the VHost associated with the given domain. If it can't find the
+        Currently tries to find the last directives to deploy the certificate
+        in the VHost associated with the given domain. If it can't find the
         directives, it searches the "included" confs. The function verifies
         that it has located the three directives and finally modifies them
         to point to the correct destination. After the certificate is
@@ -280,9 +299,112 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             a lack of directives
 
         """
-        # Choose vhost before (possible) enabling of mod_ssl, to keep the
-        # vhost choice namespace similar with the pre-validation one.
-        vhost = self.choose_vhost(domain)
+        vhosts = self.choose_vhosts(domain)
+        for vhost in vhosts:
+            self._deploy_cert(vhost, cert_path, key_path, chain_path, fullchain_path)
+
+    def choose_vhosts(self, domain, create_if_no_ssl=True):
+        """
+        Finds VirtualHosts that can be used with the provided domain
+
+        :param str domain: Domain name to match VirtualHosts to
+        :param bool create_if_no_ssl: If found VirtualHost doesn't have a HTTPS
+            counterpart, should one get created
+
+        :returns: List of VirtualHosts or None
+        :rtype: `list` of :class:`~certbot_apache.obj.VirtualHost`
+        """
+
+        if self._wildcard_domain(domain):
+            if domain in self._wildcard_vhosts:
+                # Vhosts for a wildcard domain were already selected
+                return self._wildcard_vhosts[domain]
+            # Ask user which VHosts to support.
+            # Returned objects are guaranteed to be ssl vhosts
+            return self._choose_vhosts_wildcard(domain, create_if_no_ssl)
+        else:
+            return [self.choose_vhost(domain, create_if_no_ssl)]
+
+    def _vhosts_for_wildcard(self, domain):
+        """
+        Get VHost objects for every VirtualHost that the user wants to handle
+        with the wildcard certificate.
+        """
+
+        # Collect all vhosts that match the name
+        matched = set()
+        for vhost in self.vhosts:
+            for name in vhost.get_names():
+                if self._in_wildcard_scope(name, domain):
+                    matched.add(vhost)
+
+        return list(matched)
+
+    def _in_wildcard_scope(self, name, domain):
+        """
+        Helper method for _vhosts_for_wildcard() that makes sure that the domain
+        is in the scope of wildcard domain.
+
+        eg. in scope: domain = *.wild.card, name = 1.wild.card
+        not in scope: domain = *.wild.card, name = 1.2.wild.card
+        """
+        if len(name.split(".")) == len(domain.split(".")):
+            return fnmatch.fnmatch(name, domain)
+
+
+    def _choose_vhosts_wildcard(self, domain, create_ssl=True):
+        """Prompts user to choose vhosts to install a wildcard certificate for"""
+
+        # Get all vhosts that are covered by the wildcard domain
+        vhosts = self._vhosts_for_wildcard(domain)
+
+        # Go through the vhosts, making sure that we cover all the names
+        # present, but preferring the SSL vhosts
+        filtered_vhosts = dict()
+        for vhost in vhosts:
+            for name in vhost.get_names():
+                if vhost.ssl:
+                    # Always prefer SSL vhosts
+                    filtered_vhosts[name] = vhost
+                elif name not in filtered_vhosts and create_ssl:
+                    # Add if not in list previously
+                    filtered_vhosts[name] = vhost
+
+        # Only unique VHost objects
+        dialog_input = set([vhost for vhost in filtered_vhosts.values()])
+
+        # Ask the user which of names to enable, expect list of names back
+        dialog_output = display_ops.select_vhost_multiple(list(dialog_input))
+
+        if not dialog_output:
+            logger.error(
+                "No vhost exists with servername or alias for domain %s. "
+                "No vhost was selected. Please specify ServerName or ServerAlias "
+                "in the Apache config.",
+                domain)
+            raise errors.PluginError("No vhost selected")
+
+        # Make sure we create SSL vhosts for the ones that are HTTP only
+        # if requested.
+        return_vhosts = list()
+        for vhost in dialog_output:
+            if not vhost.ssl:
+                return_vhosts.append(self.make_vhost_ssl(vhost))
+            else:
+                return_vhosts.append(vhost)
+
+        self._wildcard_vhosts[domain] = return_vhosts
+        return return_vhosts
+
+
+    def _deploy_cert(self, vhost, cert_path, key_path, chain_path, fullchain_path):
+        """
+        Helper function for deploy_cert() that handles the actual deployment
+        this exists because we might want to do multiple deployments per
+        domain originally passed for deploy_cert(). This is especially true
+        with wildcard certificates
+        """
+
 
         # This is done first so that ssl module is enabled and cert_path,
         # cert_key... can all be parsed appropriately
@@ -302,16 +424,22 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             path["chain_path"] = self.parser.find_dir(
                 "SSLCertificateChainFile", None, vhost.path)
 
-        if not path["cert_path"] or not path["cert_key"]:
-            # Throw some can't find all of the directives error"
+        # Handle errors when certificate/key directives cannot be found
+        if not path["cert_path"]:
             logger.warning(
-                "Cannot find a cert or key directive in %s. "
+                "Cannot find an SSLCertificateFile directive in %s. "
                 "VirtualHost was not modified", vhost.path)
-            # Presumably break here so that the virtualhost is not modified
             raise errors.PluginError(
-                "Unable to find cert and/or key directives")
+                "Unable to find an SSLCertificateFile directive")
+        elif not path["cert_key"]:
+            logger.warning(
+                "Cannot find an SSLCertificateKeyFile directive for "
+                "certificate in %s. VirtualHost was not modified", vhost.path)
+            raise errors.PluginError(
+                "Unable to find an SSLCertificateKeyFile directive for "
+                "certificate")
 
-        logger.info("Deploying Certificate for %s to VirtualHost %s", domain, vhost.filep)
+        logger.info("Deploying Certificate to VirtualHost %s", vhost.filep)
 
         if self.version < (2, 4, 8) or (chain_path and not fullchain_path):
             # install SSLCertificateFile, SSLCertificateKeyFile,
@@ -327,8 +455,8 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
                                          "version of Apache")
         else:
             if not fullchain_path:
-                raise errors.PluginError("Please provide the --fullchain-path\
- option pointing to your full chain file")
+                raise errors.PluginError("Please provide the --fullchain-path "
+                                         "option pointing to your full chain file")
             set_cert_path = fullchain_path
             self.aug.set(path["cert_path"][-1], fullchain_path)
             self.aug.set(path["cert_key"][-1], key_path)
@@ -347,20 +475,21 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         if chain_path is not None:
             self.save_notes += "\tSSLCertificateChainFile %s\n" % chain_path
 
-    def choose_vhost(self, target_name, temp=False):
+    def choose_vhost(self, target_name, create_if_no_ssl=True):
         """Chooses a virtual host based on the given domain name.
 
         If there is no clear virtual host to be selected, the user is prompted
         with all available choices.
 
-        The returned vhost is guaranteed to have TLS enabled unless temp is
-        True. If temp is True, there is no such guarantee and the result is
-        not cached.
+        The returned vhost is guaranteed to have TLS enabled unless
+        create_if_no_ssl is set to False, in which case there is no such guarantee
+        and the result is not cached.
 
         :param str target_name: domain name
-        :param bool temp: whether the vhost is only used temporarily
+        :param bool create_if_no_ssl: If found VirtualHost doesn't have a HTTPS
+            counterpart, should one get created
 
-        :returns: ssl vhost associated with name
+        :returns: vhost associated with name
         :rtype: :class:`~certbot_apache.obj.VirtualHost`
 
         :raises .errors.PluginError: If no vhost is available or chosen
@@ -373,7 +502,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         # Try to find a reasonable vhost
         vhost = self._find_best_vhost(target_name)
         if vhost is not None:
-            if temp:
+            if not create_if_no_ssl:
                 return vhost
             if not vhost.ssl:
                 vhost = self.make_vhost_ssl(vhost)
@@ -382,7 +511,9 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             self.assoc[target_name] = vhost
             return vhost
 
-        return self._choose_vhost_from_list(target_name, temp)
+        # Negate create_if_no_ssl value to indicate if we want a SSL vhost
+        # to get created if a non-ssl vhost is selected.
+        return self._choose_vhost_from_list(target_name, temp=not create_if_no_ssl)
 
     def _choose_vhost_from_list(self, target_name, temp=False):
         # Select a vhost from a list
@@ -391,7 +522,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             logger.error(
                 "No vhost exists with servername or alias of %s. "
                 "No vhost was selected. Please specify ServerName or ServerAlias "
-                "in the Apache config, or split vhosts into separate files.",
+                "in the Apache config.",
                 target_name)
             raise errors.PluginError("No vhost selected")
         elif temp:
@@ -1376,8 +1507,24 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         except KeyError:
             raise errors.PluginError(
                 "Unsupported enhancement: {0}".format(enhancement))
+
+        matched_vhosts = self.choose_vhosts(domain, create_if_no_ssl=False)
+        # We should be handling only SSL vhosts for enhancements
+        vhosts = [vhost for vhost in matched_vhosts if vhost.ssl]
+
+        if not vhosts:
+            msg_tmpl = ("Certbot was not able to find SSL VirtualHost for a "
+                        "domain {0} for enabling enhancement \"{1}\". The requested "
+                        "enhancement was not configured.")
+            msg_enhancement = enhancement
+            if options:
+                msg_enhancement += ": " + options
+            msg = msg_tmpl.format(domain, msg_enhancement)
+            logger.warning(msg)
+            raise errors.PluginError(msg)
         try:
-            func(self.choose_vhost(domain), options)
+            for vhost in vhosts:
+                func(vhost, options)
         except errors.PluginError:
             logger.warning("Failed %s for %s", enhancement, domain)
             raise
@@ -1869,10 +2016,27 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         :raises .errors.MisconfigurationError: If reload fails
 
         """
+        error = ""
         try:
             util.run_script(self.constant("restart_cmd"))
         except errors.SubprocessError as err:
-            raise errors.MisconfigurationError(str(err))
+            logger.info("Unable to restart apache using %s",
+                        self.constant("restart_cmd"))
+            alt_restart = self.constant("restart_cmd_alt")
+            if alt_restart:
+                logger.debug("Trying alternative restart command: %s",
+                             alt_restart)
+                # There is an alternative restart command available
+                # This usually is "restart" verb while original is "graceful"
+                try:
+                    util.run_script(self.constant(
+                        "restart_cmd_alt"))
+                    return
+                except errors.SubprocessError as secerr:
+                    error = str(secerr)
+            else:
+                error = str(err)
+            raise errors.MisconfigurationError(error)
 
     def config_test(self):  # pylint: disable=no-self-use
         """Check the configuration of Apache for errors.
@@ -1992,5 +2156,3 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         # to be modified.
         return common.install_version_controlled_file(options_ssl, options_ssl_digest,
             self.constant("MOD_SSL_CONF_SRC"), constants.ALL_SSL_OPTIONS_HASHES)
-
-

certbot-apache/certbot_apache/display_ops.py

@@ -13,10 +13,44 @@ import certbot.display.util as display_util
 logger = logging.getLogger(__name__)
 
 
+def select_vhost_multiple(vhosts):
+    """Select multiple Vhosts to install the certificate for
+
+    :param vhosts: Available Apache VirtualHosts
+    :type vhosts: :class:`list` of type `~obj.Vhost`
+
+    :returns: List of VirtualHosts
+    :rtype: :class:`list`of type `~obj.Vhost`
+    """
+    if not vhosts:
+        return list()
+    tags_list = [vhost.display_repr()+"\n" for vhost in vhosts]
+    # Remove the extra newline from the last entry
+    if len(tags_list):
+        tags_list[-1] = tags_list[-1][:-1]
+    code, names = zope.component.getUtility(interfaces.IDisplay).checklist(
+        "Which VirtualHosts would you like to install the wildcard certificate for?",
+        tags=tags_list, force_interactive=True)
+    if code == display_util.OK:
+        return_vhosts = _reversemap_vhosts(names, vhosts)
+        return return_vhosts
+    return []
+
+def _reversemap_vhosts(names, vhosts):
+    """Helper function for select_vhost_multiple for mapping string
+    representations back to actual vhost objects"""
+    return_vhosts = list()
+
+    for selection in names:
+        for vhost in vhosts:
+            if vhost.display_repr().strip() == selection.strip():
+                return_vhosts.append(vhost)
+    return return_vhosts
+
 def select_vhost(domain, vhosts):
     """Select an appropriate Apache Vhost.
 
-    :param vhosts: Available Apache Virtual Hosts
+    :param vhosts: Available Apache VirtualHosts
     :type vhosts: :class:`list` of type `~obj.Vhost`
 
     :returns: VirtualHost or `None`
@@ -25,13 +59,11 @@ def select_vhost(domain, vhosts):
     """
     if not vhosts:
         return None
-    while True:
-        code, tag = _vhost_menu(domain, vhosts)
-        if code == display_util.OK:
-            return vhosts[tag]
-        else:
-            return None
-
+    code, tag = _vhost_menu(domain, vhosts)
+    if code == display_util.OK:
+        return vhosts[tag]
+    else:
+        return None
 
 def _vhost_menu(domain, vhosts):
     """Select an appropriate Apache Vhost.

certbot-apache/certbot_apache/obj.py

@@ -167,6 +167,19 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
                 active="Yes" if self.enabled else "No",
                 modmacro="Yes" if self.modmacro else "No"))
 
+    def display_repr(self):
+        """Return a representation of VHost to be used in dialog"""
+        return (
+            "File: {filename}\n"
+            "Addresses: {addrs}\n"
+            "Names: {names}\n"
+            "HTTPS: {https}\n".format(
+                filename=self.filep,
+                addrs=", ".join(str(addr) for addr in self.addrs),
+                names=", ".join(self.get_names()),
+                https="Yes" if self.ssl else "No"))
+
+
     def __eq__(self, other):
         if isinstance(other, self.__class__):
             return (self.filep == other.filep and self.path == other.path and

certbot-apache/certbot_apache/override_centos.py

@@ -21,6 +21,7 @@ class CentOSConfigurator(configurator.ApacheConfigurator):
         version_cmd=['apachectl', '-v'],
         apache_cmd="apachectl",
         restart_cmd=['apachectl', 'graceful'],
+        restart_cmd_alt=['apachectl', 'restart'],
         conftest_cmd=['apachectl', 'configtest'],
         enmod=None,
         dismod=None,

certbot-apache/certbot_apache/override_gentoo.py

@@ -21,6 +21,7 @@ class GentooConfigurator(configurator.ApacheConfigurator):
         version_cmd=['/usr/sbin/apache2', '-v'],
         apache_cmd="apache2ctl",
         restart_cmd=['apache2ctl', 'graceful'],
+        restart_cmd_alt=['apache2ctl', 'restart'],
         conftest_cmd=['apache2ctl', 'configtest'],
         enmod=None,
         dismod=None,

certbot-apache/certbot_apache/tests/centos_test.py

@@ -4,6 +4,8 @@ import unittest
 
 import mock
 
+from certbot import errors
+
 from certbot_apache import obj
 from certbot_apache import override_centos
 from certbot_apache.tests import util
@@ -121,5 +123,17 @@ class MultipleVhostsTestCentOS(util.ApacheTest):
         self.assertTrue("MOCK_NOSEP" in self.config.parser.variables.keys())
         self.assertEqual("NOSEP_VAL", self.config.parser.variables["NOSEP_TWO"])
 
+    @mock.patch("certbot_apache.configurator.util.run_script")
+    def test_alt_restart_works(self, mock_run_script):
+        mock_run_script.side_effect = [None, errors.SubprocessError, None]
+        self.config.restart()
+        self.assertEquals(mock_run_script.call_count, 3)
+
+    @mock.patch("certbot_apache.configurator.util.run_script")
+    def test_alt_restart_errors(self, mock_run_script):
+        mock_run_script.side_effect = [None,
+                                       errors.SubprocessError,
+                                       errors.SubprocessError]
+        self.assertRaises(errors.MisconfigurationError, self.config.restart)
 if __name__ == "__main__":
     unittest.main()  # pragma: no cover

certbot-apache/certbot_apache/tests/configurator_test.py

@@ -246,7 +246,7 @@ class MultipleVhostsTest(util.ApacheTest):
     @mock.patch("certbot_apache.display_ops.select_vhost")
     def test_choose_vhost_select_vhost_with_temp(self, mock_select):
         mock_select.return_value = self.vh_truth[0]
-        chosen_vhost = self.config.choose_vhost("none.com", temp=True)
+        chosen_vhost = self.config.choose_vhost("none.com", create_if_no_ssl=False)
         self.assertEqual(self.vh_truth[0], chosen_vhost)
 
     @mock.patch("certbot_apache.display_ops.select_vhost")
@@ -441,13 +441,37 @@ class MultipleVhostsTest(util.ApacheTest):
             self.vh_truth[1].path))
 
     def test_deploy_cert_invalid_vhost(self):
+        """For test cases where the `ApacheConfigurator` class' `_deploy_cert`
+        method is called with an invalid vhost parameter. Currently this tests
+        that a PluginError is appropriately raised when important directives
+        are missing in an SSL module."""
         self.config.parser.modules.add("ssl_module")
-        mock_find = mock.MagicMock()
-        mock_find.return_value = []
-        self.config.parser.find_dir = mock_find
+        self.config.parser.modules.add("mod_ssl.c")
+        self.config.parser.modules.add("socache_shmcb_module")
+
+        def side_effect(*args):
+            """Mocks case where an SSLCertificateFile directive can be found
+            but an SSLCertificateKeyFile directive is missing."""
+            if "SSLCertificateFile" in args:
+                return ["example/cert.pem"]
+            else:
+                return []
+
+        mock_find_dir = mock.MagicMock(return_value=[])
+        mock_find_dir.side_effect = side_effect
+
+        self.config.parser.find_dir = mock_find_dir
 
         # Get the default 443 vhost
         self.config.assoc["random.demo"] = self.vh_truth[1]
+
+        self.assertRaises(
+            errors.PluginError, self.config.deploy_cert, "random.demo",
+            "example/cert.pem", "example/key.pem", "example/cert_chain.pem")
+
+        # Remove side_effect to mock case where both SSLCertificateFile
+        # and SSLCertificateKeyFile directives are missing
+        self.config.parser.find_dir.side_effect = None
         self.assertRaises(
             errors.PluginError, self.config.deploy_cert, "random.demo",
             "example/cert.pem", "example/key.pem", "example/cert_chain.pem")
@@ -912,6 +936,22 @@ class MultipleVhostsTest(util.ApacheTest):
             errors.PluginError,
             self.config.enhance, "certbot.demo", "unknown_enhancement")
 
+    def test_enhance_no_ssl_vhost(self):
+        with mock.patch("certbot_apache.configurator.logger.warning") as mock_log:
+            self.assertRaises(errors.PluginError, self.config.enhance,
+                              "certbot.demo", "redirect")
+            # Check that correct logger.warning was printed
+            self.assertTrue("not able to find" in mock_log.call_args[0][0])
+            self.assertTrue("\"redirect\"" in mock_log.call_args[0][0])
+
+            mock_log.reset_mock()
+
+            self.assertRaises(errors.PluginError, self.config.enhance,
+                              "certbot.demo", "ensure-http-header", "Test")
+            # Check that correct logger.warning was printed
+            self.assertTrue("not able to find" in mock_log.call_args[0][0])
+            self.assertTrue("Test" in mock_log.call_args[0][0])
+
     @mock.patch("certbot.util.exe_exists")
     def test_ocsp_stapling(self, mock_exe):
         self.config.parser.update_runtime_variables = mock.Mock()
@@ -921,6 +961,7 @@ class MultipleVhostsTest(util.ApacheTest):
         mock_exe.return_value = True
 
         # This will create an ssl vhost for certbot.demo
+        self.config.choose_vhost("certbot.demo")
         self.config.enhance("certbot.demo", "staple-ocsp")
 
         # Get the ssl vhost for certbot.demo
@@ -947,6 +988,7 @@ class MultipleVhostsTest(util.ApacheTest):
         mock_exe.return_value = True
 
         # Checking the case with already enabled ocsp stapling configuration
+        self.config.choose_vhost("ocspvhost.com")
         self.config.enhance("ocspvhost.com", "staple-ocsp")
 
         # Get the ssl vhost for letsencrypt.demo
@@ -971,6 +1013,7 @@ class MultipleVhostsTest(util.ApacheTest):
         self.config.parser.modules.add("mod_ssl.c")
         self.config.parser.modules.add("socache_shmcb_module")
         self.config.get_version = mock.Mock(return_value=(2, 2, 0))
+        self.config.choose_vhost("certbot.demo")
 
         self.assertRaises(errors.PluginError,
                 self.config.enhance, "certbot.demo", "staple-ocsp")
@@ -996,6 +1039,7 @@ class MultipleVhostsTest(util.ApacheTest):
         mock_exe.return_value = True
 
         # This will create an ssl vhost for certbot.demo
+        self.config.choose_vhost("certbot.demo")
         self.config.enhance("certbot.demo", "ensure-http-header",
                             "Strict-Transport-Security")
 
@@ -1015,7 +1059,8 @@ class MultipleVhostsTest(util.ApacheTest):
         # skip the enable mod
         self.config.parser.modules.add("headers_module")
 
-        # This will create an ssl vhost for certbot.demo
+        # This will create an ssl vhost for encryption-example.demo
+        self.config.choose_vhost("encryption-example.demo")
         self.config.enhance("encryption-example.demo", "ensure-http-header",
                             "Strict-Transport-Security")
 
@@ -1034,6 +1079,7 @@ class MultipleVhostsTest(util.ApacheTest):
         mock_exe.return_value = True
 
         # This will create an ssl vhost for certbot.demo
+        self.config.choose_vhost("certbot.demo")
         self.config.enhance("certbot.demo", "ensure-http-header",
                             "Upgrade-Insecure-Requests")
 
@@ -1055,7 +1101,8 @@ class MultipleVhostsTest(util.ApacheTest):
         # skip the enable mod
         self.config.parser.modules.add("headers_module")
 
-        # This will create an ssl vhost for certbot.demo
+        # This will create an ssl vhost for encryption-example.demo
+        self.config.choose_vhost("encryption-example.demo")
         self.config.enhance("encryption-example.demo", "ensure-http-header",
                             "Upgrade-Insecure-Requests")
 
@@ -1073,6 +1120,7 @@ class MultipleVhostsTest(util.ApacheTest):
         self.config.get_version = mock.Mock(return_value=(2, 2))
 
         # This will create an ssl vhost for certbot.demo
+        self.config.choose_vhost("certbot.demo")
         self.config.enhance("certbot.demo", "redirect")
 
         # These are not immediately available in find_dir even with save() and
@@ -1123,6 +1171,7 @@ class MultipleVhostsTest(util.ApacheTest):
         self.config.save()
 
         # This will create an ssl vhost for certbot.demo
+        self.config.choose_vhost("certbot.demo")
         self.config.enhance("certbot.demo", "redirect")
 
         # These are not immediately available in find_dir even with save() and
@@ -1189,6 +1238,9 @@ class MultipleVhostsTest(util.ApacheTest):
         self.config.parser.modules.add("rewrite_module")
         self.config.get_version = mock.Mock(return_value=(2, 3, 9))
 
+        # Creates ssl vhost for the domain
+        self.config.choose_vhost("red.blue.purple.com")
+
         self.config.enhance("red.blue.purple.com", "redirect")
         verify_no_redirect = ("certbot_apache.configurator."
                               "ApacheConfigurator._verify_no_certbot_redirect")
@@ -1200,7 +1252,7 @@ class MultipleVhostsTest(util.ApacheTest):
         # Skip the enable mod
         self.config.parser.modules.add("rewrite_module")
         self.config.get_version = mock.Mock(return_value=(2, 3, 9))
-
+        self.config.choose_vhost("red.blue.purple.com")
         self.config.enhance("red.blue.purple.com", "redirect")
         # Clear state about enabling redirect on this run
         # pylint: disable=protected-access
@@ -1337,6 +1389,108 @@ class MultipleVhostsTest(util.ApacheTest):
                           self.config.enable_mod,
                           "whatever")
 
+    def test_wildcard_domain(self):
+        # pylint: disable=protected-access
+        cases = {u"*.example.org": True, b"*.x.example.org": True,
+                 u"a.example.org": False, b"a.x.example.org": False}
+        for key in cases.keys():
+            self.assertEqual(self.config._wildcard_domain(key), cases[key])
+
+    def test_choose_vhosts_wildcard(self):
+        # pylint: disable=protected-access
+        mock_path = "certbot_apache.display_ops.select_vhost_multiple"
+        with mock.patch(mock_path) as mock_select_vhs:
+            mock_select_vhs.return_value = [self.vh_truth[3]]
+            vhs = self.config._choose_vhosts_wildcard("*.certbot.demo",
+                                                     create_ssl=True)
+            # Check that the dialog was called with one vh: certbot.demo
+            self.assertEquals(mock_select_vhs.call_args[0][0][0], self.vh_truth[3])
+            self.assertEquals(len(mock_select_vhs.call_args_list), 1)
+
+            # And the actual returned values
+            self.assertEquals(len(vhs), 1)
+            self.assertTrue(vhs[0].name == "certbot.demo")
+            self.assertTrue(vhs[0].ssl)
+
+            self.assertFalse(vhs[0] == self.vh_truth[3])
+
+    @mock.patch("certbot_apache.configurator.ApacheConfigurator.make_vhost_ssl")
+    def test_choose_vhosts_wildcard_no_ssl(self, mock_makessl):
+        # pylint: disable=protected-access
+        mock_path = "certbot_apache.display_ops.select_vhost_multiple"
+        with mock.patch(mock_path) as mock_select_vhs:
+            mock_select_vhs.return_value = [self.vh_truth[1]]
+            vhs = self.config._choose_vhosts_wildcard("*.certbot.demo",
+                                                     create_ssl=False)
+            self.assertFalse(mock_makessl.called)
+            self.assertEquals(vhs[0], self.vh_truth[1])
+
+    @mock.patch("certbot_apache.configurator.ApacheConfigurator._vhosts_for_wildcard")
+    @mock.patch("certbot_apache.configurator.ApacheConfigurator.make_vhost_ssl")
+    def test_choose_vhosts_wildcard_already_ssl(self, mock_makessl, mock_vh_for_w):
+        # pylint: disable=protected-access
+        # Already SSL vhost
+        mock_vh_for_w.return_value = [self.vh_truth[7]]
+        mock_path = "certbot_apache.display_ops.select_vhost_multiple"
+        with mock.patch(mock_path) as mock_select_vhs:
+            mock_select_vhs.return_value = [self.vh_truth[7]]
+            vhs = self.config._choose_vhosts_wildcard("whatever",
+                                                     create_ssl=True)
+            self.assertEquals(mock_select_vhs.call_args[0][0][0], self.vh_truth[7])
+            self.assertEquals(len(mock_select_vhs.call_args_list), 1)
+            # Ensure that make_vhost_ssl was not called, vhost.ssl == true
+            self.assertFalse(mock_makessl.called)
+
+            # And the actual returned values
+            self.assertEquals(len(vhs), 1)
+            self.assertTrue(vhs[0].ssl)
+            self.assertEquals(vhs[0], self.vh_truth[7])
+
+
+    def test_deploy_cert_wildcard(self):
+        # pylint: disable=protected-access
+        mock_choose_vhosts = mock.MagicMock()
+        mock_choose_vhosts.return_value = [self.vh_truth[7]]
+        self.config._choose_vhosts_wildcard = mock_choose_vhosts
+        mock_d = "certbot_apache.configurator.ApacheConfigurator._deploy_cert"
+        with mock.patch(mock_d) as mock_dep:
+            self.config.deploy_cert("*.wildcard.example.org", "/tmp/path",
+                                    "/tmp/path", "/tmp/path", "/tmp/path")
+            self.assertTrue(mock_dep.called)
+            self.assertEquals(len(mock_dep.call_args_list), 1)
+            self.assertEqual(self.vh_truth[7], mock_dep.call_args_list[0][0][0])
+
+    @mock.patch("certbot_apache.display_ops.select_vhost_multiple")
+    def test_deploy_cert_wildcard_no_vhosts(self, mock_dialog):
+        # pylint: disable=protected-access
+        mock_dialog.return_value = []
+        self.assertRaises(errors.PluginError,
+                          self.config.deploy_cert,
+                          "*.wild.cat", "/tmp/path", "/tmp/path",
+                           "/tmp/path", "/tmp/path")
+
+    @mock.patch("certbot_apache.configurator.ApacheConfigurator._choose_vhosts_wildcard")
+    def test_enhance_wildcard_after_install(self, mock_choose):
+        # pylint: disable=protected-access
+        self.config.parser.modules.add("mod_ssl.c")
+        self.config.parser.modules.add("headers_module")
+        self.vh_truth[3].ssl = True
+        self.config._wildcard_vhosts["*.certbot.demo"] = [self.vh_truth[3]]
+        self.config.enhance("*.certbot.demo", "ensure-http-header",
+                            "Upgrade-Insecure-Requests")
+        self.assertFalse(mock_choose.called)
+
+    @mock.patch("certbot_apache.configurator.ApacheConfigurator._choose_vhosts_wildcard")
+    def test_enhance_wildcard_no_install(self, mock_choose):
+        self.vh_truth[3].ssl = True
+        mock_choose.return_value = [self.vh_truth[3]]
+        self.config.parser.modules.add("mod_ssl.c")
+        self.config.parser.modules.add("headers_module")
+        self.config.enhance("*.certbot.demo", "ensure-http-header",
+                            "Upgrade-Insecure-Requests")
+        self.assertTrue(mock_choose.called)
+
+
 class AugeasVhostsTest(util.ApacheTest):
     """Test vhosts with illegal names dependent on augeas version."""
     # pylint: disable=protected-access

certbot-apache/certbot_apache/tests/debian_test.py

@@ -161,6 +161,8 @@ class MultipleVhostsTestDebian(util.ApacheTest):
         self.config.parser.modules.add("mod_ssl.c")
         self.config.get_version = mock.Mock(return_value=(2, 4, 7))
         mock_exe.return_value = True
+        # This will create an ssl vhost for certbot.demo
+        self.config.choose_vhost("certbot.demo")
         self.config.enhance("certbot.demo", "staple-ocsp")
         self.assertTrue("socache_shmcb_module" in self.config.parser.modules)
 
@@ -172,6 +174,7 @@ class MultipleVhostsTestDebian(util.ApacheTest):
         mock_exe.return_value = True
 
         # This will create an ssl vhost for certbot.demo
+        self.config.choose_vhost("certbot.demo")
         self.config.enhance("certbot.demo", "ensure-http-header",
                             "Strict-Transport-Security")
         self.assertTrue("headers_module" in self.config.parser.modules)
@@ -183,6 +186,7 @@ class MultipleVhostsTestDebian(util.ApacheTest):
         mock_exe.return_value = True
         self.config.get_version = mock.Mock(return_value=(2, 2))
         # This will create an ssl vhost for certbot.demo
+        self.config.choose_vhost("certbot.demo")
         self.config.enhance("certbot.demo", "redirect")
         self.assertTrue("rewrite_module" in self.config.parser.modules)
 

certbot-apache/certbot_apache/tests/display_ops_test.py

@@ -11,9 +11,39 @@ from certbot.tests import util as certbot_util
 
 from certbot_apache import obj
 
+from certbot_apache.display_ops import select_vhost_multiple
 from certbot_apache.tests import util
 
 
+class SelectVhostMultiTest(unittest.TestCase):
+    """Tests for certbot_apache.display_ops.select_vhost_multiple."""
+
+    def setUp(self):
+        self.base_dir = "/example_path"
+        self.vhosts = util.get_vh_truth(
+            self.base_dir, "debian_apache_2_4/multiple_vhosts")
+
+    def test_select_no_input(self):
+        self.assertFalse(select_vhost_multiple([]))
+
+    @certbot_util.patch_get_utility()
+    def test_select_correct(self, mock_util):
+        mock_util().checklist.return_value = (
+            display_util.OK, [self.vhosts[3].display_repr(),
+                              self.vhosts[2].display_repr()])
+        vhs = select_vhost_multiple([self.vhosts[3],
+                                     self.vhosts[2],
+                                     self.vhosts[1]])
+        self.assertTrue(self.vhosts[2] in vhs)
+        self.assertTrue(self.vhosts[3] in vhs)
+        self.assertFalse(self.vhosts[1] in vhs)
+
+    @certbot_util.patch_get_utility()
+    def test_select_cancel(self, mock_util):
+        mock_util().checklist.return_value = (display_util.CANCEL, "whatever")
+        vhs = select_vhost_multiple([self.vhosts[2], self.vhosts[3]])
+        self.assertFalse(vhs)
+
 class SelectVhostTest(unittest.TestCase):
     """Tests for certbot_apache.display_ops.select_vhost."""
 

certbot-apache/certbot_apache/tests/gentoo_test.py

@@ -4,6 +4,8 @@ import unittest
 
 import mock
 
+from certbot import errors
+
 from certbot_apache import override_gentoo
 from certbot_apache import obj
 from certbot_apache.tests import util
@@ -123,5 +125,11 @@ class MultipleVhostsTestGentoo(util.ApacheTest):
         self.assertEquals(len(self.config.parser.modules), 4)
         self.assertTrue("mod_another.c" in self.config.parser.modules)
 
+    @mock.patch("certbot_apache.configurator.util.run_script")
+    def test_alt_restart_works(self, mock_run_script):
+        mock_run_script.side_effect = [None, errors.SubprocessError, None]
+        self.config.restart()
+        self.assertEquals(mock_run_script.call_count, 3)
+
 if __name__ == "__main__":
     unittest.main()  # pragma: no cover

certbot-apache/certbot_apache/tests/http_01_test.py

@@ -16,30 +16,9 @@ from certbot_apache.tests import util
 NUM_ACHALLS = 3
 
 
-class ApacheHttp01TestMeta(type):
-    """Generates parmeterized tests for testing perform."""
-    def __new__(mcs, name, bases, class_dict):
-
-        def _gen_test(num_achalls, minor_version):
-            def _test(self):
-                achalls = self.achalls[:num_achalls]
-                vhosts = self.vhosts[:num_achalls]
-                self.config.version = (2, minor_version)
-                self.common_perform_test(achalls, vhosts)
-            return _test
-
-        for i in range(1, NUM_ACHALLS + 1):
-            for j in (2, 4):
-                test_name = "test_perform_{0}_{1}".format(i, j)
-                class_dict[test_name] = _gen_test(i, j)
-        return type.__new__(mcs, name, bases, class_dict)
-
-
 class ApacheHttp01Test(util.ApacheTest):
     """Test for certbot_apache.http_01.ApacheHttp01."""
 
-    __metaclass__ = ApacheHttp01TestMeta
-
     def setUp(self, *args, **kwargs):
         super(ApacheHttp01Test, self).setUp(*args, **kwargs)
 
@@ -71,7 +50,7 @@ class ApacheHttp01Test(util.ApacheTest):
         self.assertFalse(self.http.perform())
 
     @mock.patch("certbot_apache.configurator.ApacheConfigurator.enable_mod")
-    def test_enable_modules_22(self, mock_enmod):
+    def test_enable_modules_apache_2_2(self, mock_enmod):
         self.config.version = (2, 2)
         self.config.parser.modules.remove("authz_host_module")
         self.config.parser.modules.remove("mod_authz_host.c")
@@ -80,7 +59,7 @@ class ApacheHttp01Test(util.ApacheTest):
         self.assertEqual(enmod_calls[0][0][0], "authz_host")
 
     @mock.patch("certbot_apache.configurator.ApacheConfigurator.enable_mod")
-    def test_enable_modules_24(self, mock_enmod):
+    def test_enable_modules_apache_2_4(self, mock_enmod):
         self.config.parser.modules.remove("authz_core_module")
         self.config.parser.modules.remove("mod_authz_core.c")
 
@@ -137,6 +116,31 @@ class ApacheHttp01Test(util.ApacheTest):
         self.config.config.http01_port = 12345
         self.assertRaises(errors.PluginError, self.http.perform)
 
+    def test_perform_1_achall_apache_2_2(self):
+        self.combinations_perform_test(num_achalls=1, minor_version=2)
+
+    def test_perform_1_achall_apache_2_4(self):
+        self.combinations_perform_test(num_achalls=1, minor_version=4)
+
+    def test_perform_2_achall_apache_2_2(self):
+        self.combinations_perform_test(num_achalls=2, minor_version=2)
+
+    def test_perform_2_achall_apache_2_4(self):
+        self.combinations_perform_test(num_achalls=2, minor_version=4)
+
+    def test_perform_3_achall_apache_2_2(self):
+        self.combinations_perform_test(num_achalls=3, minor_version=2)
+
+    def test_perform_3_achall_apache_2_4(self):
+        self.combinations_perform_test(num_achalls=3, minor_version=4)
+
+    def combinations_perform_test(self, num_achalls, minor_version):
+        """Test perform with the given achall count and Apache version."""
+        achalls = self.achalls[:num_achalls]
+        vhosts = self.vhosts[:num_achalls]
+        self.config.version = (2, minor_version)
+        self.common_perform_test(achalls, vhosts)
+
     def common_perform_test(self, achalls, vhosts):
         """Tests perform with the given achalls."""
         challenge_dir = self.http.challenge_dir

certbot-apache/certbot_apache/tls_sni_01.py

@@ -123,7 +123,8 @@ class ApacheTlsSni01(common.TLSSNI01):
             self.configurator.config.tls_sni_01_port)))
 
         try:
-            vhost = self.configurator.choose_vhost(achall.domain, temp=True)
+            vhost = self.configurator.choose_vhost(achall.domain,
+                                                   create_if_no_ssl=False)
         except (PluginError, MissingCommandlineFlag):
             # We couldn't find the virtualhost for this domain, possibly
             # because it's a new vhost that's not configured yet

certbot-apache/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-apache/setup.py

@@ -4,17 +4,16 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
     'mock',
     'python-augeas',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.component',
     'zope.interface',
 ]
@@ -32,6 +31,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -40,10 +40,8 @@ setup(
         'Operating System :: POSIX :: Linux',
         'Programming Language :: Python',
         'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-auto

@@ -31,7 +31,7 @@ if [ -z "$VENV_PATH" ]; then
 fi
 VENV_BIN="$VENV_PATH/bin"
 BOOTSTRAP_VERSION_PATH="$VENV_PATH/certbot-auto-bootstrap-version.txt"
-LE_AUTO_VERSION="0.21.1"
+LE_AUTO_VERSION="0.23.0"
 BASENAME=$(basename $0)
 USAGE="Usage: $BASENAME [OPTIONS]
 A self-updating wrapper script for the Certbot ACME client. When run, updates
@@ -47,6 +47,7 @@ Help for certbot itself cannot be provided until it is installed.
   --no-bootstrap                            do not install OS dependencies
   --no-self-upgrade                         do not download updates
   --os-packages-only                        install OS dependencies and exit
+  --install-only                            install certbot, upgrade if needed, and exit
   -v, --verbose                             provide more output
   -q, --quiet                               provide only update/error output;
                                             implies --non-interactive
@@ -60,6 +61,8 @@ for arg in "$@" ; do
       DEBUG=1;;
     --os-packages-only)
       OS_PACKAGES_ONLY=1;;
+    --install-only)
+      INSTALL_ONLY=1;;
     --no-self-upgrade)
       # Do not upgrade this script (also prevents client upgrades, because each
       # copy of the script pins a hash of the python client)
@@ -246,7 +249,7 @@ DeprecationBootstrap() {
   fi
 }
 
-MIN_PYTHON_VERSION="2.6"
+MIN_PYTHON_VERSION="2.7"
 MIN_PYVER=$(echo "$MIN_PYTHON_VERSION" | sed 's/\.//')
 # Sets LE_PYTHON to Python version string and PYVER to the first two
 # digits of the python version
@@ -1196,24 +1199,24 @@ letsencrypt==0.7.0 \
     --hash=sha256:105a5fb107e45bcd0722eb89696986dcf5f08a86a321d6aef25a0c7c63375ade \
     --hash=sha256:c36e532c486a7e92155ee09da54b436a3c420813ec1c590b98f635d924720de9
 
-certbot==0.21.1 \
-    --hash=sha256:08f026078807fbcfd7bfab44c4d827ee287738fefcc86fbe1493ce752d2fdccb \
-    --hash=sha256:e6c8e9b0b5e38834330831d5a91e1c08accdb9b4923855d14d524e7327e6c4ea
-acme==0.21.1 \
-    --hash=sha256:4b2b5ef80c755dfa30eb5c67ab4b4e66e7f205ad922b43170502c5f8d8ef1242 \
-    --hash=sha256:296e8abf4f5a69af1a892416faceea90e15f39e2920bf87beeaad1d6ce70a60b
-certbot-apache==0.21.1 \
-    --hash=sha256:faa4af1033564a0e676d16940775593fb849527b494a15f6a816ad0ed4fa273c \
-    --hash=sha256:0bce4419d4fdabbdda2223cff8db6794c5717632fb9511b00498ec00982a3fa5
-certbot-nginx==0.21.1 \
-    --hash=sha256:3fad3b4722544558ce03132f853e18da5e516013086aaa40f1036aa6667c70a9 \
-    --hash=sha256:55a32afe0950ff49d3118f93035463a46c85c2f399d261123f5fe973afdd4f64
+certbot==0.23.0 \
+    --hash=sha256:66c42cf780ddbf582ecc52aa6a61242450a2650227b436ad0d260685c4ef8a49 \
+    --hash=sha256:6cff4c5da1228661ccaf95195064cb29e6cdf80913193bdb2eb20e164c76053e
+acme==0.23.0 \
+    --hash=sha256:02e9b596bd3bf8f0733d6d43ec2464ac8185a000acb58d2b4fd9e19223bbbf0b \
+    --hash=sha256:08c16635578507f526c338b3418c1147a9f015bf2d366abd51f38918703b4550
+certbot-apache==0.23.0 \
+    --hash=sha256:50077742d2763b7600dfda618eb89c870aeea5e6a4c00f60157877f7a7d81f7c \
+    --hash=sha256:6b7acec243e224de5268d46c2597277586dffa55e838c252b6931c30d549028e
+certbot-nginx==0.23.0 \
+    --hash=sha256:f12c21bbe3eb955ca533f1da96d28c6310378b138e844d83253562e18b6cbb32 \
+    --hash=sha256:cadf14e4bd504d9ce5987a5ec6dbd8e136638e55303ad5dc81dcb723ddd64324
 
 UNLIKELY_EOF
     # -------------------------------------------------------------------------
     cat << "UNLIKELY_EOF" > "$TEMP_DIR/pipstrap.py"
 #!/usr/bin/env python
-"""A small script that can act as a trust root for installing pip 8
+"""A small script that can act as a trust root for installing pip >=8
 
 Embed this in your project, and your VCS checkout is all you have to trust. In
 a post-peep era, this lets you claw your way to a hash-checking version of pip,
@@ -1237,6 +1240,7 @@ anything goes wrong, it will exit with a non-zero status code.
 from __future__ import print_function
 from distutils.version import StrictVersion
 from hashlib import sha256
+from os import environ
 from os.path import join
 from pipes import quote
 from shutil import rmtree
@@ -1270,14 +1274,14 @@ except ImportError:
     from urllib.parse import urlparse  # 3.4
 
 
-__version__ = 1, 3, 0
+__version__ = 1, 5, 1
 PIP_VERSION = '9.0.1'
+DEFAULT_INDEX_BASE = 'https://pypi.python.org'
 
 
 # wheel has a conditional dependency on argparse:
 maybe_argparse = (
-    [('https://pypi.python.org/packages/18/dd/'
-      'e617cfc3f6210ae183374cd9f6a26b20514bbb5a792af97949c5aacddf0f/'
+    [('18/dd/e617cfc3f6210ae183374cd9f6a26b20514bbb5a792af97949c5aacddf0f/'
       'argparse-1.4.0.tar.gz',
       '62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4')]
     if version_info < (2, 7, 0) else [])
@@ -1285,18 +1289,14 @@ maybe_argparse = (
 
 PACKAGES = maybe_argparse + [
     # Pip has no dependencies, as it vendors everything:
-    ('https://pypi.python.org/packages/11/b6/'
-     'abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/'
-     'pip-{0}.tar.gz'
-     .format(PIP_VERSION),
+    ('11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/'
+     'pip-{0}.tar.gz'.format(PIP_VERSION),
      '09f243e1a7b461f654c26a725fa373211bb7ff17a9300058b205c61658ca940d'),
     # This version of setuptools has only optional dependencies:
-    ('https://pypi.python.org/packages/69/65/'
-     '4c544cde88d4d876cdf5cbc5f3f15d02646477756d89547e9a7ecd6afa76/'
-     'setuptools-20.2.2.tar.gz',
-     '24fcfc15364a9fe09a220f37d2dcedc849795e3de3e4b393ee988e66a9cbd85a'),
-    ('https://pypi.python.org/packages/c9/1d/'
-     'bd19e691fd4cfe908c76c429fe6e4436c9e83583c4414b54f6c85471954a/'
+    ('59/88/2f3990916931a5de6fa9706d6d75eb32ee8b78627bb2abaab7ed9e6d0622/'
+     'setuptools-29.0.1.tar.gz',
+     'b539118819a4857378398891fa5366e090690e46b3e41421a1e07d6e9fd8feb0'),
+    ('c9/1d/bd19e691fd4cfe908c76c429fe6e4436c9e83583c4414b54f6c85471954a/'
      'wheel-0.29.0.tar.gz',
      '1ebb8ad7e26b448e9caa4773d2357849bf80ff9e313964bcaf79cbf0201a1648')
 ]
@@ -1317,12 +1317,13 @@ def hashed_download(url, temp, digest):
     # >=2.7.9 verifies HTTPS certs itself, and, in any case, the cert
     # authenticity has only privacy (not arbitrary code execution)
     # implications, since we're checking hashes.
-    def opener():
+    def opener(using_https=True):
         opener = build_opener(HTTPSHandler())
-        # Strip out HTTPHandler to prevent MITM spoof:
-        for handler in opener.handlers:
-            if isinstance(handler, HTTPHandler):
-                opener.handlers.remove(handler)
+        if using_https:
+            # Strip out HTTPHandler to prevent MITM spoof:
+            for handler in opener.handlers:
+                if isinstance(handler, HTTPHandler):
+                    opener.handlers.remove(handler)
         return opener
 
     def read_chunks(response, chunk_size):
@@ -1332,8 +1333,9 @@ def hashed_download(url, temp, digest):
                 break
             yield chunk
 
-    response = opener().open(url)
-    path = join(temp, urlparse(url).path.split('/')[-1])
+    parsed_url = urlparse(url)
+    response = opener(using_https=parsed_url.scheme == 'https').open(url)
+    path = join(temp, parsed_url.path.split('/')[-1])
     actual_hash = sha256()
     with open(path, 'wb') as file:
         for chunk in read_chunks(response, 4096):
@@ -1346,6 +1348,24 @@ def hashed_download(url, temp, digest):
     return path
 
 
+def get_index_base():
+    """Return the URL to the dir containing the "packages" folder.
+
+    Try to wring something out of PIP_INDEX_URL, if set. Hack "/simple" off the
+    end if it's there; that is likely to give us the right dir.
+
+    """
+    env_var = environ.get('PIP_INDEX_URL', '').rstrip('/')
+    if env_var:
+        SIMPLE = '/simple'
+        if env_var.endswith(SIMPLE):
+            return env_var[:-len(SIMPLE)]
+        else:
+            return env_var
+    else:
+        return DEFAULT_INDEX_BASE
+
+
 def main():
     pip_version = StrictVersion(check_output(['pip', '--version'])
                                 .decode('utf-8').split()[1])
@@ -1353,11 +1373,13 @@ def main():
     if pip_version >= min_pip_version:
         return 0
     has_pip_cache = pip_version >= StrictVersion('6.0')
-
+    index_base = get_index_base()
     temp = mkdtemp(prefix='pipstrap-')
     try:
-        downloads = [hashed_download(url, temp, digest)
-                     for url, digest in PACKAGES]
+        downloads = [hashed_download(index_base + '/packages/' + path,
+                                     temp,
+                                     digest)
+                     for path, digest in PACKAGES]
         check_output('pip install --no-index --no-deps -U ' +
                      # Disable cache since we're not using it and it otherwise
                      # sometimes throws permission warnings:
@@ -1428,6 +1450,12 @@ UNLIKELY_EOF
 
     say "Installation succeeded."
   fi
+
+  if [ "$INSTALL_ONLY" = 1 ]; then
+    say "Certbot is installed."
+    exit 0
+  fi
+
   "$VENV_BIN/letsencrypt" "$@"
 
 else

certbot-compatibility-test/certbot_compatibility_test/test_driver.py

@@ -10,6 +10,8 @@ import sys
 
 import OpenSSL
 
+from six.moves import xrange  # pylint: disable=import-error,redefined-builtin
+
 from acme import challenges
 from acme import crypto_util
 from acme import messages

certbot-compatibility-test/certbot_compatibility_test/validator.py

@@ -5,6 +5,7 @@ import requests
 import zope.interface
 
 import six
+from six.moves import xrange  # pylint: disable=import-error,redefined-builtin
 
 from acme import crypto_util
 from acme import errors as acme_errors

certbot-compatibility-test/nginx/roundtrip.py

@@ -8,7 +8,7 @@ from certbot_nginx import nginxparser
 def roundtrip(stuff):
     success = True
     for t in stuff:
-        print t
+        print(t)
         if not os.path.isfile(t):
             continue
         with open(t, "r") as f:

certbot-compatibility-test/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
 install_requires = [
     'certbot',
@@ -34,16 +34,15 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Intended Audience :: Developers',
         'License :: OSI Approved :: Apache Software License',
         'Programming Language :: Python',
         'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-cloudflare/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-cloudflare
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-cloudflare

certbot-dns-cloudflare/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-cloudflare's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_cloudflare
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_cloudflare
-   :members:
-
 
 Indices and tables
 ==================

certbot-dns-cloudflare/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-cloudflare/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-cloudflare[docs]

certbot-dns-cloudflare/setup.py

@@ -4,17 +4,16 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
     'cloudflare>=1.5.1',
     'mock',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.interface',
 ]
 
@@ -31,6 +30,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -39,10 +39,8 @@ setup(
         'Operating System :: POSIX :: Linux',
         'Programming Language :: Python',
         'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-cloudxns/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-cloudxns
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-cloudxns

certbot-dns-cloudxns/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-cloudxns's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_cloudxns
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_cloudxns
-   :members:
-
 
 
 Indices and tables

certbot-dns-cloudxns/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-cloudxns/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-cloudxns[docs]

certbot-dns-cloudxns/setup.py

@@ -4,17 +4,16 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
-    'dns-lexicon',
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
+    'dns-lexicon>=2.2.1', # Support for >1 TXT record per name
     'mock',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.interface',
 ]
 
@@ -31,6 +30,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -41,7 +41,6 @@ setup(
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-digitalocean/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-digitalocean
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-digitalocean

certbot-dns-digitalocean/certbot_dns_digitalocean/dns_digitalocean_test.py

@@ -50,7 +50,8 @@ class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthentic
 
 
 class DigitalOceanClientTest(unittest.TestCase):
-    id = 1
+
+    id_num = 1
     record_prefix = "_acme-challenge"
     record_name = record_prefix + "." + DOMAIN
     record_content = "bar"
@@ -70,7 +71,7 @@ class DigitalOceanClientTest(unittest.TestCase):
 
         domain_mock = mock.MagicMock()
         domain_mock.name = DOMAIN
-        domain_mock.create_new_domain_record.return_value = {'domain_record': {'id': self.id}}
+        domain_mock.create_new_domain_record.return_value = {'domain_record': {'id': self.id_num}}
 
         self.manager.get_all_domains.return_value = [wrong_domain_mock, domain_mock]
 

certbot-dns-digitalocean/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-digitalocean's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_digitalocean
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_digitalocean
-   :members:
-
 
 
 Indices and tables

certbot-dns-digitalocean/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-digitalocean/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-digitalocean[docs]

certbot-dns-digitalocean/setup.py

@@ -4,17 +4,16 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
     'mock',
     'python-digitalocean>=1.11',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'six',
     'zope.interface',
 ]
@@ -32,6 +31,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -40,10 +40,8 @@ setup(
         'Operating System :: POSIX :: Linux',
         'Programming Language :: Python',
         'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-dnsimple/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-dnsimple
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-dnsimple

certbot-dns-dnsimple/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-dnsimple's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_dnsimple
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_dnsimple
-   :members:
-
 
 
 Indices and tables

certbot-dns-dnsimple/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-dnsimple/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-dnsimple[docs]

certbot-dns-dnsimple/setup.py

@@ -4,17 +4,16 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
-    'dns-lexicon',
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
+    'dns-lexicon>=2.2.1', # Support for >1 TXT record per name
     'mock',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.interface',
 ]
 
@@ -31,6 +30,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -41,7 +41,6 @@ setup(
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-dnsmadeeasy/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-dnsmadeeasy
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-dnsmadeeasy

certbot-dns-dnsmadeeasy/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-dnsmadeeasy's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_dnsmadeeasy
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_dnsmadeeasy
-   :members:
-
 
 
 Indices and tables

certbot-dns-dnsmadeeasy/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-dnsmadeeasy/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-dnsmadeeasy[docs]

certbot-dns-dnsmadeeasy/setup.py

@@ -4,17 +4,16 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
-    'dns-lexicon',
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
+    'dns-lexicon>=2.2.1', # Support for >1 TXT record per name
     'mock',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.interface',
 ]
 
@@ -31,6 +30,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -41,7 +41,6 @@ setup(
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-google/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-google
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-google

certbot-dns-google/MANIFEST.in

@@ -1,3 +1,4 @@
 include LICENSE.txt
 include README.rst
 recursive-include docs *
+recursive-include certbot_dns_google/testdata *

certbot-dns-google/certbot_dns_google/__init__.py

@@ -29,6 +29,8 @@ for an account with the following permissions:
 * ``dns.managedZones.list``
 * ``dns.resourceRecordSets.create``
 * ``dns.resourceRecordSets.delete``
+* ``dns.resourceRecordSets.list``
+* ``dns.resourceRecordSets.update``
 
 Google provides instructions for `creating a service account <https://developers
 .google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount>`_ and

certbot-dns-google/certbot_dns_google/dns_google.py

@@ -81,7 +81,7 @@ class _GoogleClient(object):
     Encapsulates all communication with the Google Cloud DNS API.
     """
 
-    def __init__(self, account_json=None):
+    def __init__(self, account_json=None, dns_api=None):
 
         scopes = ['https://www.googleapis.com/auth/ndev.clouddns.readwrite']
         if account_json is not None:
@@ -92,7 +92,12 @@ class _GoogleClient(object):
             credentials = None
             self.project_id = self.get_project_id()
 
-        self.dns = discovery.build('dns', 'v1', credentials=credentials, cache_discovery=False)
+        if not dns_api:
+            self.dns = discovery.build('dns', 'v1',
+                                       credentials=credentials,
+                                       cache_discovery=False)
+        else:
+            self.dns = dns_api
 
     def add_txt_record(self, domain, record_name, record_content, record_ttl):
         """
@@ -107,6 +112,17 @@ class _GoogleClient(object):
 
         zone_id = self._find_managed_zone_id(domain)
 
+        record_contents = self.get_existing_txt_rrset(zone_id, record_name)
+        if record_contents is None:
+            record_contents = []
+        add_records = record_contents[:]
+
+        if "\""+record_content+"\"" in record_contents:
+            # The process was interrupted previously and validation token exists
+            return
+
+        add_records.append(record_content)
+
         data = {
             "kind": "dns#change",
             "additions": [
@@ -114,12 +130,24 @@ class _GoogleClient(object):
                     "kind": "dns#resourceRecordSet",
                     "type": "TXT",
                     "name": record_name + ".",
-                    "rrdatas": [record_content, ],
+                    "rrdatas": add_records,
                     "ttl": record_ttl,
                 },
             ],
         }
 
+        if record_contents:
+            # We need to remove old records in the same request
+            data["deletions"] = [
+                {
+                    "kind": "dns#resourceRecordSet",
+                    "type": "TXT",
+                    "name": record_name + ".",
+                    "rrdatas": record_contents,
+                    "ttl": record_ttl,
+                },
+            ]
+
         changes = self.dns.changes()  # changes | pylint: disable=no-member
 
         try:
@@ -154,6 +182,10 @@ class _GoogleClient(object):
             logger.warn('Error finding zone. Skipping cleanup.')
             return
 
+        record_contents = self.get_existing_txt_rrset(zone_id, record_name)
+        if record_contents is None:
+            record_contents = ["\"" + record_content + "\""]
+
         data = {
             "kind": "dns#change",
             "deletions": [
@@ -161,12 +193,26 @@ class _GoogleClient(object):
                     "kind": "dns#resourceRecordSet",
                     "type": "TXT",
                     "name": record_name + ".",
-                    "rrdatas": [record_content, ],
+                    "rrdatas": record_contents,
                     "ttl": record_ttl,
                 },
             ],
         }
 
+        # Remove the record being deleted from the list
+        readd_contents = [r for r in record_contents if r != "\"" + record_content + "\""]
+        if readd_contents:
+            # We need to remove old records in the same request
+            data["additions"] = [
+                {
+                    "kind": "dns#resourceRecordSet",
+                    "type": "TXT",
+                    "name": record_name + ".",
+                    "rrdatas": readd_contents,
+                    "ttl": record_ttl,
+                },
+            ]
+
         changes = self.dns.changes()  # changes | pylint: disable=no-member
 
         try:
@@ -175,6 +221,37 @@ class _GoogleClient(object):
         except googleapiclient_errors.Error as e:
             logger.warn('Encountered error deleting TXT record: %s', e)
 
+    def get_existing_txt_rrset(self, zone_id, record_name):
+        """
+        Get existing TXT records from the RRset for the record name.
+
+        If an error occurs while requesting the record set, it is suppressed
+        and None is returned.
+
+        :param str zone_id: The ID of the managed zone.
+        :param str record_name: The record name (typically beginning with '_acme-challenge.').
+
+        :returns: List of TXT record values or None
+        :rtype: `list` of `string` or `None`
+
+        """
+        rrs_request = self.dns.resourceRecordSets()  # pylint: disable=no-member
+        request = rrs_request.list(managedZone=zone_id, project=self.project_id)
+        # Add dot as the API returns absolute domains
+        record_name += "."
+        try:
+            response = request.execute()
+        except googleapiclient_errors.Error:
+            logger.info("Unable to list existing records. If you're "
+                        "requesting a wildcard certificate, this might not work.")
+            logger.debug("Error was:", exc_info=True)
+        else:
+            if response:
+                for rr in response["rrsets"]:
+                    if rr["name"] == record_name and rr["type"] == "TXT":
+                        return rr["rrdatas"]
+        return None
+
     def _find_managed_zone_id(self, domain):
         """
         Find the managed zone for a given domain.

certbot-dns-google/certbot_dns_google/dns_google_test.py

@@ -4,7 +4,9 @@ import os
 import unittest
 
 import mock
+from googleapiclient import discovery
 from googleapiclient.errors import Error
+from googleapiclient.http import HttpMock
 from httplib2 import ServerNotFoundError
 
 from certbot import errors
@@ -68,16 +70,27 @@ class GoogleClientTest(unittest.TestCase):
     def _setUp_client_with_mock(self, zone_request_side_effect):
         from certbot_dns_google.dns_google import _GoogleClient
 
-        client = _GoogleClient(ACCOUNT_JSON_PATH)
+        pwd = os.path.dirname(__file__)
+        rel_path = 'testdata/discovery.json'
+        discovery_file = os.path.join(pwd, rel_path)
+        http_mock = HttpMock(discovery_file, {'status': '200'})
+        dns_api = discovery.build('dns', 'v1', http=http_mock)
+
+        client = _GoogleClient(ACCOUNT_JSON_PATH, dns_api)
 
         # Setup
         mock_mz = mock.MagicMock()
         mock_mz.list.return_value.execute.side_effect = zone_request_side_effect
 
+        mock_rrs = mock.MagicMock()
+        rrsets = {"rrsets": [{"name": "_acme-challenge.example.org.", "type": "TXT",
+                              "rrdatas": ["\"example-txt-contents\""]}]}
+        mock_rrs.list.return_value.execute.return_value = rrsets
         mock_changes = mock.MagicMock()
 
         client.dns.managedZones = mock.MagicMock(return_value=mock_mz)
         client.dns.changes = mock.MagicMock(return_value=mock_changes)
+        client.dns.resourceRecordSets = mock.MagicMock(return_value=mock_rrs)
 
         return client, mock_changes
 
@@ -137,6 +150,30 @@ class GoogleClientTest(unittest.TestCase):
                                             managedZone=self.zone,
                                             project=PROJECT_ID)
 
+    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('certbot_dns_google.dns_google.open',
+                mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
+    def test_add_txt_record_delete_old(self, unused_credential_mock):
+        client, changes = self._setUp_client_with_mock(
+            [{'managedZones': [{'id': self.zone}]}])
+        mock_get_rrs = "certbot_dns_google.dns_google._GoogleClient.get_existing_txt_rrset"
+        with mock.patch(mock_get_rrs) as mock_rrs:
+            mock_rrs.return_value = ["sample-txt-contents"]
+            client.add_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
+            self.assertTrue(changes.create.called)
+            self.assertTrue("sample-txt-contents" in
+                changes.create.call_args_list[0][1]["body"]["deletions"][0]["rrdatas"])
+
+    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('certbot_dns_google.dns_google.open',
+                mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
+    def test_add_txt_record_noop(self, unused_credential_mock):
+        client, changes = self._setUp_client_with_mock(
+            [{'managedZones': [{'id': self.zone}]}])
+        client.add_txt_record(DOMAIN, "_acme-challenge.example.org",
+                              "example-txt-contents", self.record_ttl)
+        self.assertFalse(changes.create.called)
+
     @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
     @mock.patch('certbot_dns_google.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
@@ -172,7 +209,12 @@ class GoogleClientTest(unittest.TestCase):
     def test_del_txt_record(self, unused_credential_mock):
         client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}])
 
-        client.del_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
+        mock_get_rrs = "certbot_dns_google.dns_google._GoogleClient.get_existing_txt_rrset"
+        with mock.patch(mock_get_rrs) as mock_rrs:
+            mock_rrs.return_value = ["\"sample-txt-contents\"",
+                                     "\"example-txt-contents\""]
+            client.del_txt_record(DOMAIN, "_acme-challenge.example.org",
+                                "example-txt-contents", self.record_ttl)
 
         expected_body = {
             "kind": "dns#change",
@@ -180,8 +222,17 @@ class GoogleClientTest(unittest.TestCase):
                 {
                     "kind": "dns#resourceRecordSet",
                     "type": "TXT",
-                    "name": self.record_name + ".",
-                    "rrdatas": [self.record_content, ],
+                    "name": "_acme-challenge.example.org.",
+                    "rrdatas": ["\"sample-txt-contents\"", "\"example-txt-contents\""],
+                    "ttl": self.record_ttl,
+                },
+            ],
+            "additions": [
+                {
+                    "kind": "dns#resourceRecordSet",
+                    "type": "TXT",
+                    "name": "_acme-challenge.example.org.",
+                    "rrdatas": ["\"sample-txt-contents\"", ],
                     "ttl": self.record_ttl,
                 },
             ],
@@ -217,6 +268,31 @@ class GoogleClientTest(unittest.TestCase):
 
         client.del_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
+    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('certbot_dns_google.dns_google.open',
+                mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
+    def test_get_existing(self, unused_credential_mock):
+        client, unused_changes = self._setUp_client_with_mock(
+            [{'managedZones': [{'id': self.zone}]}])
+        # Record name mocked in setUp
+        found = client.get_existing_txt_rrset(self.zone, "_acme-challenge.example.org")
+        self.assertEquals(found, ["\"example-txt-contents\""])
+        not_found = client.get_existing_txt_rrset(self.zone, "nonexistent.tld")
+        self.assertEquals(not_found, None)
+
+    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('certbot_dns_google.dns_google.open',
+                mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
+    def test_get_existing_fallback(self, unused_credential_mock):
+        client, unused_changes = self._setUp_client_with_mock(
+            [{'managedZones': [{'id': self.zone}]}])
+        # pylint: disable=no-member
+        mock_execute = client.dns.resourceRecordSets.return_value.list.return_value.execute
+        mock_execute.side_effect = API_ERROR
+
+        rrset = client.get_existing_txt_rrset(self.zone, "_acme-challenge.example.org")
+        self.assertFalse(rrset)
+
     def test_get_project_id(self):
         from certbot_dns_google.dns_google import _GoogleClient
 

certbot-dns-google/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-google's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_google
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_google
-   :members:
-
 
 
 Indices and tables

certbot-dns-google/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-google/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-google[docs]

certbot-dns-google/setup.py

@@ -4,20 +4,19 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
     # 1.5 is the first version that supports oauth2client>=2.0
     'google-api-python-client>=1.5',
     'mock',
     # for oauth2client.service_account.ServiceAccountCredentials
     'oauth2client>=2.0',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.interface',
     # already a dependency of google-api-python-client, but added for consistency
     'httplib2'
@@ -36,6 +35,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -44,10 +44,8 @@ setup(
         'Operating System :: POSIX :: Linux',
         'Programming Language :: Python',
         'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-luadns/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-luadns
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-luadns

certbot-dns-luadns/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-luadns's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_luadns
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_luadns
-   :members:
-
 
 
 Indices and tables

certbot-dns-luadns/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-luadns/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-luadns[docs]

certbot-dns-luadns/setup.py

@@ -4,17 +4,16 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
-    'dns-lexicon',
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
+    'dns-lexicon>=2.2.1', # Support for >1 TXT record per name
     'mock',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.interface',
 ]
 
@@ -31,6 +30,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -41,7 +41,6 @@ setup(
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-nsone/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-nsone
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-nsone

certbot-dns-nsone/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-nsone's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_nsone
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_nsone
-   :members:
-
 
 
 Indices and tables

certbot-dns-nsone/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-nsone/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-nsone[docs]

certbot-dns-nsone/setup.py

@@ -4,17 +4,16 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
-    'dns-lexicon',
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
+    'dns-lexicon>=2.2.1', # Support for >1 TXT record per name
     'mock',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.interface',
 ]
 
@@ -31,6 +30,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -41,7 +41,6 @@ setup(
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-rfc2136/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-rfc2136
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-rfc2136

certbot-dns-rfc2136/certbot_dns_rfc2136/__init__.py

@@ -21,8 +21,9 @@ Credentials
 -----------
 
 Use of this plugin requires a configuration file containing the target DNS
-server that supports RFC 2136 Dynamic Updates, the name of the TSIG key, the
-TSIG key secret itself and the algorithm used if it's different to HMAC-MD5.
+server and optional port that supports RFC 2136 Dynamic Updates, the name
+of the TSIG key, the TSIG key secret itself and the algorithm used if it's
+different to HMAC-MD5.
 
 .. code-block:: ini
    :name: credentials.ini
@@ -30,6 +31,8 @@ TSIG key secret itself and the algorithm used if it's different to HMAC-MD5.
 
    # Target DNS server
    dns_rfc2136_server = 192.0.2.1
+   # Target DNS port
+   dns_rfc2136_port = 53
    # TSIG key name
    dns_rfc2136_name = keyname.
    # TSIG key secret

certbot-dns-rfc2136/certbot_dns_rfc2136/dns_rfc2136.py

@@ -36,6 +36,8 @@ class Authenticator(dns_common.DNSAuthenticator):
       'HMAC-SHA512': dns.tsig.HMAC_SHA512
     }
 
+    PORT = 53
+
     description = 'Obtain certificates using a DNS TXT record (if you are using BIND for DNS).'
     ttl = 120
 
@@ -70,14 +72,15 @@ class Authenticator(dns_common.DNSAuthenticator):
             self._validate_algorithm
         )
 
-    def _perform(self, domain, validation_name, validation):
-        self._get_rfc2136_client().add_txt_record(domain, validation_name, validation, self.ttl)
+    def _perform(self, _domain, validation_name, validation):
+        self._get_rfc2136_client().add_txt_record(validation_name, validation, self.ttl)
 
-    def _cleanup(self, domain, validation_name, validation):
-        self._get_rfc2136_client().del_txt_record(domain, validation_name, validation)
+    def _cleanup(self, _domain, validation_name, validation):
+        self._get_rfc2136_client().del_txt_record(validation_name, validation)
 
     def _get_rfc2136_client(self):
         return _RFC2136Client(self.credentials.conf('server'),
+                              int(self.credentials.conf('port') or self.PORT),
                               self.credentials.conf('name'),
                               self.credentials.conf('secret'),
                               self.ALGORITHMS.get(self.credentials.conf('algorithm'),
@@ -88,25 +91,25 @@ class _RFC2136Client(object):
     """
     Encapsulates all communication with the target DNS server.
     """
-    def __init__(self, server, key_name, key_secret, key_algorithm):
+    def __init__(self, server, port, key_name, key_secret, key_algorithm):
         self.server = server
+        self.port = port
         self.keyring = dns.tsigkeyring.from_text({
             key_name: key_secret
         })
         self.algorithm = key_algorithm
 
-    def add_txt_record(self, domain_name, record_name, record_content, record_ttl):
+    def add_txt_record(self, record_name, record_content, record_ttl):
         """
         Add a TXT record using the supplied information.
 
-        :param str domain: The domain to use to find the closest SOA.
         :param str record_name: The record name (typically beginning with '_acme-challenge.').
         :param str record_content: The record content (typically the challenge validation).
         :param int record_ttl: The record TTL (number of seconds that the record may be cached).
         :raises certbot.errors.PluginError: if an error occurs communicating with the DNS server
         """
 
-        domain = self._find_domain(domain_name)
+        domain = self._find_domain(record_name)
 
         n = dns.name.from_text(record_name)
         o = dns.name.from_text(domain)
@@ -119,7 +122,7 @@ class _RFC2136Client(object):
         update.add(rel, record_ttl, dns.rdatatype.TXT, record_content)
 
         try:
-            response = dns.query.tcp(update, self.server)
+            response = dns.query.tcp(update, self.server, port=self.port)
         except Exception as e:
             raise errors.PluginError('Encountered error adding TXT record: {0}'
                                      .format(e))
@@ -131,18 +134,17 @@ class _RFC2136Client(object):
             raise errors.PluginError('Received response from server: {0}'
                                      .format(dns.rcode.to_text(rcode)))
 
-    def del_txt_record(self, domain_name, record_name, record_content):
+    def del_txt_record(self, record_name, record_content):
         """
         Delete a TXT record using the supplied information.
 
-        :param str domain: The domain to use to find the closest SOA.
         :param str record_name: The record name (typically beginning with '_acme-challenge.').
         :param str record_content: The record content (typically the challenge validation).
         :param int record_ttl: The record TTL (number of seconds that the record may be cached).
         :raises certbot.errors.PluginError: if an error occurs communicating with the DNS server
         """
 
-        domain = self._find_domain(domain_name)
+        domain = self._find_domain(record_name)
 
         n = dns.name.from_text(record_name)
         o = dns.name.from_text(domain)
@@ -155,7 +157,7 @@ class _RFC2136Client(object):
         update.delete(rel, dns.rdatatype.TXT, record_content)
 
         try:
-            response = dns.query.tcp(update, self.server)
+            response = dns.query.tcp(update, self.server, port=self.port)
         except Exception as e:
             raise errors.PluginError('Encountered error deleting TXT record: {0}'
                                      .format(e))
@@ -167,17 +169,17 @@ class _RFC2136Client(object):
             raise errors.PluginError('Received response from server: {0}'
                                      .format(dns.rcode.to_text(rcode)))
 
-    def _find_domain(self, domain_name):
+    def _find_domain(self, record_name):
         """
         Find the closest domain with an SOA record for a given domain name.
 
-        :param str domain_name: The domain name for which to find the closest SOA record.
+        :param str record_name: The record name for which to find the closest SOA record.
         :returns: The domain, if found.
         :rtype: str
         :raises certbot.errors.PluginError: if no SOA record can be found.
         """
 
-        domain_name_guesses = dns_common.base_domain_name_guesses(domain_name)
+        domain_name_guesses = dns_common.base_domain_name_guesses(record_name)
 
         # Loop through until we find an authoritative SOA record
         for guess in domain_name_guesses:
@@ -185,7 +187,7 @@ class _RFC2136Client(object):
                 return guess
 
         raise errors.PluginError('Unable to determine base domain for {0} using names: {1}.'
-                                 .format(domain_name, domain_name_guesses))
+                                 .format(record_name, domain_name_guesses))
 
     def _query_soa(self, domain_name):
         """
@@ -204,7 +206,7 @@ class _RFC2136Client(object):
         request.flags ^= dns.flags.RD
 
         try:
-            response = dns.query.udp(request, self.server)
+            response = dns.query.udp(request, self.server, port=self.port)
             rcode = response.rcode()
 
             # Authoritative Answer bit should be set

certbot-dns-rfc2136/certbot_dns_rfc2136/dns_rfc2136_test.py

@@ -14,6 +14,7 @@ from certbot.plugins.dns_test_common import DOMAIN
 from certbot.tests import util as test_util
 
 SERVER = '192.0.2.1'
+PORT = 53
 NAME = 'a-tsig-key.'
 SECRET = 'SSB3b25kZXIgd2hvIHdpbGwgYm90aGVyIHRvIGRlY29kZSB0aGlzIHRleHQK'
 VALID_CONFIG = {"rfc2136_server": SERVER, "rfc2136_name": NAME, "rfc2136_secret": SECRET}
@@ -41,7 +42,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthentic
     def test_perform(self):
         self.auth.perform([self.achall])
 
-        expected = [mock.call.add_txt_record(DOMAIN, '_acme-challenge.'+DOMAIN, mock.ANY, mock.ANY)]
+        expected = [mock.call.add_txt_record('_acme-challenge.'+DOMAIN, mock.ANY, mock.ANY)]
         self.assertEqual(expected, self.mock_client.mock_calls)
 
     def test_cleanup(self):
@@ -49,7 +50,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthentic
         self.auth._attempt_cleanup = True
         self.auth.cleanup([self.achall])
 
-        expected = [mock.call.del_txt_record(DOMAIN, '_acme-challenge.'+DOMAIN, mock.ANY)]
+        expected = [mock.call.del_txt_record('_acme-challenge.'+DOMAIN, mock.ANY)]
         self.assertEqual(expected, self.mock_client.mock_calls)
 
     def test_invalid_algorithm_raises(self):
@@ -74,7 +75,7 @@ class RFC2136ClientTest(unittest.TestCase):
     def setUp(self):
         from certbot_dns_rfc2136.dns_rfc2136 import _RFC2136Client
 
-        self.rfc2136_client = _RFC2136Client(SERVER, NAME, SECRET, dns.tsig.HMAC_MD5)
+        self.rfc2136_client = _RFC2136Client(SERVER, PORT, NAME, SECRET, dns.tsig.HMAC_MD5)
 
     @mock.patch("dns.query.tcp")
     def test_add_txt_record(self, query_mock):
@@ -82,9 +83,9 @@ class RFC2136ClientTest(unittest.TestCase):
         # _find_domain | pylint: disable=protected-access
         self.rfc2136_client._find_domain = mock.MagicMock(return_value="example.com")
 
-        self.rfc2136_client.add_txt_record(DOMAIN, "bar", "baz", 42)
+        self.rfc2136_client.add_txt_record("bar", "baz", 42)
 
-        query_mock.assert_called_with(mock.ANY, SERVER)
+        query_mock.assert_called_with(mock.ANY, SERVER, port=PORT)
         self.assertTrue("bar. 42 IN TXT \"baz\"" in str(query_mock.call_args[0][0]))
 
     @mock.patch("dns.query.tcp")
@@ -96,7 +97,7 @@ class RFC2136ClientTest(unittest.TestCase):
         self.assertRaises(
             errors.PluginError,
             self.rfc2136_client.add_txt_record,
-            DOMAIN, "bar", "baz", 42)
+             "bar", "baz", 42)
 
     @mock.patch("dns.query.tcp")
     def test_add_txt_record_server_error(self, query_mock):
@@ -107,7 +108,7 @@ class RFC2136ClientTest(unittest.TestCase):
         self.assertRaises(
             errors.PluginError,
             self.rfc2136_client.add_txt_record,
-            DOMAIN, "bar", "baz", 42)
+             "bar", "baz", 42)
 
     @mock.patch("dns.query.tcp")
     def test_del_txt_record(self, query_mock):
@@ -115,9 +116,9 @@ class RFC2136ClientTest(unittest.TestCase):
         # _find_domain | pylint: disable=protected-access
         self.rfc2136_client._find_domain = mock.MagicMock(return_value="example.com")
 
-        self.rfc2136_client.del_txt_record(DOMAIN, "bar", "baz")
+        self.rfc2136_client.del_txt_record("bar", "baz")
 
-        query_mock.assert_called_with(mock.ANY, SERVER)
+        query_mock.assert_called_with(mock.ANY, SERVER, port=PORT)
         self.assertTrue("bar. 0 NONE TXT \"baz\"" in str(query_mock.call_args[0][0]))
 
     @mock.patch("dns.query.tcp")
@@ -129,7 +130,7 @@ class RFC2136ClientTest(unittest.TestCase):
         self.assertRaises(
             errors.PluginError,
             self.rfc2136_client.del_txt_record,
-            DOMAIN, "bar", "baz")
+             "bar", "baz")
 
     @mock.patch("dns.query.tcp")
     def test_del_txt_record_server_error(self, query_mock):
@@ -140,7 +141,7 @@ class RFC2136ClientTest(unittest.TestCase):
         self.assertRaises(
             errors.PluginError,
             self.rfc2136_client.del_txt_record,
-            DOMAIN, "bar", "baz")
+             "bar", "baz")
 
     def test_find_domain(self):
         # _query_soa | pylint: disable=protected-access
@@ -169,7 +170,7 @@ class RFC2136ClientTest(unittest.TestCase):
         # _query_soa | pylint: disable=protected-access
         result = self.rfc2136_client._query_soa(DOMAIN)
 
-        query_mock.assert_called_with(mock.ANY, SERVER)
+        query_mock.assert_called_with(mock.ANY, SERVER, port=PORT)
         self.assertTrue(result == True)
 
     @mock.patch("dns.query.udp")
@@ -179,7 +180,7 @@ class RFC2136ClientTest(unittest.TestCase):
         # _query_soa | pylint: disable=protected-access
         result = self.rfc2136_client._query_soa(DOMAIN)
 
-        query_mock.assert_called_with(mock.ANY, SERVER)
+        query_mock.assert_called_with(mock.ANY, SERVER, port=PORT)
         self.assertTrue(result == False)
 
     @mock.patch("dns.query.udp")

certbot-dns-rfc2136/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-rfc2136's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_rfc2136
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_rfc2136
-   :members:
-
 
 
 Indices and tables

certbot-dns-rfc2136/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-rfc2136/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-rfc2136[docs]

certbot-dns-rfc2136/setup.py

@@ -4,17 +4,16 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
-# Please update tox.ini when modifying dependency version requirements
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
     'dnspython',
     'mock',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.interface',
 ]
 
@@ -31,6 +30,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -39,10 +39,8 @@ setup(
         'Operating System :: POSIX :: Linux',
         'Programming Language :: Python',
         'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-dns-route53/Dockerfile

@@ -0,0 +1,5 @@
+FROM certbot/certbot
+
+COPY . src/certbot-dns-route53
+
+RUN pip install --no-cache-dir --editable src/certbot-dns-route53

certbot-dns-route53/certbot_dns_route53/dns_route53.py

@@ -1,4 +1,5 @@
 """Certbot Route53 authenticator plugin."""
+import collections
 import logging
 import time
 
@@ -33,6 +34,7 @@ class Authenticator(dns_common.DNSAuthenticator):
     def __init__(self, *args, **kwargs):
         super(Authenticator, self).__init__(*args, **kwargs)
         self.r53 = boto3.client("route53")
+        self._resource_records = collections.defaultdict(list)
 
     def more_info(self):  # pylint: disable=missing-docstring,no-self-use
         return "Solve a DNS01 challenge using AWS Route53"
@@ -40,14 +42,26 @@ class Authenticator(dns_common.DNSAuthenticator):
     def _setup_credentials(self):
         pass
 
-    def _perform(self, domain, validation_domain_name, validation):
-        try:
-            change_id = self._change_txt_record("UPSERT", validation_domain_name, validation)
+    def _perform(self, domain, validation_domain_name, validation): # pylint: disable=missing-docstring
+        pass
 
-            self._wait_for_change(change_id)
+    def perform(self, achalls):
+        self._attempt_cleanup = True
+
+        try:
+            change_ids = [
+                self._change_txt_record("UPSERT",
+                  achall.validation_domain_name(achall.domain),
+                  achall.validation(achall.account_key))
+                for achall in achalls
+            ]
+
+            for change_id in change_ids:
+                self._wait_for_change(change_id)
         except (NoCredentialsError, ClientError) as e:
             logger.debug('Encountered error during perform: %s', e, exc_info=True)
             raise errors.PluginError("\n".join([str(e), INSTRUCTIONS]))
+        return [achall.response(achall.account_key) for achall in achalls]
 
     def _cleanup(self, domain, validation_domain_name, validation):
         try:
@@ -88,6 +102,20 @@ class Authenticator(dns_common.DNSAuthenticator):
     def _change_txt_record(self, action, validation_domain_name, validation):
         zone_id = self._find_zone_id_for_domain(validation_domain_name)
 
+        rrecords = self._resource_records[validation_domain_name]
+        challenge = {"Value": '"{0}"'.format(validation)}
+        if action == "DELETE":
+            # Remove the record being deleted from the list of tracked records
+            rrecords.remove(challenge)
+            if rrecords:
+                # Need to update instead, as we're not deleting the rrset
+                action = "UPSERT"
+            else:
+                # Create a new list containing the record to use with DELETE
+                rrecords = [challenge]
+        else:
+            rrecords.append(challenge)
+
         response = self.r53.change_resource_record_sets(
             HostedZoneId=zone_id,
             ChangeBatch={
@@ -99,11 +127,7 @@ class Authenticator(dns_common.DNSAuthenticator):
                             "Name": validation_domain_name,
                             "Type": "TXT",
                             "TTL": self.ttl,
-                            "ResourceRecords": [
-                                # For some reason TXT records need to be
-                                # manually quoted.
-                                {"Value": '"{0}"'.format(validation)}
-                            ],
+                            "ResourceRecords": rrecords,
                         }
                     }
                 ]

certbot-dns-route53/certbot_dns_route53/dns_route53_test.py

@@ -186,6 +186,48 @@ class ClientTest(unittest.TestCase):
         call_count = self.client.r53.change_resource_record_sets.call_count
         self.assertEqual(call_count, 1)
 
+    def test_change_txt_record_delete(self):
+        self.client._find_zone_id_for_domain = mock.MagicMock()
+        self.client.r53.change_resource_record_sets = mock.MagicMock(
+            return_value={"ChangeInfo": {"Id": 1}})
+
+        validation = "some-value"
+        validation_record = {"Value": '"{0}"'.format(validation)}
+        self.client._resource_records[DOMAIN] = [validation_record]
+
+        self.client._change_txt_record("DELETE", DOMAIN, validation)
+
+        call_count = self.client.r53.change_resource_record_sets.call_count
+        self.assertEqual(call_count, 1)
+        call_args = self.client.r53.change_resource_record_sets.call_args_list[0][1]
+        call_args_batch = call_args["ChangeBatch"]["Changes"][0]
+        self.assertEqual(call_args_batch["Action"], "DELETE")
+        self.assertEqual(
+            call_args_batch["ResourceRecordSet"]["ResourceRecords"],
+            [validation_record])
+
+    def test_change_txt_record_multirecord(self):
+        self.client._find_zone_id_for_domain = mock.MagicMock()
+        self.client._get_validation_rrset = mock.MagicMock()
+        self.client._resource_records[DOMAIN] = [
+            {"Value": "\"pre-existing-value\""},
+            {"Value": "\"pre-existing-value-two\""},
+        ]
+        self.client.r53.change_resource_record_sets = mock.MagicMock(
+            return_value={"ChangeInfo": {"Id": 1}})
+
+        self.client._change_txt_record("DELETE", DOMAIN, "pre-existing-value")
+
+        call_count = self.client.r53.change_resource_record_sets.call_count
+        call_args = self.client.r53.change_resource_record_sets.call_args_list[0][1]
+        call_args_batch = call_args["ChangeBatch"]["Changes"][0]
+        self.assertEqual(call_args_batch["Action"], "UPSERT")
+        self.assertEqual(
+            call_args_batch["ResourceRecordSet"]["ResourceRecords"],
+            [{"Value": "\"pre-existing-value-two\""}])
+
+        self.assertEqual(call_count, 1)
+
     def test_wait_for_change(self):
         self.client.r53.get_change = mock.MagicMock(
             side_effect=[{"ChangeInfo": {"Status": "PENDING"}},

certbot-dns-route53/docs/index.rst

@@ -10,14 +10,14 @@ Welcome to certbot-dns-route53's documentation!
    :maxdepth: 2
    :caption: Contents:
 
+.. automodule:: certbot_dns_route53
+   :members:
+
 .. toctree::
    :maxdepth: 1
 
    api
 
-.. automodule:: certbot_dns_route53
-   :members:
-
 
 
 Indices and tables

certbot-dns-route53/local-oldest-requirements.txt

@@ -0,0 +1,2 @@
+acme[dev]==0.21.1
+certbot[dev]==0.21.1

certbot-dns-route53/readthedocs.org.requirements.txt

@@ -0,0 +1,12 @@
+# readthedocs.org gives no way to change the install command to "pip
+# install -e .[docs]" (that would in turn install documentation
+# dependencies), but it allows to specify a requirements.txt file at
+# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
+
+# Although ReadTheDocs certainly doesn't need to install the project
+# in --editable mode (-e), just "pip install .[docs]" does not work as
+# expected and "pip install -e .[docs]" must be used instead
+
+-e acme
+-e .
+-e certbot-dns-route53[docs]

certbot-dns-route53/setup.py

@@ -3,16 +3,16 @@ import sys
 from distutils.core import setup
 from setuptools import find_packages
 
-version = '0.22.0.dev0'
+version = '0.24.0.dev0'
 
+# Remember to update local-oldest-requirements.txt when changing the minimum
+# acme/certbot version.
 install_requires = [
-    'acme=={0}'.format(version),
-    'certbot=={0}'.format(version),
+    'acme>=0.21.1',
+    'certbot>=0.21.1',
     'boto3',
     'mock',
-    # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
-    # will tolerate; see #2599:
-    'setuptools>=1.0',
+    'setuptools',
     'zope.interface',
 ]
 
@@ -24,6 +24,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
     classifiers=[
         'Development Status :: 3 - Alpha',
         'Environment :: Plugins',
@@ -32,10 +33,8 @@ setup(
         'Operating System :: POSIX :: Linux',
         'Programming Language :: Python',
         'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',

certbot-nginx/certbot_nginx/configurator.py

@@ -23,6 +23,7 @@ from certbot import util
 from certbot.plugins import common
 
 from certbot_nginx import constants
+from certbot_nginx import display_ops
 from certbot_nginx import nginxparser
 from certbot_nginx import parser
 from certbot_nginx import tls_sni_01
@@ -60,6 +61,9 @@ class NginxConfigurator(common.Installer):
 
     DEFAULT_LISTEN_PORT = '80'
 
+    # SSL directives that Certbot can add when installing a new certificate.
+    SSL_DIRECTIVES = ['ssl_certificate', 'ssl_certificate_key', 'ssl_dhparam']
+
     @classmethod
     def add_parser_arguments(cls, add):
         add("server-root", default=constants.CLI_DEFAULTS["server_root"],
@@ -92,6 +96,11 @@ class NginxConfigurator(common.Installer):
         # For creating new vhosts if no names match
         self.new_vhost = None
 
+        # List of vhosts configured per wildcard domain on this run.
+        # used by deploy_cert() and enhance()
+        self._wildcard_vhosts = {}
+        self._wildcard_redirect_vhosts = {}
+
         # Add number of outstanding challenges
         self._chall_out = 0
 
@@ -99,6 +108,7 @@ class NginxConfigurator(common.Installer):
         self.parser = None
         self.version = version
         self._enhance_func = {"redirect": self._enable_redirect,
+                              "ensure-http-header": self._set_http_header,
                               "staple-ocsp": self._enable_ocsp_stapling}
 
         self.reverter.recovery_routine()
@@ -146,6 +156,7 @@ class NginxConfigurator(common.Installer):
             raise errors.PluginError(
                 'Unable to lock %s', self.conf('server-root'))
 
+
     # Entry point in main.py for installing cert
     def deploy_cert(self, domain, cert_path, key_path,
                     chain_path=None, fullchain_path=None):
@@ -166,14 +177,24 @@ class NginxConfigurator(common.Installer):
                 "The nginx plugin currently requires --fullchain-path to "
                 "install a cert.")
 
-        vhost = self.choose_vhost(domain, create_if_no_match=True)
+        vhosts = self.choose_vhosts(domain, create_if_no_match=True)
+        for vhost in vhosts:
+            self._deploy_cert(vhost, cert_path, key_path, chain_path, fullchain_path)
+
+    def _deploy_cert(self, vhost, cert_path, key_path, chain_path, fullchain_path):
+        # pylint: disable=unused-argument
+        """
+        Helper function for deploy_cert() that handles the actual deployment
+        this exists because we might want to do multiple deployments per
+        domain originally passed for deploy_cert(). This is especially true
+        with wildcard certificates
+        """
         cert_directives = [['\n    ', 'ssl_certificate', ' ', fullchain_path],
                            ['\n    ', 'ssl_certificate_key', ' ', key_path]]
 
-        self.parser.add_server_directives(vhost,
-                                          cert_directives, replace=True)
-        logger.info("Deployed Certificate to VirtualHost %s for %s",
-                    vhost.filep, ", ".join(vhost.names))
+        self.parser.update_or_add_server_directives(vhost,
+                                          cert_directives)
+        logger.info("Deploying Certificate to VirtualHost %s", vhost.filep)
 
         self.save_notes += ("Changed vhost at %s with addresses of %s\n" %
                             (vhost.filep,
@@ -181,10 +202,61 @@ class NginxConfigurator(common.Installer):
         self.save_notes += "\tssl_certificate %s\n" % fullchain_path
         self.save_notes += "\tssl_certificate_key %s\n" % key_path
 
+    def _choose_vhosts_wildcard(self, domain, prefer_ssl, no_ssl_filter_port=None):
+        """Prompts user to choose vhosts to install a wildcard certificate for"""
+        if prefer_ssl:
+            vhosts_cache = self._wildcard_vhosts
+            preference_test = lambda x: x.ssl
+        else:
+            vhosts_cache = self._wildcard_redirect_vhosts
+            preference_test = lambda x: not x.ssl
+
+        # Caching!
+        if domain in vhosts_cache:
+            # Vhosts for a wildcard domain were already selected
+            return vhosts_cache[domain]
+
+        # Get all vhosts whether or not they are covered by the wildcard domain
+        vhosts = self.parser.get_vhosts()
+
+        # Go through the vhosts, making sure that we cover all the names
+        # present, but preferring the SSL or non-SSL vhosts
+        filtered_vhosts = {}
+        for vhost in vhosts:
+            # Ensure we're listening non-sslishly on no_ssl_filter_port
+            if no_ssl_filter_port is not None:
+                if not self._vhost_listening_on_port_no_ssl(vhost, no_ssl_filter_port):
+                    continue
+            for name in vhost.names:
+                if preference_test(vhost):
+                    # Prefer either SSL or non-SSL vhosts
+                    filtered_vhosts[name] = vhost
+                elif name not in filtered_vhosts:
+                    # Add if not in list previously
+                    filtered_vhosts[name] = vhost
+
+        # Only unique VHost objects
+        dialog_input = set([vhost for vhost in filtered_vhosts.values()])
+
+        # Ask the user which of names to enable, expect list of names back
+        return_vhosts = display_ops.select_vhost_multiple(list(dialog_input))
+
+        for vhost in return_vhosts:
+            if domain not in vhosts_cache:
+                vhosts_cache[domain] = []
+            vhosts_cache[domain].append(vhost)
+
+        return return_vhosts
+
     #######################
     # Vhost parsing methods
     #######################
-    def choose_vhost(self, target_name, create_if_no_match=False):
+    def _choose_vhost_single(self, target_name):
+        matches = self._get_ranked_matches(target_name)
+        vhosts = [x for x in [self._select_best_name_match(matches)] if x is not None]
+        return vhosts
+
+    def choose_vhosts(self, target_name, create_if_no_match=False):
         """Chooses a virtual host based on the given domain name.
 
         .. note:: This makes the vhost SSL-enabled if it isn't already. Follows
@@ -202,17 +274,19 @@ class NginxConfigurator(common.Installer):
             when there is no match found. If we can't choose a default, raise a
             MisconfigurationError.
 
-        :returns: ssl vhost associated with name
-        :rtype: :class:`~certbot_nginx.obj.VirtualHost`
+        :returns: ssl vhosts associated with name
+        :rtype: list of :class:`~certbot_nginx.obj.VirtualHost`
 
         """
-        vhost = None
-
-        matches = self._get_ranked_matches(target_name)
-        vhost = self._select_best_name_match(matches)
-        if not vhost:
+        if util.is_wildcard_domain(target_name):
+            # Ask user which VHosts to support.
+            vhosts = self._choose_vhosts_wildcard(target_name, prefer_ssl=True)
+        else:
+            vhosts = self._choose_vhost_single(target_name)
+        if not vhosts:
             if create_if_no_match:
-                vhost = self._vhost_from_duplicated_default(target_name)
+                # result will not be [None] because it errors on failure
+                vhosts = [self._vhost_from_duplicated_default(target_name)]
             else:
                 # No matches. Raise a misconfiguration error.
                 raise errors.MisconfigurationError(
@@ -222,10 +296,11 @@ class NginxConfigurator(common.Installer):
                              "nginx configuration: "
                              "https://nginx.org/en/docs/http/server_names.html") % (target_name))
         # Note: if we are enhancing with ocsp, vhost should already be ssl.
-        if not vhost.ssl:
-            self._make_server_ssl(vhost)
+        for vhost in vhosts:
+            if not vhost.ssl:
+                self._make_server_ssl(vhost)
 
-        return vhost
+        return vhosts
 
     def ipv6_info(self, port):
         """Returns tuple of booleans (ipv6_active, ipv6only_present)
@@ -240,6 +315,9 @@ class NginxConfigurator(common.Installer):
             configuration, and existence of ipv6only directive for specified port
         :rtype: tuple of type (bool, bool)
         """
+        # port should be a string, but it's easy to mess up, so let's
+        # make sure it is one
+        port = str(port)
         vhosts = self.parser.get_vhosts()
         ipv6_active = False
         ipv6only_present = False
@@ -254,7 +332,8 @@ class NginxConfigurator(common.Installer):
     def _vhost_from_duplicated_default(self, domain, port=None):
         if self.new_vhost is None:
             default_vhost = self._get_default_vhost(port)
-            self.new_vhost = self.parser.duplicate_vhost(default_vhost, delete_default=True)
+            self.new_vhost = self.parser.duplicate_vhost(default_vhost,
+                remove_singleton_listen_params=True)
             self.new_vhost.names = set()
 
         self._add_server_name_to_vhost(self.new_vhost, domain)
@@ -266,7 +345,7 @@ class NginxConfigurator(common.Installer):
         for name in vhost.names:
             name_block[0].append(' ')
             name_block[0].append(name)
-        self.parser.add_server_directives(vhost, name_block, replace=True)
+        self.parser.update_or_add_server_directives(vhost, name_block)
 
     def _get_default_vhost(self, port):
         vhost_list = self.parser.get_vhosts()
@@ -359,7 +438,7 @@ class NginxConfigurator(common.Installer):
         return sorted(matches, key=lambda x: x['rank'])
 
 
-    def choose_redirect_vhost(self, target_name, port, create_if_no_match=False):
+    def choose_redirect_vhosts(self, target_name, port, create_if_no_match=False):
         """Chooses a single virtual host for redirect enhancement.
 
         Chooses the vhost most closely matching target_name that is
@@ -377,15 +456,20 @@ class NginxConfigurator(common.Installer):
             when there is no match found. If we can't choose a default, raise a
             MisconfigurationError.
 
-        :returns: vhost associated with name
-        :rtype: :class:`~certbot_nginx.obj.VirtualHost`
+        :returns: vhosts associated with name
+        :rtype: list of :class:`~certbot_nginx.obj.VirtualHost`
 
         """
-        matches = self._get_redirect_ranked_matches(target_name, port)
-        vhost = self._select_best_name_match(matches)
-        if not vhost and create_if_no_match:
-            vhost = self._vhost_from_duplicated_default(target_name, port=port)
-        return vhost
+        if util.is_wildcard_domain(target_name):
+            # Ask user which VHosts to enhance.
+            vhosts = self._choose_vhosts_wildcard(target_name, prefer_ssl=False,
+                no_ssl_filter_port=port)
+        else:
+            matches = self._get_redirect_ranked_matches(target_name, port)
+            vhosts = [x for x in [self._select_best_name_match(matches)]if x is not None]
+        if not vhosts and create_if_no_match:
+            vhosts = [self._vhost_from_duplicated_default(target_name, port=port)]
+        return vhosts
 
     def _port_matches(self, test_port, matching_port):
         # test_port is a number, matching is a number or "" or None
@@ -395,6 +479,23 @@ class NginxConfigurator(common.Installer):
         else:
             return test_port == matching_port
 
+    def _vhost_listening_on_port_no_ssl(self, vhost, port):
+        found_matching_port = False
+        if len(vhost.addrs) == 0:
+            # if there are no listen directives at all, Nginx defaults to
+            # listening on port 80.
+            found_matching_port = (port == self.DEFAULT_LISTEN_PORT)
+        else:
+            for addr in vhost.addrs:
+                if self._port_matches(port, addr.get_port()) and addr.ssl == False:
+                    found_matching_port = True
+
+        if found_matching_port:
+            # make sure we don't have an 'ssl on' directive
+            return not self.parser.has_ssl_on_directive(vhost)
+        else:
+            return False
+
     def _get_redirect_ranked_matches(self, target_name, port):
         """Gets a ranked list of plaintextish port-listening vhosts matching target_name
 
@@ -411,21 +512,7 @@ class NginxConfigurator(common.Installer):
         all_vhosts = self.parser.get_vhosts()
 
         def _vhost_matches(vhost, port):
-            found_matching_port = False
-            if len(vhost.addrs) == 0:
-                # if there are no listen directives at all, Nginx defaults to
-                # listening on port 80.
-                found_matching_port = (port == self.DEFAULT_LISTEN_PORT)
-            else:
-                for addr in vhost.addrs:
-                    if self._port_matches(port, addr.get_port()) and addr.ssl == False:
-                        found_matching_port = True
-
-            if found_matching_port:
-                # make sure we don't have an 'ssl on' directive
-                return not self.parser.has_ssl_on_directive(vhost)
-            else:
-                return False
+            return self._vhost_listening_on_port_no_ssl(vhost, port)
 
         matching_vhosts = [vhost for vhost in all_vhosts if _vhost_matches(vhost, port)]
 
@@ -498,7 +585,7 @@ class NginxConfigurator(common.Installer):
         # have it continue to do so.
         if len(vhost.addrs) == 0:
             listen_block = [['\n    ', 'listen', ' ', self.DEFAULT_LISTEN_PORT]]
-            self.parser.add_server_directives(vhost, listen_block, replace=False)
+            self.parser.add_server_directives(vhost, listen_block)
 
         if vhost.ipv6_enabled():
             ipv6_block = ['\n    ',
@@ -532,14 +619,14 @@ class NginxConfigurator(common.Installer):
         ])
 
         self.parser.add_server_directives(
-            vhost, ssl_block, replace=False)
+            vhost, ssl_block)
 
     ##################################
     # enhancement methods (IInstaller)
     ##################################
     def supported_enhancements(self):  # pylint: disable=no-self-use
         """Returns currently supported enhancements."""
-        return ['redirect', 'staple-ocsp']
+        return ['redirect', 'ensure-http-header', 'staple-ocsp']
 
     def enhance(self, domain, enhancement, options=None):
         """Enhance configuration.
@@ -565,13 +652,80 @@ class NginxConfigurator(common.Installer):
         test_redirect_block = _test_block_from_block(_redirect_block_for_domain(domain))
         return vhost.contains_list(test_redirect_block)
 
+    def _set_http_header(self, domain, header_substring):
+        """Enables header identified by header_substring on domain.
+
+        If the vhost is listening plaintextishly, separates out the relevant
+        directives into a new server block, and only add header directive to
+        HTTPS block.
+
+        :param str domain: the domain to enable header for.
+        :param str header_substring: String to uniquely identify a header.
+                        e.g. Strict-Transport-Security, Upgrade-Insecure-Requests
+        :returns: Success
+        :raises .errors.PluginError: If no viable HTTPS host can be created or
+            set with header header_substring.
+        """
+        vhosts = self.choose_vhosts(domain)
+        if not vhosts:
+            raise errors.PluginError(
+                "Unable to find corresponding HTTPS host for enhancement.")
+        for vhost in vhosts:
+            if vhost.has_header(header_substring):
+                raise errors.PluginEnhancementAlreadyPresent(
+                    "Existing %s header" % (header_substring))
+
+            # if there is no separate SSL block, break the block into two and
+            # choose the SSL block.
+            if vhost.ssl and any([not addr.ssl for addr in vhost.addrs]):
+                _, vhost = self._split_block(vhost)
+
+            header_directives = [
+                ['\n    ', 'add_header', ' ', header_substring, ' '] +
+                    constants.HEADER_ARGS[header_substring],
+                ['\n']]
+            self.parser.add_server_directives(vhost, header_directives)
+
     def _add_redirect_block(self, vhost, domain):
         """Add redirect directive to vhost
         """
         redirect_block = _redirect_block_for_domain(domain)
 
         self.parser.add_server_directives(
-            vhost, redirect_block, replace=False, insert_at_top=True)
+            vhost, redirect_block, insert_at_top=True)
+
+    def _split_block(self, vhost, only_directives=None):
+        """Splits this "virtual host" (i.e. this nginx server block) into
+        separate HTTP and HTTPS blocks.
+
+        :param vhost: The server block to break up into two.
+        :param list only_directives: If this exists, only duplicate these directives
+            when splitting the block.
+        :type vhost: :class:`~certbot_nginx.obj.VirtualHost`
+        :returns: tuple (http_vhost, https_vhost)
+        :rtype: tuple of type :class:`~certbot_nginx.obj.VirtualHost`
+        """
+        http_vhost = self.parser.duplicate_vhost(vhost, only_directives=only_directives)
+
+        def _ssl_match_func(directive):
+            return 'ssl' in directive
+
+        def _ssl_config_match_func(directive):
+            return self.mod_ssl_conf in directive
+
+        def _no_ssl_match_func(directive):
+            return 'ssl' not in directive
+
+        # remove all ssl addresses and related directives from the new block
+        for directive in self.SSL_DIRECTIVES:
+            self.parser.remove_server_directives(http_vhost, directive)
+        self.parser.remove_server_directives(http_vhost, 'listen', match_func=_ssl_match_func)
+        self.parser.remove_server_directives(http_vhost, 'include',
+                                             match_func=_ssl_config_match_func)
+
+        # remove all non-ssl addresses from the existing block
+        self.parser.remove_server_directives(vhost, 'listen', match_func=_no_ssl_match_func)
+        return http_vhost, vhost
 
     def _enable_redirect(self, domain, unused_options):
         """Redirect all equivalent HTTP traffic to ssl_vhost.
@@ -587,39 +741,40 @@ class NginxConfigurator(common.Installer):
         """
 
         port = self.DEFAULT_LISTEN_PORT
-        vhost = None
         # If there are blocks listening plaintextishly on self.DEFAULT_LISTEN_PORT,
         # choose the most name-matching one.
 
-        vhost = self.choose_redirect_vhost(domain, port)
+        vhosts = self.choose_redirect_vhosts(domain, port)
 
-        if vhost is None:
+        if not vhosts:
             logger.info("No matching insecure server blocks listening on port %s found.",
                 self.DEFAULT_LISTEN_PORT)
             return
 
-        new_vhost = None
+        for vhost in vhosts:
+            self._enable_redirect_single(domain, vhost)
+
+    def _enable_redirect_single(self, domain, vhost):
+        """Redirect all equivalent HTTP traffic to ssl_vhost.
+
+        If the vhost is listening plaintextishly, separate out the
+        relevant directives into a new server block and add a rewrite directive.
+
+        .. note:: This function saves the configuration
+
+        :param str domain: domain to enable redirect for
+        :param `~obj.Vhost` vhost: vhost to enable redirect for
+        """
+
+        http_vhost = None
         if vhost.ssl:
-            new_vhost = self.parser.duplicate_vhost(vhost,
-                only_directives=['listen', 'server_name'])
-
-            def _ssl_match_func(directive):
-                return 'ssl' in directive
-
-            def _no_ssl_match_func(directive):
-                return 'ssl' not in directive
-
-            # remove all ssl addresses from the new block
-            self.parser.remove_server_directives(new_vhost, 'listen', match_func=_ssl_match_func)
-
-            # remove all non-ssl addresses from the existing block
-            self.parser.remove_server_directives(vhost, 'listen', match_func=_no_ssl_match_func)
+            http_vhost, _ = self._split_block(vhost, ['listen', 'server_name'])
 
             # Add this at the bottom to get the right order of directives
             return_404_directive = [['\n    ', 'return', ' ', '404']]
-            self.parser.add_server_directives(new_vhost, return_404_directive, replace=False)
+            self.parser.add_server_directives(http_vhost, return_404_directive)
 
-            vhost = new_vhost
+            vhost = http_vhost
 
         if self._has_certbot_redirect(vhost, domain):
             logger.info("Traffic on port %s already redirecting to ssl in %s",
@@ -638,7 +793,18 @@ class NginxConfigurator(common.Installer):
         :type chain_path: `str` or `None`
 
         """
-        vhost = self.choose_vhost(domain)
+        vhosts = self.choose_vhosts(domain)
+        for vhost in vhosts:
+            self._enable_ocsp_stapling_single(vhost, chain_path)
+
+    def _enable_ocsp_stapling_single(self, vhost, chain_path):
+        """Include OCSP response in TLS handshake
+
+        :param str vhost: vhost to enable OCSP response for
+        :param chain_path: chain file path
+        :type chain_path: `str` or `None`
+
+        """
         if self.version < (1, 3, 7):
             raise errors.PluginError("Version 1.3.7 or greater of nginx "
                                      "is needed to enable OCSP stapling")
@@ -656,7 +822,7 @@ class NginxConfigurator(common.Installer):
 
         try:
             self.parser.add_server_directives(vhost,
-                                              stapling_directives, replace=False)
+                                              stapling_directives)
         except errors.MisconfigurationError as error:
             logger.debug(error)
             raise errors.PluginError("An error occurred while enabling OCSP "
@@ -730,7 +896,7 @@ class NginxConfigurator(common.Installer):
             raise errors.PluginError(
                 "Unable to run %s -V" % self.conf('ctl'))
 
-        version_regex = re.compile(r"nginx/([0-9\.]*)", re.IGNORECASE)
+        version_regex = re.compile(r"nginx version: ([^/]+)/([0-9\.]*)", re.IGNORECASE)
         version_matches = version_regex.findall(text)
 
         sni_regex = re.compile(r"TLS SNI support enabled", re.IGNORECASE)
@@ -747,7 +913,12 @@ class NginxConfigurator(common.Installer):
         if not sni_matches:
             raise errors.PluginError("Nginx build doesn't support SNI")
 
-        nginx_version = tuple([int(i) for i in version_matches[0].split(".")])
+        product_name, product_version = version_matches[0]
+        if product_name != 'nginx':
+            logger.warning("NGINX derivative %s is not officially supported by"
+                           " certbot", product_name)
+
+        nginx_version = tuple([int(i) for i in product_version.split(".")])
 
         # nginx < 0.8.48 uses machine hostname as default server_name instead of
         # the empty string
@@ -889,14 +1060,23 @@ def _test_block_from_block(block):
     parser.comment_directive(test_block, 0)
     return test_block[:-1]
 
+
 def _redirect_block_for_domain(domain):
+    updated_domain = domain
+    match_symbol = '='
+    if util.is_wildcard_domain(domain):
+        match_symbol = '~'
+        updated_domain = updated_domain.replace('.', r'\.')
+        updated_domain = updated_domain.replace('*', '[^.]+')
+        updated_domain = '^' + updated_domain + '$'
     redirect_block = [[
-        ['\n    ', 'if', ' ', '($host', ' ', '=', ' ', '%s)' % domain, ' '],
+        ['\n    ', 'if', ' ', '($host', ' ', match_symbol, ' ', '%s)' % updated_domain, ' '],
         [['\n        ', 'return', ' ', '301', ' ', 'https://$host$request_uri'],
         '\n    ']],
         ['\n']]
     return redirect_block
 
+
 def nginx_restart(nginx_ctl, nginx_conf):
     """Restarts the Nginx Server.
 

certbot-nginx/certbot_nginx/constants.py

@@ -44,3 +44,7 @@ def os_constant(key):
     :return: value of constant for active os
     """
     return CLI_DEFAULTS[key]
+
+HSTS_ARGS = ['\"max-age=31536000\"', ' ', 'always']
+
+HEADER_ARGS = {'Strict-Transport-Security': HSTS_ARGS}

certbot-nginx/certbot_nginx/display_ops.py

@@ -0,0 +1,44 @@
+"""Contains UI methods for Nginx operations."""
+import logging
+
+import zope.component
+
+from certbot import interfaces
+
+import certbot.display.util as display_util
+
+
+logger = logging.getLogger(__name__)
+
+
+def select_vhost_multiple(vhosts):
+    """Select multiple Vhosts to install the certificate for
+    :param vhosts: Available Nginx VirtualHosts
+    :type vhosts: :class:`list` of type `~obj.Vhost`
+    :returns: List of VirtualHosts
+    :rtype: :class:`list`of type `~obj.Vhost`
+    """
+    if not vhosts:
+        return list()
+    tags_list = [vhost.display_repr()+"\n" for vhost in vhosts]
+    # Remove the extra newline from the last entry
+    if len(tags_list):
+        tags_list[-1] = tags_list[-1][:-1]
+    code, names = zope.component.getUtility(interfaces.IDisplay).checklist(
+        "Which server blocks would you like to modify?",
+        tags=tags_list, force_interactive=True)
+    if code == display_util.OK:
+        return_vhosts = _reversemap_vhosts(names, vhosts)
+        return return_vhosts
+    return []
+
+def _reversemap_vhosts(names, vhosts):
+    """Helper function for select_vhost_multiple for mapping string
+    representations back to actual vhost objects"""
+    return_vhosts = list()
+
+    for selection in names:
+        for vhost in vhosts:
+            if vhost.display_repr().strip() == selection.strip():
+                return_vhosts.append(vhost)
+    return return_vhosts

certbot-nginx/certbot_nginx/http_01.py

@@ -159,16 +159,22 @@ class NginxHttp01(common.ChallengePerformer):
         document_root = os.path.join(
             self.configurator.config.work_dir, "http_01_nonexistent")
 
+        block.extend([['server_name', ' ', achall.domain],
+                      ['root', ' ', document_root],
+                      self._location_directive_for_achall(achall)
+                      ])
+        # TODO: do we want to return something else if they otherwise access this block?
+        return [['server'], block]
+
+    def _location_directive_for_achall(self, achall):
         validation = achall.validation(achall.account_key)
         validation_path = self._get_validation_path(achall)
 
-        block.extend([['server_name', ' ', achall.domain],
-                      ['root', ' ', document_root],
-                      [['location', ' ', '=', ' ', validation_path],
-                        [['default_type', ' ', 'text/plain'],
-                         ['return', ' ', '200', ' ', validation]]]])
-        # TODO: do we want to return something else if they otherwise access this block?
-        return [['server'], block]
+        location_directive = [['location', ' ', '=', ' ', validation_path],
+                              [['default_type', ' ', 'text/plain'],
+                               ['return', ' ', '200', ' ', validation]]]
+        return location_directive
+
 
     def _make_or_mod_server_block(self, achall):
         """Modifies a server block to respond to a challenge.
@@ -179,25 +185,24 @@ class NginxHttp01(common.ChallengePerformer):
 
         """
         try:
-            vhost = self.configurator.choose_redirect_vhost(achall.domain,
+            vhosts = self.configurator.choose_redirect_vhosts(achall.domain,
                 '%i' % self.configurator.config.http01_port, create_if_no_match=True)
         except errors.MisconfigurationError:
             # Couldn't find either a matching name+port server block
             # or a port+default_server block, so create a dummy block
             return self._make_server_block(achall)
 
-        # Modify existing server block
-        validation = achall.validation(achall.account_key)
-        validation_path = self._get_validation_path(achall)
+        # len is max 1 because Nginx doesn't authenticate wildcards
+        # if len were or vhosts None, we would have errored
+        vhost = vhosts[0]
 
-        location_directive = [[['location', ' ', '=', ' ', validation_path],
-                               [['default_type', ' ', 'text/plain'],
-                                ['return', ' ', '200', ' ', validation]]]]
+        # Modify existing server block
+        location_directive = [self._location_directive_for_achall(achall)]
 
         self.configurator.parser.add_server_directives(vhost,
-            location_directive, replace=False)
+            location_directive)
 
         rewrite_directive = [['rewrite', ' ', '^(/.well-known/acme-challenge/.*)',
                                 ' ', '$1', ' ', 'break']]
         self.configurator.parser.add_server_directives(vhost,
-            rewrite_directive, replace=False, insert_at_top=True)
+            rewrite_directive, insert_at_top=True)

certbot-nginx/certbot_nginx/obj.py

@@ -5,7 +5,7 @@ import six
 
 from certbot.plugins import common
 
-REDIRECT_DIRECTIVES = ['return', 'rewrite']
+ADD_HEADER_DIRECTIVE = 'add_header'
 
 class Addr(common.Addr):
     r"""Represents an Nginx address, i.e. what comes after the 'listen'
@@ -29,6 +29,8 @@ class Addr(common.Addr):
     :param str port: port number or "\*" or ""
     :param bool ssl: Whether the directive includes 'ssl'
     :param bool default: Whether the directive includes 'default_server'
+    :param bool default: Whether this is an IPv6 address
+    :param bool ipv6only: Whether the directive includes 'ipv6only=on'
 
     """
     UNSPECIFIED_IPV4_ADDRESSES = ('', '*', '0.0.0.0')
@@ -88,6 +90,8 @@ class Addr(common.Addr):
                 ssl = True
             elif nextpart == 'default_server':
                 default = True
+            elif nextpart == 'default':
+                default = True
             elif nextpart == "ipv6only=on":
                 ipv6only = True
 
@@ -193,6 +197,19 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
 
         return False
 
+    def __hash__(self):
+        return hash((self.filep, tuple(self.path),
+                     tuple(self.addrs), tuple(self.names),
+                     self.ssl, self.enabled))
+
+    def has_header(self, header_name):
+        """Determine if this server block has a particular header set.
+        :param str header_name: The name of the header to check for, e.g.
+            'Strict-Transport-Security'
+        """
+        found = _find_directive(self.raw, ADD_HEADER_DIRECTIVE, header_name)
+        return found is not None
+
     def contains_list(self, test):
         """Determine if raw server block contains test list at top level
         """
@@ -216,3 +233,31 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
         for a in self.addrs:
             if not a.ipv6:
                 return True
+
+    def display_repr(self):
+        """Return a representation of VHost to be used in dialog"""
+        return (
+            "File: {filename}\n"
+            "Addresses: {addrs}\n"
+            "Names: {names}\n"
+            "HTTPS: {https}\n".format(
+                filename=self.filep,
+                addrs=", ".join(str(addr) for addr in self.addrs),
+                names=", ".join(self.names),
+                https="Yes" if self.ssl else "No"))
+
+def _find_directive(directives, directive_name, match_content=None):
+    """Find a directive of type directive_name in directives. If match_content is given,
+       Searches for `match_content` in the directive arguments.
+    """
+    if not directives or isinstance(directives, six.string_types) or len(directives) == 0:
+        return None
+
+    # If match_content is None, just match on directive type. Otherwise, match on
+    # both directive type -and- the content!
+    if directives[0] == directive_name and \
+            (match_content is None or match_content in directives):
+        return directives
+
+    matches = (_find_directive(line, directive_name, match_content) for line in directives)
+    return next((m for m in matches if m is not None), None)

certbot-nginx/certbot_nginx/parser.py

@@ -276,15 +276,13 @@ class NginxParser(object):
 
         return False
 
-    def add_server_directives(self, vhost, directives, replace, insert_at_top=False):
-        """Add or replace directives in the server block identified by vhost.
+    def add_server_directives(self, vhost, directives, insert_at_top=False):
+        """Add directives to the server block identified by vhost.
 
         This method modifies vhost to be fully consistent with the new directives.
 
-        ..note :: If replace is True and the directive already exists, the first
-        instance will be replaced. Otherwise, the directive is added.
-        ..note :: If replace is False nothing gets added if an identical
-        block exists already.
+        ..note :: It's an error to try and add a nonrepeatable directive that already
+            exists in the config block with a conflicting value.
 
         ..todo :: Doesn't match server blocks whose server_name directives are
             split across multiple conf files.
@@ -292,13 +290,34 @@ class NginxParser(object):
         :param :class:`~certbot_nginx.obj.VirtualHost` vhost: The vhost
             whose information we use to match on
         :param list directives: The directives to add
-        :param bool replace: Whether to only replace existing directives
         :param bool insert_at_top: True if the directives need to be inserted at the top
             of the server block instead of the bottom
 
         """
         self._modify_server_directives(vhost,
-            functools.partial(_add_directives, directives, replace, insert_at_top))
+            functools.partial(_add_directives, directives, insert_at_top))
+
+    def update_or_add_server_directives(self, vhost, directives, insert_at_top=False):
+        """Add or replace directives in the server block identified by vhost.
+
+        This method modifies vhost to be fully consistent with the new directives.
+
+        ..note :: When a directive with the same name already exists in the
+        config block, the first instance will be replaced. Otherwise, the directive
+        will be appended/prepended to the config block as in add_server_directives.
+
+        ..todo :: Doesn't match server blocks whose server_name directives are
+            split across multiple conf files.
+
+        :param :class:`~certbot_nginx.obj.VirtualHost` vhost: The vhost
+            whose information we use to match on
+        :param list directives: The directives to add
+        :param bool insert_at_top: True if the directives need to be inserted at the top
+            of the server block instead of the bottom
+
+        """
+        self._modify_server_directives(vhost,
+            functools.partial(_update_or_add_directives, directives, insert_at_top))
 
     def remove_server_directives(self, vhost, directive_name, match_func=None):
         """Remove all directives of type directive_name.
@@ -335,13 +354,14 @@ class NginxParser(object):
         except errors.MisconfigurationError as err:
             raise errors.MisconfigurationError("Problem in %s: %s" % (filename, str(err)))
 
-    def duplicate_vhost(self, vhost_template, delete_default=False, only_directives=None):
+    def duplicate_vhost(self, vhost_template, remove_singleton_listen_params=False,
+        only_directives=None):
         """Duplicate the vhost in the configuration files.
 
         :param :class:`~certbot_nginx.obj.VirtualHost` vhost_template: The vhost
             whose information we copy
-        :param bool delete_default: If we should remove default_server
-            from listen directives in the block.
+        :param bool remove_singleton_listen_params: If we should remove parameters
+            from listen directives in the block that can only be used once per address
         :param list only_directives: If it exists, only duplicate the named directives. Only
             looks at first level of depth; does not expand includes.
 
@@ -368,15 +388,21 @@ class NginxParser(object):
 
         enclosing_block.append(raw_in_parsed)
         new_vhost.path[-1] = len(enclosing_block) - 1
-        if delete_default:
+        if remove_singleton_listen_params:
             for addr in new_vhost.addrs:
                 addr.default = False
+                addr.ipv6only = False
             for directive in enclosing_block[new_vhost.path[-1]][1]:
-                if (len(directive) > 0 and directive[0] == 'listen'
-                    and 'default_server' in directive):
-                    del directive[directive.index('default_server')]
+                if len(directive) > 0 and directive[0] == 'listen':
+                    if 'default_server' in directive:
+                        del directive[directive.index('default_server')]
+                    if 'default' in directive:
+                        del directive[directive.index('default')]
+                    if 'ipv6only=on' in directive:
+                        del directive[directive.index('ipv6only=on')]
         return new_vhost
 
+
 def _parse_ssl_options(ssl_options):
     if ssl_options is not None:
         try:
@@ -523,32 +549,23 @@ def _is_ssl_on_directive(entry):
             len(entry) == 2 and entry[0] == 'ssl' and
             entry[1] == 'on')
 
-def _add_directives(directives, replace, insert_at_top, block):
-    """Adds or replaces directives in a config block.
-
-    When replace=False, it's an error to try and add a nonrepeatable directive that already
-    exists in the config block with a conflicting value.
-
-    When replace=True and a directive with the same name already exists in the
-    config block, the first instance will be replaced. Otherwise, the directive
-    will be added to the config block.
-
-    ..todo :: Find directives that are in included files.
-
-    :param list directives: The new directives.
-    :param bool replace: Described above.
-    :param bool insert_at_top: Described above.
-    :param list block: The block to replace in
-
-    """
+def _add_directives(directives, insert_at_top, block):
+    """Adds directives to a config block."""
     for directive in directives:
-        _add_directive(block, directive, replace, insert_at_top)
+        _add_directive(block, directive, insert_at_top)
+    if block and '\n' not in block[-1]:  # could be "   \n  " or ["\n"] !
+        block.append(nginxparser.UnspacedList('\n'))
+
+def _update_or_add_directives(directives, insert_at_top, block):
+    """Adds or replaces directives in a config block."""
+    for directive in directives:
+        _update_or_add_directive(block, directive, insert_at_top)
     if block and '\n' not in block[-1]:  # could be "   \n  " or ["\n"] !
         block.append(nginxparser.UnspacedList('\n'))
 
 
 INCLUDE = 'include'
-REPEATABLE_DIRECTIVES = set(['server_name', 'listen', INCLUDE, 'location', 'rewrite'])
+REPEATABLE_DIRECTIVES = set(['server_name', 'listen', INCLUDE, 'rewrite'])
 COMMENT = ' managed by Certbot'
 COMMENT_BLOCK = [' ', '#', COMMENT]
 
@@ -600,28 +617,20 @@ def _find_location(block, directive_name, match_func=None):
     return next((index for index, line in enumerate(block) \
         if line and line[0] == directive_name and (match_func is None or match_func(line))), None)
 
-def _add_directive(block, directive, replace, insert_at_top):
-    """Adds or replaces a single directive in a config block.
+def _is_whitespace_or_comment(directive):
+    """Is this directive either a whitespace or comment directive?"""
+    return len(directive) == 0 or directive[0] == '#'
 
-    See _add_directives for more documentation.
-
-    """
-    directive = nginxparser.UnspacedList(directive)
-    def is_whitespace_or_comment(directive):
-        """Is this directive either a whitespace or comment directive?"""
-        return len(directive) == 0 or directive[0] == '#'
-    if is_whitespace_or_comment(directive):
+def _add_directive(block, directive, insert_at_top):
+    if not isinstance(directive, nginxparser.UnspacedList):
+        directive = nginxparser.UnspacedList(directive)
+    if _is_whitespace_or_comment(directive):
         # whitespace or comment
         block.append(directive)
         return
 
     location = _find_location(block, directive[0])
 
-    if replace:
-        if location is not None:
-            block[location] = directive
-            comment_directive(block, location)
-            return
     # Append or prepend directive. Fail if the name is not a repeatable directive name,
     # and there is already a copy of that directive with a different value
     # in the config file.
@@ -646,7 +655,7 @@ def _add_directive(block, directive, replace, insert_at_top):
         for included_directive in included_directives:
             included_dir_loc = _find_location(block, included_directive[0])
             included_dir_name = included_directive[0]
-            if not is_whitespace_or_comment(included_directive) \
+            if not _is_whitespace_or_comment(included_directive) \
                 and not can_append(included_dir_loc, included_dir_name):
                 if block[included_dir_loc] != included_directive:
                     raise errors.MisconfigurationError(err_fmt.format(included_directive,
@@ -667,6 +676,30 @@ def _add_directive(block, directive, replace, insert_at_top):
     elif block[location] != directive:
         raise errors.MisconfigurationError(err_fmt.format(directive, block[location]))
 
+def _update_directive(block, directive, location):
+    block[location] = directive
+    comment_directive(block, location)
+
+def _update_or_add_directive(block, directive, insert_at_top):
+    if not isinstance(directive, nginxparser.UnspacedList):
+        directive = nginxparser.UnspacedList(directive)
+    if _is_whitespace_or_comment(directive):
+        # whitespace or comment
+        block.append(directive)
+        return
+
+    location = _find_location(block, directive[0])
+
+    # we can update directive
+    if location is not None:
+        _update_directive(block, directive, location)
+        return
+
+    _add_directive(block, directive, insert_at_top)
+
+def _is_certbot_comment(directive):
+    return '#' in directive and COMMENT in directive
+
 def _remove_directives(directive_name, match_func, block):
     """Removes directives of name directive_name from a config block if match_func matches.
     """
@@ -674,6 +707,9 @@ def _remove_directives(directive_name, match_func, block):
         location = _find_location(block, directive_name, match_func=match_func)
         if location is None:
             return
+        # if the directive was made by us, remove the comment following
+        if location + 1 < len(block) and _is_certbot_comment(block[location + 1]):
+            del block[location + 1]
         del block[location]
 
 def _apply_global_addr_ssl(addr_to_ssl, parsed_server):
@@ -707,7 +743,7 @@ def _parse_server_raw(server):
                 if addr.ssl:
                     parsed_server['ssl'] = True
         elif directive[0] == 'server_name':
-            parsed_server['names'].update(directive[1:])
+            parsed_server['names'].update(x.strip('"\'') for x in directive[1:])
         elif _is_ssl_on_directive(directive):
             parsed_server['ssl'] = True
             apply_ssl_to_all_addrs = True

certbot-nginx/certbot_nginx/tests/configurator_test.py

@@ -94,7 +94,7 @@ class NginxConfiguratorTest(util.NginxTest):
              "globalssl.com", "globalsslsetssl.com", "ipv6.com", "ipv6ssl.com"]))
 
     def test_supported_enhancements(self):
-        self.assertEqual(['redirect', 'staple-ocsp'],
+        self.assertEqual(['redirect', 'ensure-http-header', 'staple-ocsp'],
                          self.config.supported_enhancements())
 
     def test_enhance(self):
@@ -113,8 +113,7 @@ class NginxConfiguratorTest(util.NginxTest):
                                      None, [0])
         self.config.parser.add_server_directives(
             mock_vhost,
-            [['listen', ' ', '5001', ' ', 'ssl']],
-            replace=False)
+            [['listen', ' ', '5001', ' ', 'ssl']])
         self.config.save()
 
         # pylint: disable=protected-access
@@ -128,7 +127,7 @@ class NginxConfiguratorTest(util.NginxTest):
                             ['#', parser.COMMENT]]]],
                          parsed[0])
 
-    def test_choose_vhost(self):
+    def test_choose_vhosts(self):
         localhost_conf = set(['localhost', r'~^(www\.)?(example|bar)\.'])
         server_conf = set(['somename', 'another.alias', 'alias'])
         example_conf = set(['.example.com', 'example.*'])
@@ -159,7 +158,7 @@ class NginxConfiguratorTest(util.NginxTest):
                        '69.255.225.155']
 
         for name in results:
-            vhost = self.config.choose_vhost(name)
+            vhost = self.config.choose_vhosts(name)[0]
             path = os.path.relpath(vhost.filep, self.temp_dir)
 
             self.assertEqual(results[name], vhost.names)
@@ -173,7 +172,7 @@ class NginxConfiguratorTest(util.NginxTest):
 
         for name in bad_results:
             self.assertRaises(errors.MisconfigurationError,
-                              self.config.choose_vhost, name)
+                              self.config.choose_vhosts, name)
 
     def test_ipv6only(self):
         # ipv6_info: (ipv6_active, ipv6only_present)
@@ -181,6 +180,18 @@ class NginxConfiguratorTest(util.NginxTest):
         # Port 443 has ipv6only=on because of ipv6ssl.com vhost
         self.assertEquals((True, True), self.config.ipv6_info("443"))
 
+    def test_ipv6only_detection(self):
+        self.config.version = (1, 3, 1)
+
+        self.config.deploy_cert(
+            "ipv6.com",
+            "example/cert.pem",
+            "example/key.pem",
+            "example/chain.pem",
+            "example/fullchain.pem")
+
+        for addr in self.config.choose_vhosts("ipv6.com")[0].addrs:
+            self.assertFalse(addr.ipv6only)
 
     def test_more_info(self):
         self.assertTrue('nginx.conf' in self.config.more_info())
@@ -194,9 +205,9 @@ class NginxConfiguratorTest(util.NginxTest):
             "example/chain.pem",
             None)
 
-    @mock.patch('certbot_nginx.parser.NginxParser.add_server_directives')
-    def test_deploy_cert_raise_on_add_error(self, mock_add_server_directives):
-        mock_add_server_directives.side_effect = errors.MisconfigurationError()
+    @mock.patch('certbot_nginx.parser.NginxParser.update_or_add_server_directives')
+    def test_deploy_cert_raise_on_add_error(self, mock_update_or_add_server_directives):
+        mock_update_or_add_server_directives.side_effect = errors.MisconfigurationError()
         self.assertRaises(
             errors.PluginError,
             self.config.deploy_cert,
@@ -498,6 +509,54 @@ class NginxConfiguratorTest(util.NginxTest):
                ['return', '404'], ['#', ' managed by Certbot'], [], [], []]]],
             generated_conf)
 
+    def test_split_for_headers(self):
+        example_conf = self.config.parser.abs_path('sites-enabled/example.com')
+        self.config.deploy_cert(
+            "example.org",
+            "example/cert.pem",
+            "example/key.pem",
+            "example/chain.pem",
+            "example/fullchain.pem")
+        self.config.enhance("www.example.com", "ensure-http-header", "Strict-Transport-Security")
+        generated_conf = self.config.parser.parsed[example_conf]
+        self.assertEqual(
+            [[['server'], [
+               ['server_name', '.example.com'],
+               ['server_name', 'example.*'], [],
+               ['listen', '5001', 'ssl'], ['#', ' managed by Certbot'],
+               ['ssl_certificate', 'example/fullchain.pem'], ['#', ' managed by Certbot'],
+               ['ssl_certificate_key', 'example/key.pem'], ['#', ' managed by Certbot'],
+               ['include', self.config.mod_ssl_conf], ['#', ' managed by Certbot'],
+               ['ssl_dhparam', self.config.ssl_dhparams], ['#', ' managed by Certbot'],
+               [], [],
+               ['add_header', 'Strict-Transport-Security', '"max-age=31536000"', 'always'],
+               ['#', ' managed by Certbot'],
+               [], []]],
+             [['server'], [
+               ['listen', '69.50.225.155:9000'],
+               ['listen', '127.0.0.1'],
+               ['server_name', '.example.com'],
+               ['server_name', 'example.*'],
+               [], [], []]]],
+            generated_conf)
+
+    def test_http_header_hsts(self):
+        example_conf = self.config.parser.abs_path('sites-enabled/example.com')
+        self.config.enhance("www.example.com", "ensure-http-header",
+                            "Strict-Transport-Security")
+        expected = ['add_header', 'Strict-Transport-Security', '"max-age=31536000"', 'always']
+        generated_conf = self.config.parser.parsed[example_conf]
+        self.assertTrue(util.contains_at_depth(generated_conf, expected, 2))
+
+    def test_http_header_hsts_twice(self):
+        self.config.enhance("www.example.com", "ensure-http-header",
+                            "Strict-Transport-Security")
+        self.assertRaises(
+            errors.PluginEnhancementAlreadyPresent,
+            self.config.enhance, "www.example.com",
+            "ensure-http-header", "Strict-Transport-Security")
+
+
     @mock.patch('certbot_nginx.obj.VirtualHost.contains_list')
     def test_certbot_redirect_exists(self, mock_contains_list):
         # Test that we add no redirect statement if there is already a
@@ -580,7 +639,7 @@ class NginxConfiguratorTest(util.NginxTest):
         self.assertEqual([[['server'],
                            [['listen', 'myhost', 'default_server'],
                             ['listen', 'otherhost', 'default_server'],
-                            ['server_name', 'www.example.org'],
+                            ['server_name', '"www.example.org"'],
                             [['location', '/'],
                              [['root', 'html'],
                               ['index', 'index.html', 'index.htm']]]]],
@@ -702,6 +761,100 @@ class NginxConfiguratorTest(util.NginxTest):
         self.config.rollback_checkpoints()
         self.assertTrue(mock_parser_load.call_count == 3)
 
+    def test_choose_vhosts_wildcard(self):
+        # pylint: disable=protected-access
+        mock_path = "certbot_nginx.display_ops.select_vhost_multiple"
+        with mock.patch(mock_path) as mock_select_vhs:
+            vhost = [x for x in self.config.parser.get_vhosts()
+              if 'summer.com' in x.names][0]
+            mock_select_vhs.return_value = [vhost]
+            vhs = self.config._choose_vhosts_wildcard("*.com",
+                                                     prefer_ssl=True)
+            # Check that the dialog was called with migration.com
+            self.assertTrue(vhost in mock_select_vhs.call_args[0][0])
+
+            # And the actual returned values
+            self.assertEquals(len(vhs), 1)
+            self.assertEqual(vhs[0], vhost)
+
+    def test_choose_vhosts_wildcard_redirect(self):
+        # pylint: disable=protected-access
+        mock_path = "certbot_nginx.display_ops.select_vhost_multiple"
+        with mock.patch(mock_path) as mock_select_vhs:
+            vhost = [x for x in self.config.parser.get_vhosts()
+              if 'summer.com' in x.names][0]
+            mock_select_vhs.return_value = [vhost]
+            vhs = self.config._choose_vhosts_wildcard("*.com",
+                                                     prefer_ssl=False)
+            # Check that the dialog was called with migration.com
+            self.assertTrue(vhost in mock_select_vhs.call_args[0][0])
+
+            # And the actual returned values
+            self.assertEquals(len(vhs), 1)
+            self.assertEqual(vhs[0], vhost)
+
+    def test_deploy_cert_wildcard(self):
+        # pylint: disable=protected-access
+        mock_choose_vhosts = mock.MagicMock()
+        vhost = [x for x in self.config.parser.get_vhosts()
+            if 'geese.com' in x.names][0]
+        mock_choose_vhosts.return_value = [vhost]
+        self.config._choose_vhosts_wildcard = mock_choose_vhosts
+        mock_d = "certbot_nginx.configurator.NginxConfigurator._deploy_cert"
+        with mock.patch(mock_d) as mock_dep:
+            self.config.deploy_cert("*.com", "/tmp/path",
+                                    "/tmp/path", "/tmp/path", "/tmp/path")
+            self.assertTrue(mock_dep.called)
+            self.assertEquals(len(mock_dep.call_args_list), 1)
+            self.assertEqual(vhost, mock_dep.call_args_list[0][0][0])
+
+    @mock.patch("certbot_nginx.display_ops.select_vhost_multiple")
+    def test_deploy_cert_wildcard_no_vhosts(self, mock_dialog):
+        # pylint: disable=protected-access
+        mock_dialog.return_value = []
+        self.assertRaises(errors.PluginError,
+                          self.config.deploy_cert,
+                          "*.wild.cat", "/tmp/path", "/tmp/path",
+                           "/tmp/path", "/tmp/path")
+
+    @mock.patch("certbot_nginx.display_ops.select_vhost_multiple")
+    def test_enhance_wildcard_ocsp_after_install(self, mock_dialog):
+        # pylint: disable=protected-access
+        vhost = [x for x in self.config.parser.get_vhosts()
+            if 'geese.com' in x.names][0]
+        self.config._wildcard_vhosts["*.com"] = [vhost]
+        self.config.enhance("*.com", "staple-ocsp", "example/chain.pem")
+        self.assertFalse(mock_dialog.called)
+
+    @mock.patch("certbot_nginx.display_ops.select_vhost_multiple")
+    def test_enhance_wildcard_redirect_or_ocsp_no_install(self, mock_dialog):
+        vhost = [x for x in self.config.parser.get_vhosts()
+            if 'summer.com' in x.names][0]
+        mock_dialog.return_value = [vhost]
+        self.config.enhance("*.com", "staple-ocsp", "example/chain.pem")
+        self.assertTrue(mock_dialog.called)
+
+    @mock.patch("certbot_nginx.display_ops.select_vhost_multiple")
+    def test_enhance_wildcard_double_redirect(self, mock_dialog):
+      # pylint: disable=protected-access
+        vhost = [x for x in self.config.parser.get_vhosts()
+            if 'summer.com' in x.names][0]
+        self.config._wildcard_redirect_vhosts["*.com"] = [vhost]
+        self.config.enhance("*.com", "redirect")
+        self.assertFalse(mock_dialog.called)
+
+    def test_choose_vhosts_wildcard_no_ssl_filter_port(self):
+        # pylint: disable=protected-access
+        mock_path = "certbot_nginx.display_ops.select_vhost_multiple"
+        with mock.patch(mock_path) as mock_select_vhs:
+            mock_select_vhs.return_value = []
+            self.config._choose_vhosts_wildcard("*.com",
+                                                prefer_ssl=False,
+                                                no_ssl_filter_port='80')
+            # Check that the dialog was called with only port 80 vhosts
+            self.assertEqual(len(mock_select_vhs.call_args[0][0]), 4)
+
+
 class InstallSslOptionsConfTest(util.NginxTest):
     """Test that the options-ssl-nginx.conf file is installed and updated properly."""
 

certbot-nginx/certbot_nginx/tests/display_ops_test.py

@@ -0,0 +1,45 @@
+"""Test certbot_apache.display_ops."""
+import unittest
+
+from certbot.display import util as display_util
+
+from certbot.tests import util as certbot_util
+
+from certbot_nginx import parser
+
+from certbot_nginx.display_ops import select_vhost_multiple
+from certbot_nginx.tests import util
+
+
+class SelectVhostMultiTest(util.NginxTest):
+    """Tests for certbot_nginx.display_ops.select_vhost_multiple."""
+
+    def setUp(self):
+        super(SelectVhostMultiTest, self).setUp()
+        nparser = parser.NginxParser(self.config_path)
+        self.vhosts = nparser.get_vhosts()
+
+    def test_select_no_input(self):
+        self.assertFalse(select_vhost_multiple([]))
+
+    @certbot_util.patch_get_utility()
+    def test_select_correct(self, mock_util):
+        mock_util().checklist.return_value = (
+            display_util.OK, [self.vhosts[3].display_repr(),
+                              self.vhosts[2].display_repr()])
+        vhs = select_vhost_multiple([self.vhosts[3],
+                                     self.vhosts[2],
+                                     self.vhosts[1]])
+        self.assertTrue(self.vhosts[2] in vhs)
+        self.assertTrue(self.vhosts[3] in vhs)
+        self.assertFalse(self.vhosts[1] in vhs)
+
+    @certbot_util.patch_get_utility()
+    def test_select_cancel(self, mock_util):
+        mock_util().checklist.return_value = (display_util.CANCEL, "whatever")
+        vhs = select_vhost_multiple([self.vhosts[2], self.vhosts[3]])
+        self.assertFalse(vhs)
+
+
+if __name__ == "__main__":
+    unittest.main()  # pragma: no cover