work work

letstest-certbot/LICENSE.txt

@@ -0,0 +1,190 @@
+   Copyright 2015 Electronic Frontier Foundation and others
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS

letstest-certbot/MANIFEST.in

@@ -0,0 +1,4 @@
+include LICENSE.txt
+include README.rst
+recursive-include docs *
+recursive-include letstest-certbot/testdata *

letstest-certbot/README.rst

@@ -0,0 +1,2 @@
+Let's test Certbot runs integration tests on Certbot against Let's Encrypt's
+ACME CA Boulder.

letstest-certbot/docs/.gitignore

@@ -0,0 +1 @@
+/_build/

letstest-certbot/docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = letstest-certbot
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

letstest-certbot/docs/api.rst

@@ -0,0 +1,8 @@
+=================
+API Documentation
+=================
+
+.. toctree::
+   :glob:
+
+   api/**

letstest-certbot/docs/conf.py

@@ -0,0 +1,180 @@
+# -*- coding: utf-8 -*-
+#
+# letstest-certbot documentation build configuration file, created by
+# sphinx-quickstart on Mon Jun 26 12:32:31 2017.
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = ['sphinx.ext.autodoc',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.todo',
+    'sphinx.ext.coverage',
+    'sphinx.ext.viewcode']
+
+autodoc_member_order = 'bysource'
+autodoc_default_flags = ['show-inheritance', 'private-members']
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'letstest-certbot'
+copyright = u'2017, Certbot Project'
+author = u'Certbot Project'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = u'0'
+# The full version, including alpha/beta/rc tags.
+release = u'0'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = 'en'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This patterns also effect to html_static_path and html_extra_path
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+default_role = 'py:obj'
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+
+# http://docs.readthedocs.org/en/latest/theme.html#how-do-i-use-this-locally-and-on-read-the-docs
+# on_rtd is whether we are on readthedocs.org
+on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
+if not on_rtd:  # only import and set the theme if we're building docs locally
+    import sphinx_rtd_theme
+    html_theme = 'sphinx_rtd_theme'
+    html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+# otherwise, readthedocs.org uses their theme by default, so no need to specify it
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+
+# -- Options for HTMLHelp output ------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'letstest-certbotdoc'
+
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'letstest-certbot.tex', u'letstest-certbot Documentation',
+     u'Certbot Project', 'manual'),
+]
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'letstest-certbot', u'letstest-certbot Documentation',
+     [author], 1)
+]
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'letstest-certbot', u'letstest-certbot Documentation',
+     author, 'letstest-certbot', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+
+
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/', None),
+    'acme': ('https://acme-python.readthedocs.org/en/latest/', None),
+    'certbot': ('https://certbot.eff.org/docs/', None),
+}

letstest-certbot/docs/index.rst

@@ -0,0 +1,28 @@
+.. letstest-certbot documentation master file, created by
+   sphinx-quickstart on Mon Jun 26 12:32:31 2017.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to letstest-certbot's documentation!
+============================================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+.. toctree::
+   :maxdepth: 1
+
+   api
+
+.. automodule:: letstest_certbot
+   :members:
+
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

letstest-certbot/docs/make.bat

@@ -0,0 +1,36 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+set SPHINXPROJ=letstest-certbot
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+
+:end
+popd

letstest-certbot/letstest_certbot/__main__.py

@@ -0,0 +1,11 @@
+"""Runs integration tests on Certbot against Boulder using Docker."""
+import letstest
+
+
+def main():
+    """Runs integration tests."""
+    letstest.main()
+
+
+if __name__ == '__main__':
+    main()

letstest-certbot/letstest_certbot/letstest.py

@@ -0,0 +1,526 @@
+"""Runs integration tests on Certbot against Boulder using Docker."""
+from __future__ import print_function
+
+import argparse
+import atexit
+import collections
+import functools
+import grp
+import json
+import logging
+import os
+import pkg_resources
+import shutil
+import subprocess
+import sys
+import tempfile
+
+
+TESTDATA_PATH = os.path.abspath(pkg_resources.resource_filename(
+    __name__, 'testdata'))
+"""Path to Let's test's test data."""
+
+
+COMPOSE_PATH = os.path.join(TESTDATA_PATH, 'docker-compose.yml')
+"""Path to Let's test's docker-compose.yml."""
+
+
+logger = logging.getLogger(__name__)
+
+
+class Error(Exception):
+    """Let's test error type."""
+
+
+def main():
+    """Runs integration tests."""
+    config = parse_args()
+    logging.basicConfig(level=config.verbose * -10)
+    logging.debug('Parse args are:\n%s', config)
+
+    verify_environment()
+    boulder_compose_path = set_up_boulder()
+    environ = set_up_certbot(config)
+
+    run_tests(boulder_compose_path, environ)
+
+
+def run_tests(boulder_compose_path, env):
+    """Runs Certbot tests with Boulder using Docker Compose.
+
+    :param str boulder_compose_path: path to boulder docker-compose.yml
+    :param env dict: environment variables for docker-compose
+
+    """
+    test_services = start_tests(boulder_compose_path, env)
+    list_services(test_services)
+    print_result(test_services)
+
+
+def start_tests(boulder_compose_path, env):
+    """Begins running tests.
+
+    atexit is used so Docker is properly cleaned up when the program
+    exits.
+
+    :param str boulder_compose_path: path to boulder docker-compose.yml
+    :param env dict: environment variables for docker-compose
+
+    :returns: Mapping from test service names to their containers
+    :rtype: dict
+
+    """
+    os.chdir(os.path.dirname(boulder_compose_path))
+    check_call(['docker-compose', '-f', boulder_compose_path,
+               '-f', COMPOSE_PATH, 'up', '-d'], env=env)
+
+    try:
+        test_services = get_test_services_and_containers(boulder_compose_path,
+                                                         env)
+        for container in test_services.values():
+            atexit.register(check_call, ['docker', 'stop', container])
+        # By omitting COMPOSE_PATH, the test services aren't deleted
+        atexit.register(check_call, 
+                        ['docker-compose', '-f', boulder_compose_path, 'down'],
+                        env=env)
+    except: 
+        check_call(['docker-compose', '-f',
+                    boulder_compose_path, '-f', COMPOSE_PATH, 'down'], env=env)
+        raise
+
+    return test_services
+
+
+def list_services(test_services):
+    """Prints output about the running test services.
+
+    :param dict test_services: map from service to container
+
+    """
+    for service, container in test_services.items():
+        print('{0} running in {1}'.format(service, container))
+    print()
+    print('You can access logs by running `docker logs -f <container>`.\n')
+
+
+def print_result(test_services):
+    """Waits for tests to finish and prints the results.
+
+    :param dict test_services: map from service to container
+    
+    """
+    failure = False
+
+    for service, container in test_services.items():
+        exit_code = int(check_output(['docker', 'wait', container]).strip())
+        if exit_code:
+            print('{0} failed'.format(service))
+        else:
+            print('{0} passed'.format(service))
+
+    if failure:
+        raise Error('One or more test instances failed!')
+
+
+def set_up_certbot(config):
+    """Sets up the system environment for testing.
+
+    The requested Certbot repo and branch is cloned to a temporary
+    directory and environment variables are set in preparation for
+    running tests.
+
+    :param argparse.Namespace config: parsed command line arguments
+
+    :returns: environment variables to use with docker-compose up
+    :rtype: dict
+
+    """
+    verify_script_path(config.test_script)
+
+    env = os.environ.copy()
+    env['LETSTEST_SCRIPT'] = config.test_script
+    if config.pip_extra_index_url:
+        env['LETSTEST_PIP_EXTRA_INDEX_URL'] = config.pip_extra_index_url
+
+    certbot_path = git_clone_to_temp_dir(config.repo, config.branch)
+    env['CERTBOT_HOST_REPO_PATH'] = certbot_path
+    env['CERTBOT_REPO_PATH'] = '/opt/certbot'
+
+    env['LETSTEST_HOST_TESTDATA_PATH'] = TESTDATA_PATH
+    env['LETSTEST_TESTDATA_PATH'] = '/opt/letstest'
+
+    return env
+
+
+def verify_script_path(test_script):
+    """Validates the selected test script.
+
+    :param str test_script: basename of test script to run
+
+    :raises Error: if test_script is invalid
+
+    """
+    verify_exe(pkg_resources.resource_filename(
+        __name__, os.path.join('testdata', 'scripts', test_script)))
+
+
+def verify_exe(path):
+    """Asserts that path refers to an executable.
+
+    :param str path: path to test
+
+    :raises Error: if path doesn't refer to an executable
+
+    """
+    if not os.path.isfile(path):
+        raise Error("{0} doesn't exist!".format(path))
+    if not os.access(path, os.X_OK):
+        raise Error("{0} isn't executable!".format(path))
+
+
+def parse_args(args=None):
+    """Parse command line arguments.
+
+    If args is not provided, it is taken from argv.
+
+    :param list args: command line arguments to parse
+
+    :returns: parsed command line arguments
+    :rtype: argparse.Namespace
+
+    """
+    if args is None:
+        args = sys.argv[1:]
+    return build_parser().parse_args(args)
+
+
+def build_parser():
+    """Create and prepare an argparse parser.
+
+    :returns: argparse parser ready to parse command line arguments
+    :rtype: argparse.ArgumentParser
+
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--branch', default='master',
+                        help='Certbot git branch to use.')
+    parser.add_argument('--pip-extra-index-url',
+                        help='An extra URL for pip to pull packages from.')
+    parser.add_argument('--repo', default='https://github.com/certbot/certbot',
+                        help='Certbot git repository to use.')
+    # WARNING logging level is used by default
+    parser.add_argument('-v', '--verbose', action='count',
+                        default=logging.WARNING / -10,
+                        help='Increase verbosity of output.')
+    parser.add_argument('test_script', type=os.path.basename,
+                        help='Script to run in tests.')
+    return parser
+
+
+def verify_environment():
+    """Asserts the environment will allow this script to work.
+
+    This function tests that all command line utilities are available,
+    the user's version of docker-compose is new enough, and the user has
+    permission to use docker-compose.
+
+    :raises Error: if there's a problem with the environment
+
+    """
+    for command in ('docker', 'docker-compose', 'git',):
+        verify_exe_exists(command)
+    verify_docker_compose_version()
+    verify_permissions()
+
+
+def verify_exe_exists(command):
+    """Asserts command exists in the user's path
+
+    If the command cannot be found, a helpful exception is raised.
+
+    :param str command: command to find in the user's path
+
+    :raises Error: if command cannot be found
+
+    """
+    try:
+        which(command)
+    except subprocess.CalledProcessError:
+        logger.debug('Encountered from which(%s):', command, exc_info=True)
+        raise Error('command {0} could not be found but '
+                    'is required to run this script.'.format(command))
+
+
+def verify_docker_compose_version():
+    """Asserts the user's docker-compose version is new enough.
+
+    :raises Error: if it's not new enough
+
+    """
+    version_output = check_output('docker-compose --version'.split())
+    version_string = version_output.split()[2]
+    if version_string.endswith(','):
+        version_string = version_string[:-1]
+    version = [int(part) for part in version_string.split('.')]
+    logger.debug('docker-compose version is %s', version)
+    if version < [1, 10, 0]:
+        raise Error('docker-compose >= 1.10.0 is required to use this script')
+
+
+def verify_permissions():
+    """Verify we're root or part of the docker group.
+
+    :raises Error: if we have insufficient permissions
+
+    """
+    if os.geteuid() == 0:
+        return
+    try:
+        docker_group = grp.getgrnam('docker').gr_gid
+    except KeyError:
+        pass
+    else:
+        if docker_group in os.getgroups():
+            return
+    raise Error("You must run this script as root "
+                "or be a member of the 'docker' group")
+            
+        
+def which(command):
+    """Returns the absolute path to the command command.
+
+    :param str command: command to find in the user's path
+
+    :returns: absolute path to command
+    :rtype: str
+
+    :raises subprocess.CalledProcessError: if the command isn't found
+
+    """
+    return check_output('command -v {0}'.format(command), shell=True)
+
+
+def set_up_boulder():
+    """Prepares boulder files in a temporary directory.
+
+    :returns: Path to Boulder's docker-compose.yml
+    :rtype: str
+
+    """
+    temp_dir = git_clone_to_temp_dir('https://github.com/letsencrypt/boulder',
+                                     'master', ['--depth', '1'])
+    return boulder_surgery(temp_dir)
+
+
+def boulder_surgery(boulder_path):
+    """Edits Boulder files in preparation for running tests.
+
+    This function causes services in Boulder's docker-compose file to
+    use the default network_mode and configures Boulder to use Docker's
+    embedded DNS server rather than always resolving domains to a
+    specified IP.
+
+    :param str boulder_path: path to the local boulder repo
+
+    :returns: path to the modified Docker Compose file
+    :rtype: str
+
+    """
+    change_dns_resolvers(boulder_path)
+    compose_path = os.path.join(boulder_path, 'docker-compose.yml')
+    remove_network_mode(compose_path)
+    return compose_path
+
+
+def change_dns_resolvers(boulder_path):
+    """Edits Boulder's configuration to use Docker's DNS resolver.
+
+    :param str boulder_path: path to the local boulder repo
+
+    """
+    config_dir = os.path.join(boulder_path, 'test', 'config')
+    assert os.path.isdir(config_dir), 'Missing Boulder config dir!'
+    change_dns_resolver(os.path.join(config_dir, 'ra.json'))
+    change_dns_resolver(os.path.join(config_dir, 'va.json'))
+
+
+def change_dns_resolver(config_path):
+    """Edits a Boulder configuration file to use Docker's DNS resolver.
+
+    :param str config_path: path to the boulder config file to edit
+
+    """
+    with open(config_path) as f:
+        data = json.load(f)
+
+    data['common']['dnsResolver'] = '127.0.0.11:53'
+    dumped_data = json.dumps(data)
+    dumped_data += '\n'
+
+    with open(config_path, 'w') as f:
+        f.write(dumped_data)
+    logger.debug('Updated %s to:\n%s', config_path, dumped_data)
+
+
+def remove_network_mode(compose_path):
+    """Use the default network_mode in a docker-compose file.
+
+    Any lines changing the network_mode will be removed from the file.
+
+    :param str compose_path: path to a Docker Compose file
+
+    """
+    with open(compose_path) as f:
+        original = f.read()
+    with open(compose_path, 'w') as f:
+        f.writelines(line + '\n' for line in original.splitlines()
+                     if not line.lstrip().startswith('network_mode'))
+
+
+def get_test_services_and_containers(boulder_compose_path, env):
+    """Provides a mapping from test service to Docker container.
+
+    :param str boulder_compose_path: path to Boulder's Compose file
+    :param env dict: environment variables for docker-compose
+
+    :returns: Mapping from test service names to their containers
+    :rtype: dict
+
+    """
+    return collections.OrderedDict(
+        (name, get_container(boulder_compose_path, name, env))
+        for name in get_test_services(boulder_compose_path, env))
+
+
+def get_container(boulder_compose_path, service_name, env):
+    """Returns the container for the specified service.
+
+    :param str boulder_compose_path: path to Boulder's Compose file
+    :param str service_name: Compose service to find the container of
+    :param env dict: environment variables for docker-compose
+
+    :returns: Docker container running the specified service
+    :rtype: str
+
+    """
+    return check_output(
+        ['docker-compose', '-f', boulder_compose_path,
+        '-f', COMPOSE_PATH, 'ps', '-q', service_name], env=env).strip()
+
+
+def get_test_services(boulder_compose_path, env):
+    """Returns the names of Docker Compose services for testing.
+
+    Service names are returned in sorted order.
+
+    :param str boulder_compose_path: path to Boulder's docker-compose.yml
+    :param env dict: environment variables for docker-compose
+    
+    :returns: Docker compose services used for testing
+    :rtype: list of str
+
+    """
+    all_services = docker_compose_services(
+        env, boulder_compose_path, COMPOSE_PATH)
+    boulder_services = docker_compose_services(env, boulder_compose_path)
+    return [service for service in sorted(all_services)
+            if service not in boulder_services]
+
+
+def docker_compose_services(env, *compose_files):
+    """Determines the list of services from a Docker Compose setup.
+
+    Files are passed to docker-compose in the same order they are given
+    compose_files.
+
+    :param env dict: environment variables for docker-compose
+    :param list compose_files: paths to Docker compose files
+
+    :returns: list of service names
+    :rtype: list of str
+
+    """
+    assert compose_files, 'At least one compose file is required!'
+
+    cmd = ['docker-compose']
+    for f in compose_files:
+        cmd.append('-f')
+        cmd.append(f)
+    cmd.extend(('config', '--services',))
+
+    services = check_output(cmd, env=env).splitlines()
+    logger.debug('Services found in %s are %s', compose_files, services)
+    return services
+
+
+def git_clone_to_temp_dir(repo, branch, extra_args=None):
+    """Clones the specified repo and branch into a temporary directory.
+
+    This function ensures the temporary directory is deleted when this
+    script exits.
+
+    :param str repo: git repository to clone
+    :param str branch: branch of repo to clone
+    :param list extra_args: additional arguments to git clone
+
+    :returns: temporary directory containing the cloned contents
+    :rtype: str
+
+    """
+    temp_dir = tempfile.mkdtemp()
+    atexit.register(
+        functools.partial(shutil.rmtree, temp_dir, ignore_errors=True))
+
+    args = ['git', 'clone', '--branch', branch]
+    if extra_args:
+        args += extra_args
+    args += [repo, temp_dir]
+    check_call(args)
+    return temp_dir
+
+
+def check_call(args, shell=False, env=None):
+    """subprocess.check_call with logging.
+
+    :param args: command to run with subprocess
+    :type args: list or str
+    :param bool shell: whether the command should be executed in a shell
+    :param dict env: environment variables to use
+
+    :raises subprocess.CalledProcessError: if the command fails
+
+    """
+    check_output(args, shell, env)
+
+
+def check_output(args, shell=False, env=None):
+    """subprocess.check_output with logging.
+
+    :param args: command to run with subprocess
+    :type args: list or str
+    :param bool shell: whether the command should be executed in a shell
+    :param dict env: environment variables to use
+
+    :returns: stdout output
+    :rtype: str
+
+    :raises subprocess.CalledProcessError: if the command fails
+
+    """
+    logger.debug('Calling %s', args)
+    process = subprocess.Popen(args, stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE, shell=shell, env=env,
+                               universal_newlines=True)
+
+    stdout, stderr = process.communicate()
+    if stdout:
+        logger.debug('stdout was:\n%s', stdout)
+    if stderr:
+        logger.debug('stderr was:\n%s', stderr)
+    if process.returncode:
+        raise subprocess.CalledProcessError(process.returncode, args)
+    return stdout
+
+
+if __name__ == '__main__':
+    main()

letstest-certbot/letstest_certbot/testdata/docker-compose.yml

@@ -0,0 +1,156 @@
+version: '2'
+services:
+    centos6.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=centos6.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: centos:6
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    centos7.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=centos7.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: centos:7
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    debian-stretch.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=debian-stretch.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: debian:stretch
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    debian-jessie.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=debian-jessie.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: debian:jessie
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    debian-wheezy.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=debian-wheezy.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: debian:wheezy
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    ubuntu-trusty.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=ubuntu-trusty.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: ubuntu:trusty
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    fedora24.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=fedora24.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: fedora:24
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    fedora25.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=fedora25.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: fedora:25
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    ubuntu-xenial.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=ubuntu-xenial.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: ubuntu:xenial
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    ubuntu-yakkety.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=ubuntu-yakkety.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: ubuntu:yakkety
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro
+    ubuntu-zesty.test:
+        entrypoint: ${LETSTEST_TESTDATA_PATH}/test_runner.sh
+        environment:
+            - CERTBOT_REPO_PATH
+            - LETSTEST_DOMAIN=ubuntu-zesty.test
+            - LETSTEST_PIP_EXTRA_INDEX_URL
+            - LETSTEST_SCRIPT
+            - LETSTEST_TESTDATA_PATH
+        image: ubuntu:zesty
+        links:
+            - boulder
+        volumes:
+            - ${LETSTEST_HOST_TESTDATA_PATH}:${LETSTEST_TESTDATA_PATH}:ro
+            - ${CERTBOT_HOST_REPO_PATH}:${CERTBOT_REPO_PATH}:ro

letstest-certbot/letstest_certbot/testdata/scripts/_install_git

@@ -0,0 +1,15 @@
+#!/bin/bash -xe
+#
+# Installs git if it's not already available
+if ! command -v git ; then
+  if command -v apt-get; then
+    apt-get update && apt-get install git -y
+  elif command -v dnf; then
+    dnf install git -y
+  elif command -v yum; then
+    yum install git -y
+  else
+    echo "Unable to install git!" >&2
+    exit 1
+  fi
+fi

letstest-certbot/letstest_certbot/testdata/scripts/leauto_upgrades

@@ -0,0 +1,21 @@
+#!/bin/bash -xe
+
+"$(dirname $0)/_install_git"
+BRANCH=`git rev-parse --abbrev-ref HEAD`
+# 0.5.0 is the oldest version of letsencrypt-auto that can be used because
+# it's the first version that pins package versions, properly supports
+# --no-self-upgrade, and doesn't use peep which doesn't work with newer
+# versions of pip.
+git checkout -f v0.5.0
+if ! ./letsencrypt-auto -v --debug --version --no-self-upgrade 2>&1 | grep 0.5.0 ; then
+    echo initial installation appeared to fail
+    exit 1
+fi
+
+git checkout -f "$BRANCH"
+EXPECTED_VERSION=$(grep -m1 LE_AUTO_VERSION letsencrypt-auto | cut -d\" -f2)
+if ! ./letsencrypt-auto -v --debug --version --no-self-upgrade 2>&1 | grep $EXPECTED_VERSION ; then
+    echo upgrade appeared to fail
+    exit 1
+fi
+echo upgrade appeared to be successful

letstest-certbot/letstest_certbot/testdata/test_runner.sh

@@ -0,0 +1,7 @@
+#!/bin/sh -xe
+#
+# Sets up and runs an integration test.
+REPO_DEST="~"
+cp -r $CERTBOT_REPO_PATH $REPO_DEST
+cd $REPO_DEST
+exec "$LETSTEST_TESTDATA_PATH/scripts/$LETSTEST_SCRIPT"

letstest-certbot/setup.cfg

@@ -0,0 +1,2 @@
+[bdist_wheel]
+universal = 1

letstest-certbot/setup.py

@@ -0,0 +1,58 @@
+import sys
+
+from setuptools import setup
+from setuptools import find_packages
+
+
+version = '0.1.0'
+
+install_requires = [
+    'setuptools',  # pkg_resources
+]
+
+docs_extras = [
+    'Sphinx>=1.0',  # autodoc_member_order = 'bysource', autodoc_default_flags
+    'sphinx_rtd_theme',
+]
+
+setup(
+    name='letstest-certbot',
+    version=version,
+    description="Tests Certbot against Let's Encrypt's Boulder",
+    url='https://github.com/certbot/certbot',
+    author="Certbot Project",
+    author_email='certbot-dev@eff.org',
+    license='Apache License 2.0',
+    classifiers=[
+        'Development Status :: 3 - Alpha',
+        'Intended Audience :: System Administrators',
+        'License :: OSI Approved :: Apache Software License',
+        'Operating System :: POSIX :: Linux',
+        'Programming Language :: Python',
+        'Programming Language :: Python :: 2',
+        'Programming Language :: Python :: 2.7',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.3',
+        'Programming Language :: Python :: 3.4',
+        'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
+        'Topic :: Internet :: WWW/HTTP',
+        'Topic :: Security',
+        'Topic :: System :: Installation/Setup',
+        'Topic :: System :: Networking',
+        'Topic :: System :: Systems Administration',
+        'Topic :: Utilities',
+    ],
+
+    packages=find_packages(),
+    include_package_data=True,
+    install_requires=install_requires,
+    extras_require={
+        'docs': docs_extras,
+    },
+    entry_points={
+        'console_scripts': [
+            'letstest-certbot = letstest_certbot.__main__:main',
+        ],
+    },
+)