remove extra tests

When I wrote https://github.com/certbot/certbot/pull/7813, I didn't understand the default behavior for pull requests if you don't specify `pr` in the yaml file. According to https://docs.microsoft.com/en-us/azure/devops/pipelines/build/triggers?view=azure-devops&tabs=yaml#pr-triggers:

> If no pr triggers appear in your YAML file, pull request builds are automatically enabled for all branches...

This is not the behavior we want. This PR fixes the problem by disabling builds on PRs.

You should be able to see this working because the advanced tests should not run on this PR but they did run on https://github.com/certbot/certbot/pull/7811.

.azure-pipelines/INSTALL.md

@@ -0,0 +1,119 @@
+# Configuring Azure Pipelines with Certbot
+
+Let's begin. All pipelines are defined in `.azure-pipelines`. Currently there are two:
+* `.azure-pipelines/main.yml` is the main one, executed on PRs for master, and pushes to master,
+* `.azure-pipelines/advanced.yml` add installer testing on top of the main pipeline, and is executed for `test-*` branches, release branches, and nightly run for master.
+
+Several templates are defined in `.azure-pipelines/templates`. These YAML files aggregate common jobs configuration that can be reused in several pipelines.
+
+Unlike Travis, where CodeCov is working without any action required, CodeCov supports Azure Pipelines
+using the coverage-bash utility (not python-coverage for now) only if you provide the Codecov repo token
+using the `CODECOV_TOKEN` environment variable. So `CODECOV_TOKEN` needs to be set as a secured
+environment variable to allow the main pipeline to publish coverage reports to CodeCov.
+
+This INSTALL.md file explains how to configure Azure Pipelines with Certbot in order to execute the CI/CD logic defined in `.azure-pipelines` folder with it.
+During this installation step, warnings describing user access and legal comitments will be displayed like this:
+```
+!!! ACCESS REQUIRED !!!
+```
+
+This document suppose that the Azure DevOps organization is named _certbot_, and the Azure DevOps project is also _certbot_.
+
+## Useful links
+
+* https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema
+* https://www.azuredevopslabs.com/labs/azuredevops/github-integration/
+* https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/python?view=azure-devops
+
+## Prerequisites
+
+### Having a GitHub account
+
+Use your GitHub user for a normal GitHub account, or a user that has administrative rights to the GitHub organization if relevant.
+
+### Having an Azure DevOps account
+- Go to https://dev.azure.com/, click "Start free with GitHub"
+- Login to GitHub
+
+```
+!!! ACCESS REQUIRED !!!
+Personal user data (email + profile info, in read-only)
+```
+
+- Microsoft will create a Live account using the email referenced for the GitHub account. This account is also linked to GitHub account (meaning you can log it using GitHub authentication)
+- Proceed with account registration (birth date, country), add details about name and email contact
+
+```
+!!! ACCESS REQUIRED !!!
+Microsoft proposes to send commercial links to this mail
+Azure DevOps terms of service need to be accepted
+```
+
+_Logged to Azure DevOps, account is ready._
+
+### Installing Azure Pipelines to GitHub
+
+- On GitHub, go to Marketplace
+- Select Azure Pipeline, and "Set up a plan"
+- Select Free, then "Install it for free"
+- Click "Complete order and begin installation"
+
+```
+!!! ACCESS !!!
+Azure Pipeline needs RW on code, RO on metadata, RW on checks, commit statuses, deployments, issues, pull requests.
+RW access here is required to allow update of the pipelines YAML files from Azure DevOps interface, and to
+update the status of builds and PRs on GitHub side when Azure Pipelines are triggered.
+Note however that no admin access is defined here: this means that Azure Pipelines cannot do anything with
+protected branches, like master, and cannot modify the security context around this on GitHub.
+Access can be defined for all or only selected repositories, which is nice.
+```
+
+- Redirected to Azure DevOps, select the account created in _Having an Azure DevOps account_ section.
+- Select the organization, and click "Create a new project" (let's name it the same than the targeted github repo)
+- The Visibility is public, to profit from 10 parallel jobs
+
+```
+!!! ACCESS !!!
+Azure Pipelines needs access to the GitHub account (in term of being able to check it is valid), and the Resources shared between the GitHub account and Azure Pipelines.
+```
+
+_Done. We can move to pipelines configuration._
+
+## Import an existing pipelines from `.azure-pipelines` folder
+
+- On Azure DevOps, go to your organization (eg. _certbot_) then your project (eg. _certbot_)
+- Click "Pipelines" tab
+- Click "New pipeline"
+- Where is your code?: select "__Use the classic editor__"
+
+__Warning: Do not choose the GitHub option in Where is your code? section. Indeed, this option will trigger an OAuth
+grant permissions from Azure Pipelines to GitHub in order to setup a GitHub OAuth Application. The permissions asked
+then are way too large (admin level on almost everything), while the classic approach does not add any more
+permissions, and works perfectly well.__
+
+- Select GitHub in "Select your repository section", choose certbot/certbot in Repository, master in default branch.
+- Click on YAML option for "Select a template"
+- Choose a name for the pipeline (eg. test-pipeline), and browse to the actual pipeline YAML definition in the
+  "YAML file path" input (eg. `.azure-pipelines/test-pipeline.yml`)
+- Click "Save & queue", choose the master branch to build the first pipeline, and click "Save and run" button.
+
+_Done. Pipeline is operational. Repeat to add more pipelines from existing YAML files in `.azure-pipelines`._
+
+## Add a secret variable to a pipeline (like `CODECOV_TOKEN`)
+
+__NB: Following steps suppose that you already setup the YAML pipeline file to
+consume the secret variable that these steps will create as an environment variable.
+For a variable named `CODECOV_TOKEN` consuming the variable `codecov_token`,
+in the YAML file this setup would take the form of the following:
+```
+steps:
+    - script: ./do_something_that_consumes_CODECOV_TOKEN  # Eg. `codecov -F windows`
+      env:
+        CODECOV_TOKEN: $(codecov_token)
+```
+
+To set up a variable that is shared between pipelines, follow the instructions
+at
+https://docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups.
+When adding variables to a group, don't forget to tick "Keep this value secret"
+if it shouldn't be shared publcily.

.azure-pipelines/advanced-test.yml

@@ -0,0 +1,13 @@
+# Advanced pipeline for running our full test suite on demand.
+trigger:
+  # When changing these triggers, please ensure the documentation under
+  # "Running tests in CI" is still correct.
+  - azure-test-*
+  - test-*
+pr: none
+
+jobs:
+  # Any addition here should be reflected in the advanced and release pipelines.
+  # It is advised to declare all jobs here as templates to improve maintainability.
+  - template: templates/tests-suite.yml
+  - template: templates/installer-tests.yml

.azure-pipelines/advanced.yml

@@ -0,0 +1,18 @@
+# Advanced pipeline for running our full test suite on protected branches.
+trigger:
+  - '*.x'
+pr: none
+# This pipeline is also nightly run on master
+schedules:
+  - cron: "0 4 * * *"
+    displayName: Nightly build
+    branches:
+      include:
+      - master
+    always: true
+
+jobs:
+  # Any addition here should be reflected in the advanced-test and release pipelines.
+  # It is advised to declare all jobs here as templates to improve maintainability.
+  - template: templates/tests-suite.yml
+  - template: templates/installer-tests.yml

.azure-pipelines/main.yml

@@ -0,0 +1,12 @@
+trigger:
+  # apache-parser-v2 is a temporary branch for doing work related to
+  # rewriting the parser in the Apache plugin.
+  - apache-parser-v2
+  - master
+pr:
+  - apache-parser-v2
+  - master
+  - '*.x'
+
+jobs:
+  - template: templates/tests-suite.yml

.azure-pipelines/release.yml

@@ -0,0 +1,13 @@
+# Release pipeline to build and deploy Certbot for Windows for GitHub release tags
+trigger:
+  tags:
+    include:
+      - v*
+pr: none
+
+jobs:
+  # Any addition here should be reflected in the advanced and advanced-test pipelines.
+  # It is advised to declare all jobs here as templates to improve maintainability.
+  - template: templates/tests-suite.yml
+  - template: templates/installer-tests.yml
+  - template: templates/changelog.yml

.azure-pipelines/templates/changelog.yml

@@ -0,0 +1,14 @@
+jobs:
+  - job: changelog
+    pool:
+      vmImage: vs2017-win2016
+    steps:
+      - bash: |
+          CERTBOT_VERSION="$(cd certbot && python -c "import certbot; print(certbot.__version__)" && cd ~-)"
+          "${BUILD_REPOSITORY_LOCALPATH}\tools\extract_changelog.py" "${CERTBOT_VERSION}" >> "${BUILD_ARTIFACTSTAGINGDIRECTORY}/release_notes.md"
+        displayName: Prepare changelog
+      - task: PublishPipelineArtifact@1
+        inputs:
+          path: $(Build.ArtifactStagingDirectory)
+          artifact: changelog
+        displayName: Publish changelog

.azure-pipelines/templates/installer-tests.yml

@@ -0,0 +1,56 @@
+jobs:
+  - job: installer_build
+    pool:
+      vmImage: vs2017-win2016
+    steps:
+      - task: UsePythonVersion@0
+        inputs:
+          versionSpec: 3.7
+          architecture: x86
+          addToPath: true
+      - script: python windows-installer/construct.py
+        displayName: Build Certbot installer
+      - task: CopyFiles@2
+        inputs:
+          sourceFolder: $(System.DefaultWorkingDirectory)/windows-installer/build/nsis
+          contents: '*.exe'
+          targetFolder: $(Build.ArtifactStagingDirectory)
+      - task: PublishPipelineArtifact@1
+        inputs:
+          path: $(Build.ArtifactStagingDirectory)
+          artifact: windows-installer
+        displayName: Publish Windows installer
+  - job: installer_run
+    dependsOn: installer_build
+    strategy:
+      matrix:
+        win2019:
+          imageName: windows-2019
+        win2016:
+          imageName: vs2017-win2016
+        win2012r2:
+          imageName: vs2015-win2012r2
+    pool:
+      vmImage: $(imageName)
+    steps:
+      - powershell: Invoke-WebRequest https://www.python.org/ftp/python/3.8.1/python-3.8.1-amd64-webinstall.exe -OutFile C:\py3-setup.exe
+        displayName: Get Python
+      - script: C:\py3-setup.exe /quiet PrependPath=1 InstallAllUsers=1 Include_launcher=1 InstallLauncherAllUsers=1 Include_test=0 Include_doc=0 Include_dev=1 Include_debug=0 Include_tcltk=0 TargetDir=C:\py3
+        displayName: Install Python
+      - task: DownloadPipelineArtifact@2
+        inputs:
+          artifact: windows-installer
+          path: $(Build.SourcesDirectory)/bin
+        displayName: Retrieve Windows installer
+      - script: |
+          py -3 -m venv venv
+          venv\Scripts\python tools\pip_install.py -e certbot-ci
+        displayName: Prepare Certbot-CI
+      - script: |
+          set PATH=%ProgramFiles(x86)%\Certbot\bin;%PATH%
+          venv\Scripts\python -m pytest certbot-ci\windows_installer_integration_tests --allow-persistent-changes --installer-path $(Build.SourcesDirectory)\bin\certbot-beta-installer-win32.exe
+        displayName: Run windows installer integration tests
+      - script: |
+          set PATH=%ProgramFiles(x86)%\Certbot\bin;%PATH%
+          venv\Scripts\python -m pytest certbot-ci\certbot_integration_tests\certbot_tests -n 4
+        displayName: Run certbot integration tests

.azure-pipelines/templates/tests-suite.yml

@@ -0,0 +1,39 @@
+jobs:
+  - job: test
+    strategy:
+      matrix:
+        macos-py27:
+          IMAGE_NAME: macOS-10.14
+          PYTHON_VERSION: 2.7
+          TOXENV: py27
+        macos-py38:
+          IMAGE_NAME: macOS-10.14
+          PYTHON_VERSION: 3.8
+          TOXENV: py38
+        windows-py35:
+          IMAGE_NAME: vs2017-win2016
+          PYTHON_VERSION: 3.5
+          TOXENV: py35
+        windows-py37-cover:
+          IMAGE_NAME: vs2017-win2016
+          PYTHON_VERSION: 3.7
+          TOXENV: py37-cover
+        windows-integration-certbot:
+          IMAGE_NAME: vs2017-win2016
+          PYTHON_VERSION: 3.7
+          TOXENV: integration-certbot
+          PYTEST_ADDOPTS: --numprocesses 4
+    pool:
+      vmImage: $(IMAGE_NAME)
+    steps:
+    - bash: brew install augeas
+      condition: startswith(variables['IMAGE_NAME'], 'macOS')
+      displayName: Install Augeas
+    - task: UsePythonVersion@0
+      inputs:
+        versionSpec: $(PYTHON_VERSION)
+        addToPath: true
+    - script: python tools/pip_install.py -U tox coverage
+      displayName: Install dependencies
+    - script: python -m tox
+      displayName: Run tox

.codecov.yml

@@ -1,18 +0,0 @@
-coverage:
-  status:
-    project:
-      default: off
-      linux:
-        flags: linux
-        # Fixed target instead of auto set by #7173, can
-        # be removed when flags in Codecov are added back.
-        target: 97.6
-        threshold: 0.1
-        base: auto
-      windows:
-        flags: windows
-        # Fixed target instead of auto set by #7173, can
-        # be removed when flags in Codecov are added back.
-        target: 97.0
-        threshold: 0.1
-        base: auto

.coveragerc

@@ -1,2 +1,6 @@
+[run]
+omit = */setup.py
+
 [report]
 omit = */setup.py
+precision = 2

.gitignore

@@ -26,6 +26,7 @@ tags
 \#*#
 .idea
 .ropeproject
+.vscode
 
 # auth --cert-path --chain-path
 /*.pem

.isort.cfg

@@ -0,0 +1,7 @@
+[settings]
+skip_glob=venv*
+skip=letsencrypt-auto-source
+force_sort_within_sections=True
+force_single_line=True
+order_by_type=False
+line_length=400

.pylintrc

@@ -24,6 +24,11 @@ persistent=yes
 # usually to register additional checkers.
 load-plugins=linter_plugin
 
+# A comma-separated list of package or module names from where C extensions may
+# be loaded. Extensions are loading into the active Python interpreter and may
+# run arbitrary code.
+extension-pkg-whitelist=pywintypes,win32api,win32file,win32security
+
 
 [MESSAGES CONTROL]
 
@@ -41,10 +46,14 @@ load-plugins=linter_plugin
 # --enable=similarities". If you want to run only the classes checker, but have
 # no Warning level messages displayed, use"--disable=all --enable=classes
 # --disable=W"
-disable=fixme,locally-disabled,locally-enabled,abstract-class-not-used,abstract-class-little-used,bad-continuation,too-few-public-methods,no-self-use,invalid-name,too-many-instance-attributes,cyclic-import,duplicate-code
-# abstract-class-not-used cannot be disabled locally (at least in
-# pylint 1.4.1), same for abstract-class-little-used
-
+# CERTBOT COMMENT
+# 1) Once certbot codebase is claimed to be compatible exclusively with Python 3,
+#    the useless-object-inheritance check can be enabled again, and code fixed accordingly.
+# 2) Check unsubscriptable-object tends to create a lot of false positives. Let's disable it.
+#    See https://github.com/PyCQA/pylint/issues/1498.
+# 3) Same as point 2 for no-value-for-parameter.
+#    See https://github.com/PyCQA/pylint/issues/2820.
+disable=fixme,locally-disabled,locally-enabled,bad-continuation,no-self-use,invalid-name,cyclic-import,duplicate-code,design,import-outside-toplevel,useless-object-inheritance,unsubscriptable-object,no-value-for-parameter,no-else-return,no-else-raise,no-else-break,no-else-continue
 
 [REPORTS]
 
@@ -297,40 +306,6 @@ valid-classmethod-first-arg=cls
 valid-metaclass-classmethod-first-arg=mcs
 
 
-[DESIGN]
-
-# Maximum number of arguments for function / method
-max-args=6
-
-# Argument names that match this expression will be ignored. Default to name
-# with leading underscore
-ignored-argument-names=(unused)?_.*|dummy
-
-# Maximum number of locals for function / method body
-max-locals=15
-
-# Maximum number of return / yield for function / method body
-max-returns=6
-
-# Maximum number of branch for function / method body
-max-branches=12
-
-# Maximum number of statements in function / method body
-max-statements=50
-
-# Maximum number of parents for a class (see R0901).
-max-parents=12
-
-# Maximum number of attributes for a class (see R0902).
-max-attributes=7
-
-# Minimum number of public methods for a class (see R0903).
-min-public-methods=2
-
-# Maximum number of public methods for a class (see R0904).
-max-public-methods=20
-
-
 [EXCEPTIONS]
 
 # Exceptions that will emit a warning when being caught. Defaults to

.travis.yml

@@ -1,27 +1,31 @@
 language: python
+dist: xenial
 
 cache:
     directories:
         - $HOME/.cache/pip
 
 before_script:
-  - 'if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then ulimit -n 1024 ; fi'
   # On Travis, the fastest parallelization for integration tests has proved to be 4.
   - 'if [[ "$TOXENV" == *"integration"* ]]; then export PYTEST_ADDOPTS="--numprocesses 4"; fi'
+  # Use Travis retry feature for farm tests since they are flaky
+  - 'if [[ "$TOXENV" == "travis-test-farm"* ]]; then export TRAVIS_RETRY=travis_retry; fi'
   - export TOX_TESTENV_PASSENV=TRAVIS
 
 # Only build pushes to the master branch, PRs, and branches beginning with
-# `test-` or of the form `digit(s).digit(s).x`. This reduces the number of
-# simultaneous Travis runs, which speeds turnaround time on review since there
-# is a cap of on the number of simultaneous runs.
+# `test-`, `travis-test-`, or of the form `digit(s).digit(s).x`. This reduces
+# the number of simultaneous Travis runs, which speeds turnaround time on
+# review since there is a cap of on the number of simultaneous runs.
 branches:
+  # When changing these branches, please ensure the documentation under
+  # "Running tests in CI" is still correct.
   only:
     # apache-parser-v2 is a temporary branch for doing work related to
     # rewriting the parser in the Apache plugin.
     - apache-parser-v2
     - master
     - /^\d+\.\d+\.x$/
-    - /^test-.*$/
+    - /^(travis-)?test-.*$/
 
 # Jobs for the main test suite are always executed (including on PRs) except for pushes on master.
 not-on-master: &not-on-master
@@ -37,91 +41,12 @@ matrix:
     # Main test suite
     - python: "2.7"
       env: ACME_SERVER=pebble TOXENV=integration
-      sudo: required
-      services: docker
       <<: *not-on-master
 
     # This job is always executed, including on master
     - python: "2.7"
       env: TOXENV=py27-cover FYI="py27 tests + code coverage"
 
-    - python: "2.7"
-      env: TOXENV=lint
-      <<: *not-on-master
-    - python: "3.4"
-      env: TOXENV=mypy
-      <<: *not-on-master
-    - python: "3.5"
-      env: TOXENV=mypy
-      <<: *not-on-master
-    - python: "2.7"
-      env: TOXENV='py27-{acme,apache,certbot,dns,nginx}-oldest'
-      sudo: required
-      services: docker
-      <<: *not-on-master
-    - python: "3.4"
-      env: TOXENV=py34
-      sudo: required
-      services: docker
-      <<: *not-on-master
-    - python: "3.7"
-      dist: xenial
-      env: TOXENV=py37
-      sudo: required
-      services: docker
-      <<: *not-on-master
-    - sudo: required
-      env: TOXENV=apache_compat
-      services: docker
-      before_install:
-      addons:
-      <<: *not-on-master
-    - sudo: required
-      env: TOXENV=le_auto_xenial
-      services: docker
-      <<: *not-on-master
-    - python: "2.7"
-      env: TOXENV=apacheconftest-with-pebble
-      sudo: required
-      services: docker
-      <<: *not-on-master
-    - python: "2.7"
-      env: TOXENV=nginxroundtrip
-      <<: *not-on-master
-
-    # Extended test suite on cron jobs and pushes to tested branches other than master
-    - sudo: required
-      env: TOXENV=nginx_compat
-      services: docker
-      before_install:
-      addons:
-      <<: *extended-test-suite
-    - python: "2.7"
-      env:
-        - TOXENV=travis-test-farm-apache2
-        - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw="
-      <<: *extended-test-suite
-    - python: "2.7"
-      env:
-        - TOXENV=travis-test-farm-leauto-upgrades
-        - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw="
-      git:
-        depth: false  # This is needed to have the history to checkout old versions of certbot-auto.
-      <<: *extended-test-suite
-    - python: "2.7"
-      env:
-        - TOXENV=travis-test-farm-certonly-standalone
-        - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw="
-      <<: *extended-test-suite
-    - python: "2.7"
-      env:
-        - TOXENV=travis-test-farm-sdists
-        - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw="
-      <<: *extended-test-suite
-    - python: "3.7"
-      dist: xenial
-      env: TOXENV=py37 CERTBOT_NO_PIN=1
-      <<: *extended-test-suite
     - python: "2.7"
       env: ACME_SERVER=boulder-v1 TOXENV=integration
       sudo: required
@@ -132,55 +57,6 @@ matrix:
       sudo: required
       services: docker
       <<: *extended-test-suite
-    - python: "2.7"
-      env: TOXENV=py27-certbot-oldest
-      <<: *extended-test-suite
-    - python: "2.7"
-      env: TOXENV=py27-nginx-oldest
-      <<: *extended-test-suite
-    - python: "2.7"
-      env: ACME_SERVER=boulder-v1 TOXENV=integration-certbot-oldest
-      sudo: required
-      services: docker
-      <<: *extended-test-suite
-    - python: "2.7"
-      env: ACME_SERVER=boulder-v2 TOXENV=integration-certbot-oldest
-      sudo: required
-      services: docker
-      <<: *extended-test-suite
-    - python: "2.7"
-      env: ACME_SERVER=boulder-v1 TOXENV=integration-nginx-oldest
-      sudo: required
-      services: docker
-      <<: *extended-test-suite
-    - python: "2.7"
-      env: ACME_SERVER=boulder-v2 TOXENV=integration-nginx-oldest
-      sudo: required
-      services: docker
-      <<: *extended-test-suite
-    - python: "3.4"
-      env: TOXENV=py34
-      <<: *extended-test-suite
-    - python: "3.5"
-      env: TOXENV=py35
-      <<: *extended-test-suite
-    - python: "3.6"
-      env: TOXENV=py36
-      <<: *extended-test-suite
-    - python: "3.7"
-      dist: xenial
-      env: TOXENV=py37
-      <<: *extended-test-suite
-    - python: "3.4"
-      env: ACME_SERVER=boulder-v1 TOXENV=integration
-      sudo: required
-      services: docker
-      <<: *extended-test-suite
-    - python: "3.4"
-      env: ACME_SERVER=boulder-v2 TOXENV=integration
-      sudo: required
-      services: docker
-      <<: *extended-test-suite
     - python: "3.5"
       env: ACME_SERVER=boulder-v1 TOXENV=integration
       sudo: required
@@ -202,56 +78,20 @@ matrix:
       services: docker
       <<: *extended-test-suite
     - python: "3.7"
-      dist: xenial
       env: ACME_SERVER=boulder-v1 TOXENV=integration
       sudo: required
       services: docker
       <<: *extended-test-suite
     - python: "3.7"
-      dist: xenial
       env: ACME_SERVER=boulder-v2 TOXENV=integration
       sudo: required
       services: docker
       <<: *extended-test-suite
-    - sudo: required
-      env: TOXENV=le_auto_jessie
-      services: docker
+    - python: "3.8"
+      env: ACME_SERVER=boulder-v1 TOXENV=integration
       <<: *extended-test-suite
-    - sudo: required
-      env: TOXENV=le_auto_centos6
-      services: docker
-      <<: *extended-test-suite
-    - sudo: required
-      env: TOXENV=docker_dev
-      services: docker
-      addons:
-        apt:
-          packages:  # don't install nginx and apache
-            - libaugeas0
-      <<: *extended-test-suite
-    - language: generic
-      env: TOXENV=py27
-      os: osx
-      # Using this osx_image is a workaround for
-      # https://travis-ci.community/t/xcode-8-3-homebrew-outdated-error/3798.
-      osx_image: xcode10.2
-      addons:
-        homebrew:
-          packages:
-            - augeas
-            - python2
-      <<: *extended-test-suite
-    - language: generic
-      env: TOXENV=py3
-      os: osx
-      # Using this osx_image is a workaround for
-      # https://travis-ci.community/t/xcode-8-3-homebrew-outdated-error/3798.
-      osx_image: xcode10.2
-      addons:
-        homebrew:
-          packages:
-            - augeas
-            - python3
+    - python: "3.8"
+      env: ACME_SERVER=boulder-v2 TOXENV=integration
       <<: *extended-test-suite
 
 # container-based infrastructure
@@ -274,15 +114,20 @@ addons:
 # except in tests where the environment variable CERTBOT_NO_PIN is set.
 # virtualenv is listed here explicitly to make sure it is upgraded when
 # CERTBOT_NO_PIN is set to work around failures we've seen when using an older
-# version of virtualenv.
-install: "tools/pip_install.py -U codecov tox virtualenv"
-script: tox
-
-after_success: '[ "$TOXENV" == "py27-cover" ] && codecov -F linux'
+# version of virtualenv. The option "-I" is set so when CERTBOT_NO_PIN is also
+# set, pip updates dependencies it thinks are already satisfied to avoid some
+# problems with its lack of real dependency resolution.
+install: 'tools/pip_install.py -I tox virtualenv'
+# Most of the time TRAVIS_RETRY is an empty string, and has no effect on the
+# script command. It is set only to `travis_retry` during farm tests, in
+# order to trigger the Travis retry feature, and compensate the inherent
+# flakiness of these specific tests.
+script: '$TRAVIS_RETRY tox'
 
 notifications:
   email: false
   irc:
+    if: NOT branch =~ ^(travis-)?test-.*$
     channels:
       # This is set to a secure variable to prevent forks from sending
       # notifications. This value was created by installing

AUTHORS.md

@@ -15,8 +15,10 @@ Authors
 * [Alex Gaynor](https://github.com/alex)
 * [Alex Halderman](https://github.com/jhalderm)
 * [Alex Jordan](https://github.com/strugee)
+* [Alex Zorin](https://github.com/alexzorin)
 * [Amjad Mashaal](https://github.com/TheNavigat)
 * [Andrew Murray](https://github.com/radarhere)
+* [Andrzej Górski](https://github.com/andrzej3393)
 * [Anselm Levskaya](https://github.com/levskaya)
 * [Antoine Jacoutot](https://github.com/ajacoutot)
 * [asaph](https://github.com/asaph)
@@ -34,6 +36,7 @@ Authors
 * [Brad Warren](https://github.com/bmw)
 * [Brandon Kraft](https://github.com/kraftbj)
 * [Brandon Kreisel](https://github.com/kraftbj)
+* [Cameron Steel](https://github.com/Tugzrida)
 * [Ceesjan Luiten](https://github.com/quinox)
 * [Chad Whitacre](https://github.com/whit537)
 * [Chhatoi Pritam Baral](https://github.com/pritambaral)
@@ -98,6 +101,7 @@ Authors
 * [Harlan Lieberman-Berg](https://github.com/hlieberman)
 * [Henri Salo](https://github.com/fgeek)
 * [Henry Chen](https://github.com/henrychen95)
+* [Hugo van Kemenade](https://github.com/hugovk)
 * [Ingolf Becker](https://github.com/watercrossing)
 * [Jaap Eldering](https://github.com/eldering)
 * [Jacob Hoffman-Andrews](https://github.com/jsha)
@@ -122,10 +126,12 @@ Authors
 * [Jonathan Herlin](https://github.com/Jonher937)
 * [Jon Walsh](https://github.com/code-tree)
 * [Joona Hoikkala](https://github.com/joohoi)
+* [Josh McCullough](https://github.com/JoshMcCullough)
 * [Josh Soref](https://github.com/jsoref)
 * [Joubin Jabbari](https://github.com/joubin)
 * [Juho Juopperi](https://github.com/jkjuopperi)
 * [Kane York](https://github.com/riking)
+* [Kenichi Maehashi](https://github.com/kmaehashi)
 * [Kenneth Skovhede](https://github.com/kenkendk)
 * [Kevin Burke](https://github.com/kevinburke)
 * [Kevin London](https://github.com/kevinlondon)
@@ -161,8 +167,10 @@ Authors
 * [Michael Schumacher](https://github.com/schumaml)
 * [Michael Strache](https://github.com/Jarodiv)
 * [Michael Sverdlin](https://github.com/sveder)
+* [Michael Watters](https://github.com/blackknight36)
 * [Michal Moravec](https://github.com/https://github.com/Majkl578)
 * [Michal Papis](https://github.com/mpapis)
+* [Mickaël Schoentgen](https://github.com/BoboTiG)
 * [Minn Soe](https://github.com/MinnSoe)
 * [Min RK](https://github.com/minrk)
 * [Miquel Ruiz](https://github.com/miquelruiz)
@@ -226,6 +234,7 @@ Authors
 * [Stavros Korokithakis](https://github.com/skorokithakis)
 * [Stefan Weil](https://github.com/stweil)
 * [Steve Desmond](https://github.com/stevedesmond-ca)
+* [sydneyli](https://github.com/sydneyli)
 * [Tan Jay Jun](https://github.com/jayjun)
 * [Tapple Gao](https://github.com/tapple)
 * [Telepenin Nikolay](https://github.com/telepenin)

CHANGELOG.md

@@ -0,0 +1 @@
+certbot/CHANGELOG.md

Dockerfile

@@ -1,35 +0,0 @@
-FROM python:2-alpine3.9
-
-ENTRYPOINT [ "certbot" ]
-EXPOSE 80 443
-VOLUME /etc/letsencrypt /var/lib/letsencrypt
-WORKDIR /opt/certbot
-
-COPY CHANGELOG.md README.rst setup.py src/
-
-# Generate constraints file to pin dependency versions
-COPY letsencrypt-auto-source/pieces/dependency-requirements.txt .
-COPY tools /opt/certbot/tools
-RUN sh -c 'cat dependency-requirements.txt | /opt/certbot/tools/strip_hashes.py > unhashed_requirements.txt'
-RUN sh -c 'cat tools/dev_constraints.txt unhashed_requirements.txt | /opt/certbot/tools/merge_requirements.py > docker_constraints.txt'
-
-COPY acme src/acme
-COPY certbot src/certbot
-
-RUN apk add --no-cache --virtual .certbot-deps \
-        libffi \
-        libssl1.1 \
-        openssl \
-        ca-certificates \
-        binutils
-RUN apk add --no-cache --virtual .build-deps \
-        gcc \
-        linux-headers \
-        openssl-dev \
-        musl-dev \
-        libffi-dev \
-    && pip install -r /opt/certbot/dependency-requirements.txt \
-    && pip install --no-cache-dir --no-deps \
-        --editable /opt/certbot/src/acme \
-        --editable /opt/certbot/src \
-    && apk del .build-deps

Dockerfile-dev

@@ -1,21 +1,20 @@
 # This Dockerfile builds an image for development.
-FROM ubuntu:xenial
+FROM debian:buster
 
 # Note: this only exposes the port to other docker containers.
 EXPOSE 80 443
 
 WORKDIR /opt/certbot/src
 
-# TODO: Install Apache/Nginx for plugin development.
 COPY . .
 RUN apt-get update && \
-    apt-get install apache2 git nginx-light -y && \
-    letsencrypt-auto-source/letsencrypt-auto --os-packages-only && \
+    apt-get install apache2 git python3-dev python3-venv gcc libaugeas0 \
+                    libssl-dev libffi-dev ca-certificates openssl nginx-light -y && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/* \
            /tmp/* \
            /var/tmp/*
 
-RUN VENV_NAME="../venv" python tools/venv.py
+RUN VENV_NAME="../venv3" python3 tools/venv3.py
 
-ENV PATH /opt/certbot/venv/bin:$PATH
+ENV PATH /opt/certbot/venv3/bin:$PATH

README.rst

@@ -1,131 +0,0 @@
-.. This file contains a series of comments that are used to include sections of this README in other files. Do not modify these comments unless you know what you are doing. tag:intro-begin
-
-Certbot is part of EFF’s effort to encrypt the entire Internet. Secure communication over the Web relies on HTTPS, which requires the use of a digital certificate that lets browsers verify the identity of web servers (e.g., is that really google.com?). Web servers obtain their certificates from trusted third parties called certificate authorities (CAs). Certbot is an easy-to-use client that fetches a certificate from Let’s Encrypt—an open certificate authority launched by the EFF, Mozilla, and others—and deploys it to a web server.
-
-Anyone who has gone through the trouble of setting up a secure website knows what a hassle getting and maintaining a certificate is. Certbot and Let’s Encrypt can automate away the pain and let you turn on and manage HTTPS with simple commands. Using Certbot and Let's Encrypt is free, so there’s no need to arrange payment.
-
-How you use Certbot depends on the configuration of your web server. The best way to get started is to use our `interactive guide <https://certbot.eff.org>`_. It generates instructions based on your configuration settings. In most cases, you’ll need `root or administrator access <https://certbot.eff.org/faq/#does-certbot-require-root-administrator-privileges>`_ to your web server to run Certbot.
-
-Certbot is meant to be run directly on your web server, not on your personal computer. If you’re using a hosted service and don’t have direct access to your web server, you might not be able to use Certbot. Check with your hosting provider for documentation about uploading certificates or using certificates issued by Let’s Encrypt.
-
-Certbot is a fully-featured, extensible client for the Let's
-Encrypt CA (or any other CA that speaks the `ACME
-<https://github.com/ietf-wg-acme/acme/blob/master/draft-ietf-acme-acme.md>`_
-protocol) that can automate the tasks of obtaining certificates and
-configuring webservers to use them. This client runs on Unix-based operating
-systems.
-
-To see the changes made to Certbot between versions please refer to our
-`changelog <https://github.com/certbot/certbot/blob/master/CHANGELOG.md>`_.
-
-Until May 2016, Certbot was named simply ``letsencrypt`` or ``letsencrypt-auto``,
-depending on install method. Instructions on the Internet, and some pieces of the
-software, may still refer to this older name.
-
-Contributing
-------------
-
-If you'd like to contribute to this project please read `Developer Guide
-<https://certbot.eff.org/docs/contributing.html>`_.
-
-This project is governed by `EFF's Public Projects Code of Conduct <https://www.eff.org/pages/eppcode>`_.
-
-.. _installation:
-
-How to run the client
----------------------
-
-The easiest way to install and run Certbot is by visiting `certbot.eff.org`_,
-where you can find the correct instructions for many web server and OS
-combinations.  For more information, see `Get Certbot
-<https://certbot.eff.org/docs/install.html>`_.
-
-.. _certbot.eff.org: https://certbot.eff.org/
-
-Understanding the client in more depth
---------------------------------------
-
-To understand what the client is doing in detail, it's important to
-understand the way it uses plugins.  Please see the `explanation of
-plugins <https://certbot.eff.org/docs/using.html#plugins>`_ in
-the User Guide.
-
-Links
-=====
-
-.. Do not modify this comment unless you know what you're doing. tag:links-begin
-
-Documentation: https://certbot.eff.org/docs
-
-Software project: https://github.com/certbot/certbot
-
-Notes for developers: https://certbot.eff.org/docs/contributing.html
-
-Main Website: https://certbot.eff.org
-
-Let's Encrypt Website: https://letsencrypt.org
-
-Community: https://community.letsencrypt.org
-
-ACME spec: http://ietf-wg-acme.github.io/acme/
-
-ACME working area in github: https://github.com/ietf-wg-acme/acme
-
-|build-status| |coverage| |docs| |container|
-
-.. |build-status| image:: https://travis-ci.com/certbot/certbot.svg?branch=master
-   :target: https://travis-ci.com/certbot/certbot
-   :alt: Travis CI status
-
-.. |coverage| image:: https://codecov.io/gh/certbot/certbot/branch/master/graph/badge.svg
-   :target: https://codecov.io/gh/certbot/certbot
-   :alt: Coverage status
-
-.. |docs| image:: https://readthedocs.org/projects/letsencrypt/badge/
-   :target: https://readthedocs.org/projects/letsencrypt/
-   :alt: Documentation status
-
-.. |container| image:: https://quay.io/repository/letsencrypt/letsencrypt/status
-   :target: https://quay.io/repository/letsencrypt/letsencrypt
-   :alt: Docker Repository on Quay.io
-
-.. Do not modify this comment unless you know what you're doing. tag:links-end
-
-System Requirements
-===================
-
-See https://certbot.eff.org/docs/install.html#system-requirements.
-
-.. Do not modify this comment unless you know what you're doing. tag:intro-end
-
-.. Do not modify this comment unless you know what you're doing. tag:features-begin
-
-Current Features
-=====================
-
-* Supports multiple web servers:
-
-  - apache/2.x
-  - nginx/0.8.48+
-  - webroot (adds files to webroot directories in order to prove control of
-    domains and obtain certs)
-  - standalone (runs its own simple webserver to prove you control a domain)
-  - other server software via `third party plugins <https://certbot.eff.org/docs/using.html#third-party-plugins>`_
-
-* The private key is generated locally on your system.
-* Can talk to the Let's Encrypt CA or optionally to other ACME
-  compliant services.
-* Can get domain-validated (DV) certificates.
-* Can revoke certificates.
-* Adjustable RSA key bit-length (2048 (default), 4096, ...).
-* Can optionally install a http -> https redirect, so your site effectively
-  runs https only (Apache only)
-* Fully automated.
-* Configuration changes are logged and can be reverted.
-* Supports an interactive text UI, or can be driven entirely from the
-  command line.
-* Free and Open Source Software, made with Python.
-
-.. Do not modify this comment unless you know what you're doing. tag:features-end
-
-For extensive documentation on using and contributing to Certbot, go to https://certbot.eff.org/docs. If you would like to contribute to the project or run the latest code from git, you should read our `developer guide <https://certbot.eff.org/docs/contributing.html>`_.

README.rst

@@ -0,0 +1 @@
+certbot/README.rst

acme/MANIFEST.in

@@ -3,4 +3,6 @@ include README.rst
 include pytest.ini
 recursive-include docs *
 recursive-include examples *
-recursive-include acme/testdata *
+recursive-include tests *
+global-exclude __pycache__
+global-exclude *.py[cod]

acme/acme/__init__.py

@@ -13,7 +13,6 @@ import warnings
 #
 # It is based on
 # https://github.com/requests/requests/blob/1278ecdf71a312dc2268f3bfc0aabfab3c006dcf/requests/packages.py
-
 import josepy as jose
 
 for mod in list(sys.modules):
@@ -21,30 +20,3 @@ for mod in list(sys.modules):
     # preserved (acme.jose.* is josepy.*)
     if mod == 'josepy' or mod.startswith('josepy.'):
         sys.modules['acme.' + mod.replace('josepy', 'jose', 1)] = sys.modules[mod]
-
-
-# This class takes a similar approach to the cryptography project to deprecate attributes
-# in public modules. See the _ModuleWithDeprecation class here:
-# https://github.com/pyca/cryptography/blob/91105952739442a74582d3e62b3d2111365b0dc7/src/cryptography/utils.py#L129
-class _TLSSNI01DeprecationModule(object):
-    """
-    Internal class delegating to a module, and displaying warnings when
-    attributes related to TLS-SNI-01 are accessed.
-    """
-    def __init__(self, module):
-        self.__dict__['_module'] = module
-
-    def __getattr__(self, attr):
-        if 'TLSSNI01' in attr:
-            warnings.warn('{0} attribute is deprecated, and will be removed soon.'.format(attr),
-                          DeprecationWarning, stacklevel=2)
-        return getattr(self._module, attr)
-
-    def __setattr__(self, attr, value):  # pragma: no cover
-        setattr(self._module, attr, value)
-
-    def __delattr__(self, attr):  # pragma: no cover
-        delattr(self._module, attr)
-
-    def __dir__(self):  # pragma: no cover
-        return ['_module'] + dir(self._module)

acme/acme/challenges.py

@@ -3,19 +3,13 @@ import abc
 import functools
 import hashlib
 import logging
-import socket
-import sys
 
 from cryptography.hazmat.primitives import hashes  # type: ignore
 import josepy as jose
-import OpenSSL
 import requests
 import six
 
-from acme import errors
-from acme import crypto_util
 from acme import fields
-from acme import _TLSSNI01DeprecationModule
 
 logger = logging.getLogger(__name__)
 
@@ -60,8 +54,7 @@ class UnrecognizedChallenge(Challenge):
         object.__setattr__(self, "jobj", jobj)
 
     def to_partial_json(self):
-        # pylint: disable=no-member
-        return self.jobj
+        return self.jobj  # pylint: disable=no-member
 
     @classmethod
     def from_json(cls, jobj):
@@ -119,7 +112,7 @@ class KeyAuthorizationChallengeResponse(ChallengeResponse):
         :rtype: bool
 
         """
-        parts = self.key_authorization.split('.')  # pylint: disable=no-member
+        parts = self.key_authorization.split('.')
         if len(parts) != 2:
             logger.debug("Key authorization (%r) is not well formed",
                          self.key_authorization)
@@ -237,7 +230,7 @@ class DNS01Response(KeyAuthorizationChallengeResponse):
         return verified
 
 
-@Challenge.register  # pylint: disable=too-many-ancestors
+@Challenge.register
 class DNS01(KeyAuthorizationChallenge):
     """ACME dns-01 challenge."""
     response_cls = DNS01Response
@@ -310,7 +303,7 @@ class HTTP01Response(KeyAuthorizationChallengeResponse):
         uri = chall.uri(domain)
         logger.debug("Verifying %s at %s...", chall.typ, uri)
         try:
-            http_response = requests.get(uri)
+            http_response = requests.get(uri, verify=False)
         except requests.exceptions.RequestException as error:
             logger.error("Unable to reach %s: %s", uri, error)
             return False
@@ -327,7 +320,7 @@ class HTTP01Response(KeyAuthorizationChallengeResponse):
         return True
 
 
-@Challenge.register  # pylint: disable=too-many-ancestors
+@Challenge.register
 class HTTP01(KeyAuthorizationChallenge):
     """ACME http-01 challenge."""
     response_cls = HTTP01Response
@@ -367,148 +360,6 @@ class HTTP01(KeyAuthorizationChallenge):
         return self.key_authorization(account_key)
 
 
-@ChallengeResponse.register
-class TLSSNI01Response(KeyAuthorizationChallengeResponse):
-    """ACME tls-sni-01 challenge response."""
-    typ = "tls-sni-01"
-
-    DOMAIN_SUFFIX = b".acme.invalid"
-    """Domain name suffix."""
-
-    PORT = 443
-    """Verification port as defined by the protocol.
-
-    You can override it (e.g. for testing) by passing ``port`` to
-    `simple_verify`.
-
-    """
-
-    @property
-    def z(self):  # pylint: disable=invalid-name
-        """``z`` value used for verification.
-
-        :rtype bytes:
-
-        """
-        return hashlib.sha256(
-            self.key_authorization.encode("utf-8")).hexdigest().lower().encode()
-
-    @property
-    def z_domain(self):
-        """Domain name used for verification, generated from `z`.
-
-        :rtype bytes:
-
-        """
-        return self.z[:32] + b'.' + self.z[32:] + self.DOMAIN_SUFFIX
-
-    def gen_cert(self, key=None, bits=2048):
-        """Generate tls-sni-01 certificate.
-
-        :param OpenSSL.crypto.PKey key: Optional private key used in
-            certificate generation. If not provided (``None``), then
-            fresh key will be generated.
-        :param int bits: Number of bits for newly generated key.
-
-        :rtype: `tuple` of `OpenSSL.crypto.X509` and `OpenSSL.crypto.PKey`
-
-        """
-        if key is None:
-            key = OpenSSL.crypto.PKey()
-            key.generate_key(OpenSSL.crypto.TYPE_RSA, bits)
-        return crypto_util.gen_ss_cert(key, [
-            # z_domain is too big to fit into CN, hence first dummy domain
-            'dummy', self.z_domain.decode()], force_san=True), key
-
-    def probe_cert(self, domain, **kwargs):
-        """Probe tls-sni-01 challenge certificate.
-
-        :param unicode domain:
-
-        """
-        # TODO: domain is not necessary if host is provided
-        if "host" not in kwargs:
-            host = socket.gethostbyname(domain)
-            logger.debug('%s resolved to %s', domain, host)
-            kwargs["host"] = host
-
-        kwargs.setdefault("port", self.PORT)
-        kwargs["name"] = self.z_domain
-        # TODO: try different methods?
-        return crypto_util.probe_sni(**kwargs)
-
-    def verify_cert(self, cert):
-        """Verify tls-sni-01 challenge certificate.
-
-        :param OpensSSL.crypto.X509 cert: Challenge certificate.
-
-        :returns: Whether the certificate was successfully verified.
-        :rtype: bool
-
-        """
-        # pylint: disable=protected-access
-        sans = crypto_util._pyopenssl_cert_or_req_san(cert)
-        logger.debug('Certificate %s. SANs: %s', cert.digest('sha256'), sans)
-        return self.z_domain.decode() in sans
-
-    def simple_verify(self, chall, domain, account_public_key,
-                      cert=None, **kwargs):
-        """Simple verify.
-
-        Verify ``validation`` using ``account_public_key``, optionally
-        probe tls-sni-01 certificate and check using `verify_cert`.
-
-        :param .challenges.TLSSNI01 chall: Corresponding challenge.
-        :param str domain: Domain name being validated.
-        :param JWK account_public_key:
-        :param OpenSSL.crypto.X509 cert: Optional certificate. If not
-            provided (``None``) certificate will be retrieved using
-            `probe_cert`.
-        :param int port: Port used to probe the certificate.
-
-
-        :returns: ``True`` iff client's control of the domain has been
-            verified.
-        :rtype: bool
-
-        """
-        if not self.verify(chall, account_public_key):
-            logger.debug("Verification of key authorization in response failed")
-            return False
-
-        if cert is None:
-            try:
-                cert = self.probe_cert(domain=domain, **kwargs)
-            except errors.Error as error:
-                logger.debug(str(error), exc_info=True)
-                return False
-
-        return self.verify_cert(cert)
-
-
-@Challenge.register  # pylint: disable=too-many-ancestors
-class TLSSNI01(KeyAuthorizationChallenge):
-    """ACME tls-sni-01 challenge."""
-    response_cls = TLSSNI01Response
-    typ = response_cls.typ
-
-    # boulder#962, ietf-wg-acme#22
-    #n = jose.Field("n", encoder=int, decoder=int)
-
-    def validation(self, account_key, **kwargs):
-        """Generate validation.
-
-        :param JWK account_key:
-        :param OpenSSL.crypto.PKey cert_key: Optional private key used
-            in certificate generation. If not provided (``None``), then
-            fresh key will be generated.
-
-        :rtype: `tuple` of `OpenSSL.crypto.X509` and `OpenSSL.crypto.PKey`
-
-        """
-        return self.response(account_key).gen_cert(key=kwargs.get('cert_key'))
-
-
 @ChallengeResponse.register
 class TLSALPN01Response(KeyAuthorizationChallengeResponse):
     """ACME TLS-ALPN-01 challenge response.
@@ -520,7 +371,7 @@ class TLSALPN01Response(KeyAuthorizationChallengeResponse):
     typ = "tls-alpn-01"
 
 
-@Challenge.register  # pylint: disable=too-many-ancestors
+@Challenge.register
 class TLSALPN01(KeyAuthorizationChallenge):
     """ACME tls-alpn-01 challenge.
 
@@ -617,7 +468,3 @@ class DNSResponse(ChallengeResponse):
 
         """
         return chall.check_validation(self.validation, account_public_key)
-
-
-# Patching ourselves to warn about TLS-SNI challenge deprecation and removal.
-sys.modules[__name__] = _TLSSNI01DeprecationModule(sys.modules[__name__])

acme/acme/client.py

@@ -5,25 +5,26 @@ import datetime
 from email.utils import parsedate_tz
 import heapq
 import logging
-import time
 import re
 import sys
+import time
 
-import six
-from six.moves import http_client  # pylint: disable=import-error
 import josepy as jose
 import OpenSSL
 import requests
 from requests.adapters import HTTPAdapter
 from requests_toolbelt.adapters.source import SourceAddressAdapter
+import six
+from six.moves import http_client
 
 from acme import crypto_util
 from acme import errors
 from acme import jws
 from acme import messages
-# pylint: disable=unused-import, no-name-in-module
-from acme.magic_typing import Dict, List, Set, Text
-
+from acme.magic_typing import Dict
+from acme.magic_typing import List
+from acme.magic_typing import Set
+from acme.magic_typing import Text
 
 logger = logging.getLogger(__name__)
 
@@ -33,10 +34,9 @@ logger = logging.getLogger(__name__)
 # https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning
 if sys.version_info < (2, 7, 9):  # pragma: no cover
     try:
-        # pylint: disable=no-member
         requests.packages.urllib3.contrib.pyopenssl.inject_into_urllib3()  # type: ignore
     except AttributeError:
-        import urllib3.contrib.pyopenssl  # pylint: disable=import-error
+        import urllib3.contrib.pyopenssl
         urllib3.contrib.pyopenssl.inject_into_urllib3()
 
 DEFAULT_NETWORK_TIMEOUT = 45
@@ -44,7 +44,7 @@ DEFAULT_NETWORK_TIMEOUT = 45
 DER_CONTENT_TYPE = 'application/pkix-cert'
 
 
-class ClientBase(object):  # pylint: disable=too-many-instance-attributes
+class ClientBase(object):
     """ACME client base object.
 
     :ivar messages.Directory directory:
@@ -123,6 +123,22 @@ class ClientBase(object):  # pylint: disable=too-many-instance-attributes
         """
         return self.update_registration(regr, update={'status': 'deactivated'})
 
+    def deactivate_authorization(self, authzr):
+        # type: (messages.AuthorizationResource) -> messages.AuthorizationResource
+        """Deactivate authorization.
+
+        :param messages.AuthorizationResource authzr: The Authorization resource
+            to be deactivated.
+
+        :returns: The Authorization resource that was deactivated.
+        :rtype: `.AuthorizationResource`
+
+        """
+        body = messages.UpdateAuthorization(status='deactivated')
+        response = self._post(authzr.uri, body)
+        return self._authzr_from_response(response,
+            authzr.body.identifier, authzr.uri)
+
     def _authzr_from_response(self, response, identifier=None, uri=None):
         authzr = messages.AuthorizationResource(
             body=messages.Authorization.from_json(response.json()),
@@ -238,7 +254,6 @@ class Client(ClientBase):
             URI from which the resource will be downloaded.
 
         """
-        # pylint: disable=too-many-arguments
         self.key = key
         if net is None:
             net = ClientNetwork(key, alg=alg, verify_ssl=verify_ssl)
@@ -264,7 +279,6 @@ class Client(ClientBase):
         assert response.status_code == http_client.CREATED
 
         # "Instance of 'Field' has no key/contact member" bug:
-        # pylint: disable=no-member
         return self._regr_from_response(response)
 
     def query_registration(self, regr):
@@ -419,7 +433,6 @@ class Client(ClientBase):
             was marked by the CA as invalid
 
         """
-        # pylint: disable=too-many-locals
         assert max_attempts > 0
         attempts = collections.defaultdict(int) # type: Dict[messages.AuthorizationResource, int]
         exhausted = set()
@@ -450,7 +463,6 @@ class Client(ClientBase):
             updated[authzr] = updated_authzr
 
             attempts[authzr] += 1
-            # pylint: disable=no-member
             if updated_authzr.body.status not in (
                     messages.STATUS_VALID, messages.STATUS_INVALID):
                 if attempts[authzr] < max_attempts:
@@ -591,7 +603,6 @@ class ClientV2(ClientBase):
         if response.status_code == 200 and 'Location' in response.headers:
             raise errors.ConflictError(response.headers.get('Location'))
         # "Instance of 'Field' has no key/contact member" bug:
-        # pylint: disable=no-member
         regr = self._regr_from_response(response)
         self.net.account = regr
         return regr
@@ -655,7 +666,7 @@ class ClientV2(ClientBase):
         response = self._post(self.directory['newOrder'], order)
         body = messages.Order.from_json(response.json())
         authorizations = []
-        for url in body.authorizations:  # pylint: disable=not-an-iterable
+        for url in body.authorizations:
             authorizations.append(self._authzr_from_response(self._post_as_get(url), uri=url))
         return messages.OrderResource(
             body=body,
@@ -715,7 +726,7 @@ class ClientV2(ClientBase):
         for authzr in responses:
             if authzr.body.status != messages.STATUS_VALID:
                 for chall in authzr.body.challenges:
-                    if chall.error != None:
+                    if chall.error is not None:
                         failed.append(authzr)
         if failed:
             raise errors.ValidationError(failed)
@@ -765,29 +776,13 @@ class ClientV2(ClientBase):
 
     def _post_as_get(self, *args, **kwargs):
         """
-        Send GET request using the POST-as-GET protocol if needed.
-        The request will be first issued using POST-as-GET for ACME v2. If the ACME CA servers do
-        not support this yet and return an error, request will be retried using GET.
-        For ACME v1, only GET request will be tried, as POST-as-GET is not supported.
+        Send GET request using the POST-as-GET protocol.
         :param args:
         :param kwargs:
         :return:
         """
-        if self.acme_version >= 2:
-            # We add an empty payload for POST-as-GET requests
-            new_args = args[:1] + (None,) + args[1:]
-            try:
-                return self._post(*new_args, **kwargs)
-            except messages.Error as error:
-                if error.code == 'malformed':
-                    logger.debug('Error during a POST-as-GET request, '
-                                 'your ACME CA server may not support it:\n%s', error)
-                    logger.debug('Retrying request with GET.')
-                else:  # pragma: no cover
-                    raise
-
-        # If POST-as-GET is not supported yet, we use a GET instead.
-        return self.net.get(*args, **kwargs)
+        new_args = args[:1] + (None,) + args[1:]
+        return self._post(*new_args, **kwargs)
 
 
 class BackwardsCompatibleClientV2(object):
@@ -931,7 +926,7 @@ class BackwardsCompatibleClientV2(object):
         return self.client.external_account_required()
 
 
-class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
+class ClientNetwork(object):
     """Wrapper around requests that signs POSTs for authentication.
 
     Also adds user agent, and handles Content-Type.
@@ -947,7 +942,7 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
     :param messages.RegistrationResource account: Account object. Required if you are
             planning to use .post() with acme_version=2 for anything other than
             creating a new account; may be set later after registering.
-    :param josepy.JWASignature alg: Algoritm to use in signing JWS.
+    :param josepy.JWASignature alg: Algorithm to use in signing JWS.
     :param bool verify_ssl: Whether to verify certificates on SSL connections.
     :param str user_agent: String to send as User-Agent header.
     :param float timeout: Timeout for requests.
@@ -957,7 +952,6 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
     def __init__(self, key, account=None, alg=jose.RS256, verify_ssl=True,
                  user_agent='acme-python', timeout=DEFAULT_NETWORK_TIMEOUT,
                  source_address=None):
-        # pylint: disable=too-many-arguments
         self.key = key
         self.account = account
         self.alg = alg
@@ -1028,6 +1022,9 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
 
         """
         response_ct = response.headers.get('Content-Type')
+        # Strip parameters from the media-type (rfc2616#section-3.7)
+        if response_ct:
+            response_ct = response_ct.split(';')[0].strip()
         try:
             # TODO: response.json() is called twice, once here, and
             # once in _get and _post clients
@@ -1065,7 +1062,6 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
         return response
 
     def _send_request(self, method, url, *args, **kwargs):
-        # pylint: disable=too-many-locals
         """Send HTTP request.
 
         Makes sure that `verify_ssl` is respected. Logs request and
@@ -1112,10 +1108,9 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
             err_regex = r".*host='(\S*)'.*Max retries exceeded with url\: (\/\w*).*(\[Errno \d+\])([A-Za-z ]*)"
             m = re.match(err_regex, str(e))
             if m is None:
-                raise # pragma: no cover
-            else:
-                host, path, _err_no, err_msg = m.groups()
-                raise ValueError("Requesting {0}{1}:{2}".format(host, path, err_msg))
+                raise  # pragma: no cover
+            host, path, _err_no, err_msg = m.groups()
+            raise ValueError("Requesting {0}{1}:{2}".format(host, path, err_msg))
 
         # If content is DER, log the base64 of it instead of raw bytes, to keep
         # binary data out of the logs.
@@ -1181,8 +1176,7 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
             if error.code == 'badNonce':
                 logger.debug('Retrying request after error:\n%s', error)
                 return self._post_once(*args, **kwargs)
-            else:
-                raise
+            raise
 
     def _post_once(self, url, obj, content_type=JOSE_CONTENT_TYPE,
             acme_version=1, **kwargs):

acme/acme/crypto_util.py

@@ -6,15 +6,14 @@ import os
 import re
 import socket
 
-from OpenSSL import crypto
-from OpenSSL import SSL # type: ignore # https://github.com/python/typeshed/issues/2052
 import josepy as jose
+from OpenSSL import crypto
+from OpenSSL import SSL  # type: ignore # https://github.com/python/typeshed/issues/2052
 
 from acme import errors
-# pylint: disable=unused-import, no-name-in-module
-from acme.magic_typing import Callable, Union, Tuple, Optional
-# pylint: enable=unused-import, no-name-in-module
-
+from acme.magic_typing import Callable
+from acme.magic_typing import Tuple
+from acme.magic_typing import Union
 
 logger = logging.getLogger(__name__)
 
@@ -28,7 +27,7 @@ logger = logging.getLogger(__name__)
 _DEFAULT_SSL_METHOD = SSL.SSLv23_METHOD  # type: ignore
 
 
-class SSLSocket(object):  # pylint: disable=too-few-public-methods
+class SSLSocket(object):
     """SSL wrapper for sockets.
 
     :ivar socket sock: Original wrapped socket.
@@ -74,7 +73,7 @@ class SSLSocket(object):  # pylint: disable=too-few-public-methods
     class FakeConnection(object):
         """Fake OpenSSL.SSL.Connection."""
 
-        # pylint: disable=too-few-public-methods,missing-docstring
+        # pylint: disable=missing-function-docstring
 
         def __init__(self, connection):
             self._wrapped = connection
@@ -86,7 +85,7 @@ class SSLSocket(object):  # pylint: disable=too-few-public-methods
             # OpenSSL.SSL.Connection.shutdown doesn't accept any args
             return self._wrapped.shutdown()
 
-    def accept(self):  # pylint: disable=missing-docstring
+    def accept(self):  # pylint: disable=missing-function-docstring
         sock, addr = self.sock.accept()
 
         context = SSL.Context(self.method)
@@ -298,7 +297,6 @@ def dump_pyopenssl_chain(chain, filetype=crypto.FILETYPE_PEM):
 
     def _dump_cert(cert):
         if isinstance(cert, jose.ComparableX509):
-            # pylint: disable=protected-access
             cert = cert.wrapped
         return crypto.dump_certificate(filetype, cert)
 

acme/acme/errors.py

@@ -29,7 +29,12 @@ class NonceError(ClientError):
 class BadNonce(NonceError):
     """Bad nonce error."""
     def __init__(self, nonce, error, *args, **kwargs):
-        super(BadNonce, self).__init__(*args, **kwargs)
+        # MyPy complains here that there is too many arguments for BaseException constructor.
+        # This is an error fixed in typeshed, see https://github.com/python/mypy/issues/4183
+        # The fix is included in MyPy>=0.740, but upgrading it would bring dozen of errors due to
+        #   new types definitions. So we ignore the error until the code base is fixed to match
+        #   with MyPy>=0.740 referential.
+        super(BadNonce, self).__init__(*args, **kwargs)  # type: ignore
         self.nonce = nonce
         self.error = error
 
@@ -48,7 +53,8 @@ class MissingNonce(NonceError):
 
     """
     def __init__(self, response, *args, **kwargs):
-        super(MissingNonce, self).__init__(*args, **kwargs)
+        # See comment in BadNonce constructor above for an explanation of type: ignore here.
+        super(MissingNonce, self).__init__(*args, **kwargs)  # type: ignore
         self.response = response
 
     def __str__(self):
@@ -83,6 +89,7 @@ class PollError(ClientError):
         return '{0}(exhausted={1!r}, updated={2!r})'.format(
             self.__class__.__name__, self.exhausted, self.updated)
 
+
 class ValidationError(Error):
     """Error for authorization failures. Contains a list of authorization
     resources, each of which is invalid and should have an error field.
@@ -91,9 +98,11 @@ class ValidationError(Error):
         self.failed_authzrs = failed_authzrs
         super(ValidationError, self).__init__()
 
-class TimeoutError(Error):
+
+class TimeoutError(Error):  # pylint: disable=redefined-builtin
     """Error for when polling an authorization or an order times out."""
 
+
 class IssuanceError(Error):
     """Error sent by the server after requesting issuance of a certificate."""
 
@@ -105,6 +114,7 @@ class IssuanceError(Error):
         self.error = error
         super(IssuanceError, self).__init__()
 
+
 class ConflictError(ClientError):
     """Error for when the server returns a 409 (Conflict) HTTP status.
 

acme/acme/fields.py

@@ -4,7 +4,6 @@ import logging
 import josepy as jose
 import pyrfc3339
 
-
 logger = logging.getLogger(__name__)
 
 

acme/acme/jws.py

@@ -15,7 +15,7 @@ class Header(jose.Header):
     url = jose.Field('url', omitempty=True)
 
     @nonce.decoder
-    def nonce(value):  # pylint: disable=missing-docstring,no-self-argument
+    def nonce(value):  # pylint: disable=no-self-argument,missing-function-docstring
         try:
             return jose.decode_b64jose(value)
         except jose.DeserializationError as error:
@@ -40,10 +40,10 @@ class Signature(jose.Signature):
 class JWS(jose.JWS):
     """ACME-specific JWS. Includes none, url, and kid in protected header."""
     signature_cls = Signature
-    __slots__ = jose.JWS._orig_slots  # pylint: disable=no-member
+    __slots__ = jose.JWS._orig_slots
 
     @classmethod
-    # pylint: disable=arguments-differ,too-many-arguments
+    # pylint: disable=arguments-differ
     def sign(cls, payload, key, alg, nonce, url=None, kid=None):
         # Per ACME spec, jwk and kid are mutually exclusive, so only include a
         # jwk field if kid is not provided.

acme/acme/magic_typing.py

@@ -1,6 +1,7 @@
 """Shim class to not have to depend on typing module in prod."""
 import sys
 
+
 class TypingClass(object):
     """Ignore import errors by getting anything"""
     def __getattr__(self, name):
@@ -9,7 +10,6 @@ class TypingClass(object):
 try:
     # mypy doesn't respect modifying sys.modules
     from typing import *  # pylint: disable=wildcard-import, unused-wildcard-import
-    # pylint: disable=unused-import
     from typing import Collection, IO  # type: ignore
     # pylint: enable=unused-import
 except ImportError:

acme/acme/messages.py

@@ -1,18 +1,21 @@
 """ACME protocol messages."""
 import json
-import six
-try:
-    from collections.abc import Hashable  # pylint: disable=no-name-in-module
-except ImportError:  # pragma: no cover
-    from collections import Hashable
 
 import josepy as jose
+import six
 
 from acme import challenges
 from acme import errors
 from acme import fields
-from acme import util
 from acme import jws
+from acme import util
+
+try:
+    from collections.abc import Hashable
+except ImportError:  # pragma: no cover
+    from collections import Hashable
+
+
 
 OLD_ERROR_PREFIX = "urn:acme:error:"
 ERROR_PREFIX = "urn:ietf:params:acme:error:"
@@ -33,7 +36,7 @@ ERROR_CODES = {
                    ' domain'),
     'dns': 'There was a problem with a DNS query during identifier validation',
     'dnssec': 'The server could not validate a DNSSEC signed domain',
-    'incorrectResponse': 'Response recieved didn\'t match the challenge\'s requirements',
+    'incorrectResponse': 'Response received didn\'t match the challenge\'s requirements',
     # deprecate invalidEmail
     'invalidEmail': 'The provided email for a registration was invalid',
     'invalidContact': 'The provided contact URI was invalid',
@@ -143,7 +146,7 @@ class _Constant(jose.JSONDeSerializable, Hashable):  # type: ignore
         if jobj not in cls.POSSIBLE_NAMES:  # pylint: disable=unsupported-membership-test
             raise jose.DeserializationError(
                 '{0} not recognized'.format(cls.__name__))
-        return cls.POSSIBLE_NAMES[jobj]  # pylint: disable=unsubscriptable-object
+        return cls.POSSIBLE_NAMES[jobj]
 
     def __repr__(self):
         return '{0}({1})'.format(self.__class__.__name__, self.name)
@@ -168,6 +171,7 @@ STATUS_VALID = Status('valid')
 STATUS_INVALID = Status('invalid')
 STATUS_REVOKED = Status('revoked')
 STATUS_READY = Status('ready')
+STATUS_DEACTIVATED = Status('deactivated')
 
 
 class IdentifierType(_Constant):
@@ -241,13 +245,13 @@ class Directory(jose.JSONDeSerializable):
         try:
             return self[name.replace('_', '-')]
         except KeyError as error:
-            raise AttributeError(str(error) + ': ' + name)
+            raise AttributeError(str(error))
 
     def __getitem__(self, name):
         try:
             return self._jobj[self._canon_key(name)]
         except KeyError:
-            raise KeyError('Directory field not found')
+            raise KeyError('Directory field "' + self._canon_key(name) + '" not found')
 
     def to_partial_json(self):
         return self._jobj
@@ -456,7 +460,6 @@ class ChallengeResource(Resource):
     @property
     def uri(self):
         """The URL of the challenge body."""
-        # pylint: disable=function-redefined,no-member
         return self.body.uri
 
 
@@ -471,7 +474,7 @@ class Authorization(ResourceBody):
     :ivar datetime.datetime expires:
 
     """
-    identifier = jose.Field('identifier', decoder=Identifier.from_json)
+    identifier = jose.Field('identifier', decoder=Identifier.from_json, omitempty=True)
     challenges = jose.Field('challenges', omitempty=True)
     combinations = jose.Field('combinations', omitempty=True)
 
@@ -484,7 +487,7 @@ class Authorization(ResourceBody):
     wildcard = jose.Field('wildcard', omitempty=True)
 
     @challenges.decoder
-    def challenges(value):  # pylint: disable=missing-docstring,no-self-argument
+    def challenges(value):  # pylint: disable=no-self-argument,missing-function-docstring
         return tuple(ChallengeBody.from_json(chall) for chall in value)
 
     @property
@@ -501,6 +504,12 @@ class NewAuthorization(Authorization):
     resource = fields.Resource(resource_type)
 
 
+class UpdateAuthorization(Authorization):
+    """Update authorization."""
+    resource_type = 'authz'
+    resource = fields.Resource(resource_type)
+
+
 class AuthorizationResource(ResourceWithURI):
     """Authorization Resource.
 
@@ -575,7 +584,7 @@ class Order(ResourceBody):
     error = jose.Field('error', omitempty=True, decoder=Error.from_json)
 
     @identifiers.decoder
-    def identifiers(value):  # pylint: disable=missing-docstring,no-self-argument
+    def identifiers(value):  # pylint: disable=no-self-argument,missing-function-docstring
         return tuple(Identifier.from_json(identifier) for identifier in value)
 
 class OrderResource(ResourceWithURI):

acme/acme/standalone.py

@@ -1,30 +1,20 @@
 """Support for standalone client challenge solvers. """
-import argparse
 import collections
 import functools
 import logging
-import os
 import socket
-import sys
 import threading
 
-from six.moves import BaseHTTPServer  # type: ignore  # pylint: disable=import-error
-from six.moves import http_client  # pylint: disable=import-error
-from six.moves import socketserver  # type: ignore  # pylint: disable=import-error
-
-import OpenSSL
+from six.moves import BaseHTTPServer  # type: ignore
+from six.moves import http_client
+from six.moves import socketserver  # type: ignore
 
 from acme import challenges
 from acme import crypto_util
-from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module
-from acme import _TLSSNI01DeprecationModule
-
+from acme.magic_typing import List
 
 logger = logging.getLogger(__name__)
 
-# six.moves.* | pylint: disable=no-member,attribute-defined-outside-init
-# pylint: disable=too-few-public-methods,no-init
-
 
 class TLSServer(socketserver.TCPServer):
     """Generic TLS Server."""
@@ -37,7 +27,6 @@ class TLSServer(socketserver.TCPServer):
             self.address_family = socket.AF_INET
         self.certs = kwargs.pop("certs", {})
         self.method = kwargs.pop(
-            # pylint: disable=protected-access
             "method", crypto_util._DEFAULT_SSL_METHOD)
         self.allow_reuse_address = kwargs.pop("allow_reuse_address", True)
         socketserver.TCPServer.__init__(self, *args, **kwargs)
@@ -46,12 +35,12 @@ class TLSServer(socketserver.TCPServer):
         self.socket = crypto_util.SSLSocket(
             self.socket, certs=self.certs, method=self.method)
 
-    def server_bind(self):  # pylint: disable=missing-docstring
+    def server_bind(self):
         self._wrap_sock()
         return socketserver.TCPServer.server_bind(self)
 
 
-class ACMEServerMixin:  # pylint: disable=old-style-class
+class ACMEServerMixin:
     """ACME server common settings mixin."""
     # TODO: c.f. #858
     server_version = "ACME client standalone challenge solver"
@@ -112,7 +101,6 @@ class BaseDualNetworkedServers(object):
         """Wraps socketserver.TCPServer.serve_forever"""
         for server in self.servers:
             thread = threading.Thread(
-                # pylint: disable=no-member
                 target=server.serve_forever)
             thread.start()
             self.threads.append(thread)
@@ -132,35 +120,6 @@ class BaseDualNetworkedServers(object):
         self.threads = []
 
 
-class TLSSNI01Server(TLSServer, ACMEServerMixin):
-    """TLSSNI01 Server."""
-
-    def __init__(self, server_address, certs, ipv6=False):
-        TLSServer.__init__(
-            self, server_address, BaseRequestHandlerWithLogging, certs=certs, ipv6=ipv6)
-
-
-class TLSSNI01DualNetworkedServers(BaseDualNetworkedServers):
-    """TLSSNI01Server Wrapper. Tries everything for both. Failures for one don't
-       affect the other."""
-
-    def __init__(self, *args, **kwargs):
-        BaseDualNetworkedServers.__init__(self, TLSSNI01Server, *args, **kwargs)
-
-
-class BaseRequestHandlerWithLogging(socketserver.BaseRequestHandler):
-    """BaseRequestHandler with logging."""
-
-    def log_message(self, format, *args):  # pylint: disable=redefined-builtin
-        """Log arbitrary message."""
-        logger.debug("%s - - %s", self.client_address[0], format % args)
-
-    def handle(self):
-        """Handle request."""
-        self.log_message("Incoming request")
-        socketserver.BaseRequestHandler.handle(self)
-
-
 class HTTPServer(BaseHTTPServer.HTTPServer):
     """Generic HTTP Server."""
 
@@ -215,7 +174,7 @@ class HTTP01RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
         self.log_message("Incoming request")
         BaseHTTPServer.BaseHTTPRequestHandler.handle(self)
 
-    def do_GET(self):  # pylint: disable=invalid-name,missing-docstring
+    def do_GET(self):  # pylint: disable=invalid-name,missing-function-docstring
         if self.path == "/":
             self.handle_index()
         elif self.path.startswith("/" + challenges.HTTP01.URI_ROOT_PATH):
@@ -263,43 +222,3 @@ class HTTP01RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
         """
         return functools.partial(
             cls, simple_http_resources=simple_http_resources)
-
-
-def simple_tls_sni_01_server(cli_args, forever=True):
-    """Run simple standalone TLSSNI01 server."""
-    logging.basicConfig(level=logging.DEBUG)
-
-    parser = argparse.ArgumentParser()
-    parser.add_argument(
-        "-p", "--port", default=0, help="Port to serve at. By default "
-        "picks random free port.")
-    args = parser.parse_args(cli_args[1:])
-
-    certs = {}
-
-    _, hosts, _ = next(os.walk('.')) # type: ignore # https://github.com/python/mypy/issues/465
-    for host in hosts:
-        with open(os.path.join(host, "cert.pem")) as cert_file:
-            cert_contents = cert_file.read()
-        with open(os.path.join(host, "key.pem")) as key_file:
-            key_contents = key_file.read()
-        certs[host.encode()] = (
-            OpenSSL.crypto.load_privatekey(
-                OpenSSL.crypto.FILETYPE_PEM, key_contents),
-            OpenSSL.crypto.load_certificate(
-                OpenSSL.crypto.FILETYPE_PEM, cert_contents))
-
-    server = TLSSNI01Server(('', int(args.port)), certs=certs)
-    logger.info("Serving at https://%s:%s...", *server.socket.getsockname()[:2])
-    if forever:  # pragma: no cover
-        server.serve_forever()
-    else:
-        server.handle_request()
-
-
-# Patching ourselves to warn about TLS-SNI challenge deprecation and removal.
-sys.modules[__name__] = _TLSSNI01DeprecationModule(sys.modules[__name__])
-
-
-if __name__ == "__main__":
-    sys.exit(simple_tls_sni_01_server(sys.argv))  # pragma: no cover

acme/docs/conf.py

@@ -12,10 +12,9 @@
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-import sys
 import os
 import shlex
-
+import sys
 
 here = os.path.abspath(os.path.dirname(__file__))
 
@@ -42,7 +41,7 @@ extensions = [
 ]
 
 autodoc_member_order = 'bysource'
-autodoc_default_flags = ['show-inheritance', 'private-members']
+autodoc_default_flags = ['show-inheritance']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -114,7 +113,7 @@ pygments_style = 'sphinx'
 #keep_warnings = False
 
 # If true, `todo` and `todoList` produce output, else they produce nothing.
-todo_include_todos = True
+todo_include_todos = False
 
 
 # -- Options for HTML output ----------------------------------------------

acme/examples/http01_example.py

@@ -26,8 +26,10 @@ Workflow:
     - Deactivate Account
 """
 from contextlib import contextmanager
+
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives.asymmetric import rsa
+import josepy as jose
 import OpenSSL
 
 from acme import challenges
@@ -36,7 +38,6 @@ from acme import crypto_util
 from acme import errors
 from acme import messages
 from acme import standalone
-import josepy as jose
 
 # Constants:
 

acme/readthedocs.org.requirements.txt

@@ -1,10 +1,10 @@
 # readthedocs.org gives no way to change the install command to "pip
-# install -e .[docs]" (that would in turn install documentation
+# install -e acme[docs]" (that would in turn install documentation
 # dependencies), but it allows to specify a requirements.txt file at
 # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259)
 
 # Although ReadTheDocs certainly doesn't need to install the project
-# in --editable mode (-e), just "pip install .[docs]" does not work as
-# expected and "pip install -e .[docs]" must be used instead
+# in --editable mode (-e), just "pip install acme[docs]" does not work as
+# expected and "pip install -e acme[docs]" must be used instead
 
 -e acme[docs]

acme/setup.py

@@ -1,9 +1,10 @@
-from setuptools import setup
-from setuptools import find_packages
-from setuptools.command.test import test as TestCommand
 import sys
 
-version = '0.37.0.dev0'
+from setuptools import find_packages
+from setuptools import setup
+from setuptools.command.test import test as TestCommand
+
+version = '1.3.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [
@@ -14,8 +15,8 @@ install_requires = [
     # 1.1.0+ is required to avoid the warnings described at
     # https://github.com/certbot/josepy/issues/13.
     'josepy>=1.1.0',
-    # Connection.set_tlsext_host_name (>=0.13)
     'mock',
+    # Connection.set_tlsext_host_name (>=0.13)
     'PyOpenSSL>=0.13.1',
     'pyrfc3339',
     'pytz',
@@ -60,7 +61,7 @@ setup(
     author="Certbot Project",
     author_email='client-dev@letsencrypt.org',
     license='Apache License 2.0',
-    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*',
+    python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*',
     classifiers=[
         'Development Status :: 5 - Production/Stable',
         'Intended Audience :: Developers',
@@ -69,10 +70,10 @@ setup(
         'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
         'Topic :: Internet :: WWW/HTTP',
         'Topic :: Security',
     ],

acme/tests/challenges_test.py

@@ -3,13 +3,10 @@ import unittest
 
 import josepy as jose
 import mock
-import OpenSSL
 import requests
+from six.moves.urllib import parse as urllib_parse
 
-from six.moves.urllib import parse as urllib_parse  # pylint: disable=relative-import
-
-from acme import errors
-from acme import test_util
+import test_util
 
 CERT = test_util.load_comparable_cert('cert.pem')
 KEY = jose.JWKRSA(key=test_util.load_rsa_private_key('rsa512_key.pem'))
@@ -21,7 +18,6 @@ class ChallengeTest(unittest.TestCase):
         from acme.challenges import Challenge
         from acme.challenges import UnrecognizedChallenge
         chall = UnrecognizedChallenge({"type": "foo"})
-        # pylint: disable=no-member
         self.assertEqual(chall, Challenge.from_json(chall.jobj))
 
 
@@ -77,7 +73,6 @@ class KeyAuthorizationChallengeResponseTest(unittest.TestCase):
 
 
 class DNS01ResponseTest(unittest.TestCase):
-    # pylint: disable=too-many-instance-attributes
 
     def setUp(self):
         from acme.challenges import DNS01Response
@@ -149,7 +144,6 @@ class DNS01Test(unittest.TestCase):
 
 
 class HTTP01ResponseTest(unittest.TestCase):
-    # pylint: disable=too-many-instance-attributes
 
     def setUp(self):
         from acme.challenges import HTTP01Response
@@ -187,7 +181,7 @@ class HTTP01ResponseTest(unittest.TestCase):
         mock_get.return_value = mock.MagicMock(text=validation)
         self.assertTrue(self.response.simple_verify(
             self.chall, "local", KEY.public_key()))
-        mock_get.assert_called_once_with(self.chall.uri("local"))
+        mock_get.assert_called_once_with(self.chall.uri("local"), verify=False)
 
     @mock.patch("acme.challenges.requests.get")
     def test_simple_verify_bad_validation(self, mock_get):
@@ -203,7 +197,7 @@ class HTTP01ResponseTest(unittest.TestCase):
                   HTTP01Response.WHITESPACE_CUTSET))
         self.assertTrue(self.response.simple_verify(
             self.chall, "local", KEY.public_key()))
-        mock_get.assert_called_once_with(self.chall.uri("local"))
+        mock_get.assert_called_once_with(self.chall.uri("local"), verify=False)
 
     @mock.patch("acme.challenges.requests.get")
     def test_simple_verify_connection_error(self, mock_get):
@@ -259,152 +253,7 @@ class HTTP01Test(unittest.TestCase):
             self.msg.update(token=b'..').good_token)
 
 
-class TLSSNI01ResponseTest(unittest.TestCase):
-    # pylint: disable=too-many-instance-attributes
-
-    def setUp(self):
-        from acme.challenges import TLSSNI01
-        self.chall = TLSSNI01(
-            token=jose.b64decode(b'a82d5ff8ef740d12881f6d3c2277ab2e'))
-
-        self.response = self.chall.response(KEY)
-        self.jmsg = {
-            'resource': 'challenge',
-            'type': 'tls-sni-01',
-            'keyAuthorization': self.response.key_authorization,
-        }
-
-        # pylint: disable=invalid-name
-        label1 = b'dc38d9c3fa1a4fdcc3a5501f2d38583f'
-        label2 = b'b7793728f084394f2a1afd459556bb5c'
-        self.z = label1 + label2
-        self.z_domain = label1 + b'.' + label2 + b'.acme.invalid'
-        self.domain = 'foo.com'
-
-    def test_z_and_domain(self):
-        self.assertEqual(self.z, self.response.z)
-        self.assertEqual(self.z_domain, self.response.z_domain)
-
-    def test_to_partial_json(self):
-        self.assertEqual({k: v for k, v in self.jmsg.items() if k != 'keyAuthorization'},
-                         self.response.to_partial_json())
-
-    def test_from_json(self):
-        from acme.challenges import TLSSNI01Response
-        self.assertEqual(self.response, TLSSNI01Response.from_json(self.jmsg))
-
-    def test_from_json_hashable(self):
-        from acme.challenges import TLSSNI01Response
-        hash(TLSSNI01Response.from_json(self.jmsg))
-
-    @mock.patch('acme.challenges.socket.gethostbyname')
-    @mock.patch('acme.challenges.crypto_util.probe_sni')
-    def test_probe_cert(self, mock_probe_sni, mock_gethostbyname):
-        mock_gethostbyname.return_value = '127.0.0.1'
-        self.response.probe_cert('foo.com')
-        mock_gethostbyname.assert_called_once_with('foo.com')
-        mock_probe_sni.assert_called_once_with(
-            host='127.0.0.1', port=self.response.PORT,
-            name=self.z_domain)
-
-        self.response.probe_cert('foo.com', host='8.8.8.8')
-        mock_probe_sni.assert_called_with(
-            host='8.8.8.8', port=mock.ANY, name=mock.ANY)
-
-        self.response.probe_cert('foo.com', port=1234)
-        mock_probe_sni.assert_called_with(
-            host=mock.ANY, port=1234, name=mock.ANY)
-
-        self.response.probe_cert('foo.com', bar='baz')
-        mock_probe_sni.assert_called_with(
-            host=mock.ANY, port=mock.ANY, name=mock.ANY, bar='baz')
-
-        self.response.probe_cert('foo.com', name=b'xxx')
-        mock_probe_sni.assert_called_with(
-            host=mock.ANY, port=mock.ANY,
-            name=self.z_domain)
-
-    def test_gen_verify_cert(self):
-        key1 = test_util.load_pyopenssl_private_key('rsa512_key.pem')
-        cert, key2 = self.response.gen_cert(key1)
-        self.assertEqual(key1, key2)
-        self.assertTrue(self.response.verify_cert(cert))
-
-    def test_gen_verify_cert_gen_key(self):
-        cert, key = self.response.gen_cert()
-        self.assertTrue(isinstance(key, OpenSSL.crypto.PKey))
-        self.assertTrue(self.response.verify_cert(cert))
-
-    def test_verify_bad_cert(self):
-        self.assertFalse(self.response.verify_cert(
-            test_util.load_cert('cert.pem')))
-
-    def test_simple_verify_bad_key_authorization(self):
-        key2 = jose.JWKRSA.load(test_util.load_vector('rsa256_key.pem'))
-        self.response.simple_verify(self.chall, "local", key2.public_key())
-
-    @mock.patch('acme.challenges.TLSSNI01Response.verify_cert', autospec=True)
-    def test_simple_verify(self, mock_verify_cert):
-        mock_verify_cert.return_value = mock.sentinel.verification
-        self.assertEqual(
-            mock.sentinel.verification, self.response.simple_verify(
-                self.chall, self.domain, KEY.public_key(),
-                cert=mock.sentinel.cert))
-        mock_verify_cert.assert_called_once_with(
-            self.response, mock.sentinel.cert)
-
-    @mock.patch('acme.challenges.TLSSNI01Response.probe_cert')
-    def test_simple_verify_false_on_probe_error(self, mock_probe_cert):
-        mock_probe_cert.side_effect = errors.Error
-        self.assertFalse(self.response.simple_verify(
-            self.chall, self.domain, KEY.public_key()))
-
-
-class TLSSNI01Test(unittest.TestCase):
-
-    def setUp(self):
-        self.jmsg = {
-            'type': 'tls-sni-01',
-            'token': 'a82d5ff8ef740d12881f6d3c2277ab2e',
-        }
-        from acme.challenges import TLSSNI01
-        self.msg = TLSSNI01(
-            token=jose.b64decode('a82d5ff8ef740d12881f6d3c2277ab2e'))
-
-    def test_to_partial_json(self):
-        self.assertEqual(self.jmsg, self.msg.to_partial_json())
-
-    def test_from_json(self):
-        from acme.challenges import TLSSNI01
-        self.assertEqual(self.msg, TLSSNI01.from_json(self.jmsg))
-
-    def test_from_json_hashable(self):
-        from acme.challenges import TLSSNI01
-        hash(TLSSNI01.from_json(self.jmsg))
-
-    def test_from_json_invalid_token_length(self):
-        from acme.challenges import TLSSNI01
-        self.jmsg['token'] = jose.encode_b64jose(b'abcd')
-        self.assertRaises(
-            jose.DeserializationError, TLSSNI01.from_json, self.jmsg)
-
-    @mock.patch('acme.challenges.TLSSNI01Response.gen_cert')
-    def test_validation(self, mock_gen_cert):
-        mock_gen_cert.return_value = ('cert', 'key')
-        self.assertEqual(('cert', 'key'), self.msg.validation(
-            KEY, cert_key=mock.sentinel.cert_key))
-        mock_gen_cert.assert_called_once_with(key=mock.sentinel.cert_key)
-
-    def test_deprecation_message(self):
-        with mock.patch('acme.warnings.warn') as mock_warn:
-            from acme.challenges import TLSSNI01
-            assert TLSSNI01
-        self.assertEqual(mock_warn.call_count, 1)
-        self.assertTrue('deprecated' in mock_warn.call_args[0][0])
-
-
 class TLSALPN01ResponseTest(unittest.TestCase):
-    # pylint: disable=too-many-instance-attributes
 
     def setUp(self):
         from acme.challenges import TLSALPN01Response

acme/tests/client_test.py

@@ -5,21 +5,19 @@ import datetime
 import json
 import unittest
 
-from six.moves import http_client  # pylint: disable=import-error
-
 import josepy as jose
 import mock
 import OpenSSL
 import requests
+from six.moves import http_client  # pylint: disable=import-error
 
 from acme import challenges
 from acme import errors
 from acme import jws as acme_jws
 from acme import messages
-from acme import messages_test
-from acme import test_util
-from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module
-
+from acme.magic_typing import Dict  # pylint: disable=unused-import, no-name-in-module
+import messages_test
+import test_util
 
 CERT_DER = test_util.load_vector('cert.der')
 CERT_SAN_PEM = test_util.load_vector('cert-san.pem')
@@ -63,7 +61,7 @@ class ClientTestBase(unittest.TestCase):
         self.contact = ('mailto:cert-admin@example.com', 'tel:+12025551212')
         reg = messages.Registration(
             contact=self.contact, key=KEY.public_key())
-        the_arg = dict(reg) # type: Dict
+        the_arg = dict(reg)  # type: Dict
         self.new_reg = messages.NewRegistration(**the_arg)
         self.regr = messages.RegistrationResource(
             body=reg, uri='https://www.letsencrypt-demo.org/acme/reg/1')
@@ -318,7 +316,6 @@ class BackwardsCompatibleClientV2Test(ClientTestBase):
 
 class ClientTest(ClientTestBase):
     """Tests for acme.client.Client."""
-    # pylint: disable=too-many-instance-attributes,too-many-public-methods
 
     def setUp(self):
         super(ClientTest, self).setUp()
@@ -637,6 +634,14 @@ class ClientTest(ClientTestBase):
             errors.PollError, self.client.poll_and_request_issuance,
             csr, authzrs, mintime=mintime, max_attempts=2)
 
+    def test_deactivate_authorization(self):
+        authzb = self.authzr.body.update(status=messages.STATUS_DEACTIVATED)
+        self.response.json.return_value = authzb.to_json()
+        authzr = self.client.deactivate_authorization(self.authzr)
+        self.assertEqual(authzb, authzr.body)
+        self.assertEqual(self.client.net.post.call_count, 1)
+        self.assertTrue(self.authzr.uri in self.net.post.call_args_list[0][0])
+
     def test_check_cert(self):
         self.response.headers['Location'] = self.certr.uri
         self.response.content = CERT_DER
@@ -880,19 +885,6 @@ class ClientV2Test(ClientTestBase):
                 new_nonce_url='https://www.letsencrypt-demo.org/acme/new-nonce')
             self.client.net.get.assert_not_called()
 
-            class FakeError(messages.Error):  # pylint: disable=too-many-ancestors
-                """Fake error to reproduce a malformed request ACME error"""
-                def __init__(self):  # pylint: disable=super-init-not-called
-                    pass
-                @property
-                def code(self):
-                    return 'malformed'
-            self.client.net.post.side_effect = FakeError()
-
-            self.client.poll(self.authzr2)  # pylint: disable=protected-access
-
-            self.client.net.get.assert_called_once_with(self.authzr2.uri)
-
 
 class MockJSONDeSerializable(jose.JSONDeSerializable):
     # pylint: disable=missing-docstring
@@ -909,7 +901,6 @@ class MockJSONDeSerializable(jose.JSONDeSerializable):
 
 class ClientNetworkTest(unittest.TestCase):
     """Tests for acme.client.ClientNetwork."""
-    # pylint: disable=too-many-public-methods
 
     def setUp(self):
         self.verify_ssl = mock.MagicMock()
@@ -959,8 +950,8 @@ class ClientNetworkTest(unittest.TestCase):
 
     def test_check_response_not_ok_jobj_error(self):
         self.response.ok = False
-        self.response.json.return_value = messages.Error(
-            detail='foo', typ='serverInternal', title='some title').to_json()
+        self.response.json.return_value = messages.Error.with_code(
+            'serverInternal', detail='foo', title='some title').to_json()
         # pylint: disable=protected-access
         self.assertRaises(
             messages.Error, self.net._check_response, self.response)
@@ -985,10 +976,39 @@ class ClientNetworkTest(unittest.TestCase):
         self.response.json.side_effect = ValueError
         for response_ct in [self.net.JSON_CONTENT_TYPE, 'foo']:
             self.response.headers['Content-Type'] = response_ct
-            # pylint: disable=protected-access,no-value-for-parameter
+            # pylint: disable=protected-access
             self.assertEqual(
                 self.response, self.net._check_response(self.response))
 
+    @mock.patch('acme.client.logger')
+    def test_check_response_ok_ct_with_charset(self, mock_logger):
+        self.response.json.return_value = {}
+        self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
+        # pylint: disable=protected-access
+        self.assertEqual(self.response, self.net._check_response(
+            self.response, content_type='application/json'))
+        try:
+            mock_logger.debug.assert_called_with(
+                'Ignoring wrong Content-Type (%r) for JSON decodable response',
+                'application/json; charset=utf-8'
+            )
+        except AssertionError:
+            return
+        raise AssertionError('Expected Content-Type warning ' #pragma: no cover
+            'to not have been logged')
+
+    @mock.patch('acme.client.logger')
+    def test_check_response_ok_bad_ct(self, mock_logger):
+        self.response.json.return_value = {}
+        self.response.headers['Content-Type'] = 'text/plain'
+        # pylint: disable=protected-access
+        self.assertEqual(self.response, self.net._check_response(
+            self.response, content_type='application/json'))
+        mock_logger.debug.assert_called_with(
+            'Ignoring wrong Content-Type (%r) for JSON decodable response',
+            'text/plain'
+        )
+
     def test_check_response_conflict(self):
         self.response.ok = False
         self.response.status_code = 409
@@ -999,7 +1019,7 @@ class ClientNetworkTest(unittest.TestCase):
         self.response.json.return_value = {}
         for response_ct in [self.net.JSON_CONTENT_TYPE, 'foo']:
             self.response.headers['Content-Type'] = response_ct
-            # pylint: disable=protected-access,no-value-for-parameter
+            # pylint: disable=protected-access
             self.assertEqual(
                 self.response, self.net._check_response(self.response))
 
@@ -1115,7 +1135,6 @@ class ClientNetworkTest(unittest.TestCase):
 
 class ClientNetworkWithMockedResponseTest(unittest.TestCase):
     """Tests for acme.client.ClientNetwork which mock out response."""
-    # pylint: disable=too-many-instance-attributes
 
     def setUp(self):
         from acme.client import ClientNetwork
@@ -1125,8 +1144,8 @@ class ClientNetworkWithMockedResponseTest(unittest.TestCase):
         self.response.headers = {}
         self.response.links = {}
         self.response.checked = False
-        self.acmev1_nonce_response = mock.MagicMock(ok=False,
-            status_code=http_client.METHOD_NOT_ALLOWED)
+        self.acmev1_nonce_response = mock.MagicMock(
+            ok=False, status_code=http_client.METHOD_NOT_ALLOWED)
         self.acmev1_nonce_response.headers = {}
         self.obj = mock.MagicMock()
         self.wrapped_obj = mock.MagicMock()

acme/tests/crypto_util_test.py

@@ -5,15 +5,14 @@ import threading
 import time
 import unittest
 
-import six
-from six.moves import socketserver  #type: ignore  # pylint: disable=import-error
-
 import josepy as jose
 import OpenSSL
+import six
+from six.moves import socketserver  # type: ignore  # pylint: disable=import-error
 
 from acme import errors
-from acme import test_util
-from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module
+from acme.magic_typing import List  # pylint: disable=unused-import, no-name-in-module
+import test_util
 
 
 class SSLSocketAndProbeSNITest(unittest.TestCase):
@@ -30,7 +29,6 @@ class SSLSocketAndProbeSNITest(unittest.TestCase):
 
         class _TestServer(socketserver.TCPServer):
 
-            # pylint: disable=too-few-public-methods
             # six.moves.* | pylint: disable=attribute-defined-outside-init,no-init
 
             def server_bind(self):  # pylint: disable=missing-docstring
@@ -40,7 +38,6 @@ class SSLSocketAndProbeSNITest(unittest.TestCase):
         self.server = _TestServer(('', 0), socketserver.BaseRequestHandler)
         self.port = self.server.socket.getsockname()[1]
         self.server_thread = threading.Thread(
-            # pylint: disable=no-member
             target=self.server.handle_request)
 
     def tearDown(self):
@@ -67,7 +64,7 @@ class SSLSocketAndProbeSNITest(unittest.TestCase):
 
     def test_probe_connection_error(self):
         # pylint has a hard time with six
-        self.server.server_close()  # pylint: disable=no-member
+        self.server.server_close()
         original_timeout = socket.getdefaulttimeout()
         try:
             socket.setdefaulttimeout(1)

acme/tests/jose_test.py

@@ -2,6 +2,7 @@
 import importlib
 import unittest
 
+
 class JoseTest(unittest.TestCase):
     """Tests for acme.jose shim."""
 
@@ -20,11 +21,10 @@ class JoseTest(unittest.TestCase):
 
         # We use the imports below with eval, but pylint doesn't
         # understand that.
-        # pylint: disable=eval-used,unused-variable
-        import acme
-        import josepy
-        acme_jose_mod = eval(acme_jose_path)
-        josepy_mod = eval(josepy_path)
+        import acme  # pylint: disable=unused-import
+        import josepy  # pylint: disable=unused-import
+        acme_jose_mod = eval(acme_jose_path)  # pylint: disable=eval-used
+        josepy_mod = eval(josepy_path)  # pylint: disable=eval-used
         self.assertIs(acme_jose_mod, josepy_mod)
         self.assertIs(getattr(acme_jose_mod, attribute), getattr(josepy_mod, attribute))
 

acme/tests/jws_test.py

@@ -3,8 +3,7 @@ import unittest
 
 import josepy as jose
 
-from acme import test_util
-
+import test_util
 
 KEY = jose.JWKRSA.load(test_util.load_vector('rsa512_key.pem'))
 

acme/tests/messages_test.py

@@ -5,9 +5,8 @@ import josepy as jose
 import mock
 
 from acme import challenges
-from acme import test_util
-from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module
-
+from acme.magic_typing import Dict  # pylint: disable=unused-import, no-name-in-module
+import test_util
 
 CERT = test_util.load_comparable_cert('cert.der')
 CSR = test_util.load_comparable_csr('csr.der')
@@ -19,8 +18,7 @@ class ErrorTest(unittest.TestCase):
 
     def setUp(self):
         from acme.messages import Error, ERROR_PREFIX
-        self.error = Error(
-            detail='foo', typ=ERROR_PREFIX + 'malformed', title='title')
+        self.error = Error.with_code('malformed', detail='foo', title='title')
         self.jobj = {
             'detail': 'foo',
             'title': 'some title',
@@ -28,7 +26,6 @@ class ErrorTest(unittest.TestCase):
         }
         self.error_custom = Error(typ='custom', detail='bar')
         self.empty_error = Error()
-        self.jobj_custom = {'type': 'custom', 'detail': 'bar'}
 
     def test_default_typ(self):
         from acme.messages import Error
@@ -43,8 +40,7 @@ class ErrorTest(unittest.TestCase):
         hash(Error.from_json(self.error.to_json()))
 
     def test_description(self):
-        self.assertEqual(
-            'The request message was malformed', self.error.description)
+        self.assertEqual('The request message was malformed', self.error.description)
         self.assertTrue(self.error_custom.description is None)
 
     def test_code(self):
@@ -54,17 +50,17 @@ class ErrorTest(unittest.TestCase):
         self.assertEqual(None, Error().code)
 
     def test_is_acme_error(self):
-        from acme.messages import is_acme_error
+        from acme.messages import is_acme_error, Error
         self.assertTrue(is_acme_error(self.error))
         self.assertFalse(is_acme_error(self.error_custom))
+        self.assertFalse(is_acme_error(Error()))
         self.assertFalse(is_acme_error(self.empty_error))
         self.assertFalse(is_acme_error("must pet all the {dogs|rabbits}"))
 
     def test_unicode_error(self):
-        from acme.messages import Error, ERROR_PREFIX, is_acme_error
-        arabic_error = Error(
-                detail=u'\u0639\u062f\u0627\u0644\u0629', typ=ERROR_PREFIX + 'malformed',
-            title='title')
+        from acme.messages import Error, is_acme_error
+        arabic_error = Error.with_code(
+            'malformed', detail=u'\u0639\u062f\u0627\u0644\u0629', title='title')
         self.assertTrue(is_acme_error(arabic_error))
 
     def test_with_code(self):
@@ -305,8 +301,7 @@ class ChallengeBodyTest(unittest.TestCase):
         from acme.messages import Error
         from acme.messages import STATUS_INVALID
         self.status = STATUS_INVALID
-        error = Error(typ='urn:ietf:params:acme:error:serverInternal',
-                      detail='Unable to communicate with DNS server')
+        error = Error.with_code('serverInternal', detail='Unable to communicate with DNS server')
         self.challb = ChallengeBody(
             uri='http://challb', chall=self.chall, status=self.status,
             error=error)

acme/tests/standalone_test.py

@@ -1,26 +1,17 @@
 """Tests for acme.standalone."""
-import multiprocessing
-import os
-import shutil
 import socket
 import threading
-import tempfile
 import unittest
-import time
-from contextlib import closing
-
-from six.moves import http_client  # pylint: disable=import-error
-from six.moves import socketserver  # type: ignore  # pylint: disable=import-error
 
 import josepy as jose
 import mock
 import requests
+from six.moves import http_client  # pylint: disable=import-error
+from six.moves import socketserver  # type: ignore  # pylint: disable=import-error
 
 from acme import challenges
-from acme import crypto_util
-from acme import errors
-from acme import test_util
-from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module
+from acme.magic_typing import Set  # pylint: disable=unused-import, no-name-in-module
+import test_util
 
 
 class TLSServerTest(unittest.TestCase):
@@ -41,32 +32,6 @@ class TLSServerTest(unittest.TestCase):
             server.server_close()
 
 
-class TLSSNI01ServerTest(unittest.TestCase):
-    """Test for acme.standalone.TLSSNI01Server."""
-
-
-    def setUp(self):
-        self.certs = {b'localhost': (
-            test_util.load_pyopenssl_private_key('rsa2048_key.pem'),
-            test_util.load_cert('rsa2048_cert.pem'),
-        )}
-        from acme.standalone import TLSSNI01Server
-        self.server = TLSSNI01Server(('localhost', 0), certs=self.certs)
-        self.thread = threading.Thread(target=self.server.serve_forever)
-        self.thread.start()
-
-    def tearDown(self):
-        self.server.shutdown()
-        self.thread.join()
-
-    def test_it(self):
-        host, port = self.server.socket.getsockname()[:2]
-        cert = crypto_util.probe_sni(
-            b'localhost', host=host, port=port, timeout=1)
-        self.assertEqual(jose.ComparableX509(cert),
-                         jose.ComparableX509(self.certs[b'localhost'][1]))
-
-
 class HTTP01ServerTest(unittest.TestCase):
     """Tests for acme.standalone.HTTP01Server."""
 
@@ -170,33 +135,6 @@ class BaseDualNetworkedServersTest(unittest.TestCase):
             prev_port = port
 
 
-class TLSSNI01DualNetworkedServersTest(unittest.TestCase):
-    """Test for acme.standalone.TLSSNI01DualNetworkedServers."""
-
-
-    def setUp(self):
-        self.certs = {b'localhost': (
-            test_util.load_pyopenssl_private_key('rsa2048_key.pem'),
-            test_util.load_cert('rsa2048_cert.pem'),
-        )}
-        from acme.standalone import TLSSNI01DualNetworkedServers
-        self.servers = TLSSNI01DualNetworkedServers(('localhost', 0), certs=self.certs)
-        self.servers.serve_forever()
-
-    def tearDown(self):
-        self.servers.shutdown_and_server_close()
-
-    def test_connect(self):
-        socknames = self.servers.getsocknames()
-        # connect to all addresses
-        for sockname in socknames:
-            host, port = sockname[:2]
-            cert = crypto_util.probe_sni(
-                b'localhost', host=host, port=port, timeout=1)
-            self.assertEqual(jose.ComparableX509(cert),
-                             jose.ComparableX509(self.certs[b'localhost'][1]))
-
-
 class HTTP01DualNetworkedServersTest(unittest.TestCase):
     """Tests for acme.standalone.HTTP01DualNetworkedServers."""
 
@@ -247,60 +185,5 @@ class HTTP01DualNetworkedServersTest(unittest.TestCase):
         self.assertFalse(self._test_http01(add=False))
 
 
-class TestSimpleTLSSNI01Server(unittest.TestCase):
-    """Tests for acme.standalone.simple_tls_sni_01_server."""
-
-
-    def setUp(self):
-        # mirror ../examples/standalone
-        self.test_cwd = tempfile.mkdtemp()
-        localhost_dir = os.path.join(self.test_cwd, 'localhost')
-        os.makedirs(localhost_dir)
-        shutil.copy(test_util.vector_path('rsa2048_cert.pem'),
-                    os.path.join(localhost_dir, 'cert.pem'))
-        shutil.copy(test_util.vector_path('rsa2048_key.pem'),
-                    os.path.join(localhost_dir, 'key.pem'))
-
-        with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
-            sock.bind(('', 0))
-            sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-            self.port = sock.getsockname()[1]
-
-        from acme.standalone import simple_tls_sni_01_server
-        self.process = multiprocessing.Process(target=simple_tls_sni_01_server,
-                                               args=(['path', '-p', str(self.port)],))
-        self.old_cwd = os.getcwd()
-        os.chdir(self.test_cwd)
-
-    def tearDown(self):
-        os.chdir(self.old_cwd)
-        if self.process.is_alive():
-            self.process.terminate()
-            self.process.join(timeout=5)
-            # Check that we didn't timeout waiting for the process to
-            # terminate.
-            self.assertNotEqual(self.process.exitcode, None)
-        shutil.rmtree(self.test_cwd)
-
-    @mock.patch('acme.standalone.TLSSNI01Server.handle_request')
-    def test_mock(self, handle):
-        from acme.standalone import simple_tls_sni_01_server
-        simple_tls_sni_01_server(cli_args=['path', '-p', str(self.port)], forever=False)
-        self.assertEqual(handle.call_count, 1)
-
-    def test_live(self):
-        self.process.start()
-        cert = None
-        for _ in range(50):
-            time.sleep(0.1)
-            try:
-                cert = crypto_util.probe_sni(b'localhost', b'127.0.0.1', self.port)
-                break
-            except errors.Error:  # pragma: no cover
-                pass
-        self.assertEqual(jose.ComparableX509(cert),
-                         test_util.load_comparable_cert('rsa2048_cert.pem'))
-
-
 if __name__ == "__main__":
     unittest.main()  # pragma: no cover

acme/tests/test_util.py

@@ -4,19 +4,12 @@
 
 """
 import os
-import unittest
-import pkg_resources
 
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import serialization
 import josepy as jose
 from OpenSSL import crypto
-
-
-def vector_path(*names):
-    """Path to a test vector."""
-    return pkg_resources.resource_filename(
-        __name__, os.path.join('testdata', *names))
+import pkg_resources
 
 
 def load_vector(*names):
@@ -32,8 +25,7 @@ def _guess_loader(filename, loader_pem, loader_der):
         return loader_pem
     elif ext.lower() == '.der':
         return loader_der
-    else:  # pragma: no cover
-        raise ValueError("Loader could not be recognized based on extension")
+    raise ValueError("Loader could not be recognized based on extension")  # pragma: no cover
 
 
 def load_cert(*names):
@@ -73,23 +65,3 @@ def load_pyopenssl_private_key(*names):
     loader = _guess_loader(
         names[-1], crypto.FILETYPE_PEM, crypto.FILETYPE_ASN1)
     return crypto.load_privatekey(loader, load_vector(*names))
-
-
-def skip_unless(condition, reason):  # pragma: no cover
-    """Skip tests unless a condition holds.
-
-    This implements the basic functionality of unittest.skipUnless
-    which is only available on Python 2.7+.
-
-    :param bool condition: If ``False``, the test will be skipped
-    :param str reason: the reason for skipping the test
-
-    :rtype: callable
-    :returns: decorator that hides tests unless condition is ``True``
-
-    """
-    if hasattr(unittest, "skipUnless"):
-        return unittest.skipUnless(condition, reason)
-    elif condition:
-        return lambda cls: cls
-    return lambda cls: None

appveyor.yml

@@ -1,44 +0,0 @@
-image: Visual Studio 2015
-
-environment:
-  matrix:
-    - TOXENV: py35
-    - TOXENV: py37-cover
-
-branches:
-  only:
-    # apache-parser-v2 is a temporary branch for doing work related to
-    # rewriting the parser in the Apache plugin.
-    - apache-parser-v2
-    - master
-    - /^\d+\.\d+\.x$/ # Version branches like X.X.X
-    - /^test-.*$/
-
-init:
-  # Since master can receive only commits from PR that have already been tested, following
-  # condition avoid to launch all jobs except the coverage one for commits pushed to master.
-  - ps: |
-      if (-Not $Env:APPVEYOR_PULL_REQUEST_NUMBER -And $Env:APPVEYOR_REPO_BRANCH -Eq 'master' `
-          -And -Not ($Env:TOXENV -Like '*-cover'))
-      { $Env:APPVEYOR_SKIP_FINALIZE_ON_EXIT = 'true'; Exit-AppVeyorBuild }
-
-install:
-  # Use Python 3.7 by default
-  - "SET PATH=C:\\Python37;C:\\Python37\\Scripts;%PATH%"
-  # Check env
-  - "python --version"
-  # Upgrade pip to avoid warnings
-  - "python -m pip install --upgrade pip"
-  # Ready to install tox and coverage
-  # tools/pip_install.py is used to pin packages to a known working version.
-  - "python tools\\pip_install.py tox codecov"
-
-build: off
-
-test_script:
-  - set TOX_TESTENV_PASSENV=APPVEYOR
-  # Test env is set by TOXENV env variable
-  - tox
-
-on_success:
-  - if exist .coverage codecov -F windows

certbot-apache/MANIFEST.in

@@ -1,7 +1,7 @@
 include LICENSE.txt
 include README.rst
-recursive-include docs *
-recursive-include certbot_apache/tests/testdata *
-include certbot_apache/centos-options-ssl-apache.conf
-include certbot_apache/options-ssl-apache.conf
-recursive-include certbot_apache/augeas_lens *.aug
+recursive-include tests *
+include certbot_apache/_internal/options-ssl-apache.conf
+recursive-include certbot_apache/_internal/augeas_lens *.aug
+global-exclude __pycache__
+global-exclude *.py[cod]

certbot-apache/certbot_apache/_internal/__init__.py

@@ -0,0 +1 @@
+"""Certbot Apache plugin."""

certbot-apache/certbot_apache/_internal/apache_util.py

@@ -0,0 +1,243 @@
+""" Utility functions for certbot-apache plugin """
+import binascii
+import fnmatch
+import logging
+import re
+import subprocess
+
+from certbot import errors
+from certbot import util
+
+from certbot.compat import os
+
+logger = logging.getLogger(__name__)
+
+
+def get_mod_deps(mod_name):
+    """Get known module dependencies.
+
+    .. note:: This does not need to be accurate in order for the client to
+        run.  This simply keeps things clean if the user decides to revert
+        changes.
+    .. warning:: If all deps are not included, it may cause incorrect parsing
+        behavior, due to enable_mod's shortcut for updating the parser's
+        currently defined modules (`.ApacheParser.add_mod`)
+        This would only present a major problem in extremely atypical
+        configs that use ifmod for the missing deps.
+
+    """
+    deps = {
+        "ssl": ["setenvif", "mime"]
+    }
+    return deps.get(mod_name, [])
+
+
+def get_file_path(vhost_path):
+    """Get file path from augeas_vhost_path.
+
+    Takes in Augeas path and returns the file name
+
+    :param str vhost_path: Augeas virtual host path
+
+    :returns: filename of vhost
+    :rtype: str
+
+    """
+    if not vhost_path or not vhost_path.startswith("/files/"):
+        return None
+
+    return _split_aug_path(vhost_path)[0]
+
+
+def get_internal_aug_path(vhost_path):
+    """Get the Augeas path for a vhost with the file path removed.
+
+    :param str vhost_path: Augeas virtual host path
+
+    :returns: Augeas path to vhost relative to the containing file
+    :rtype: str
+
+    """
+    return _split_aug_path(vhost_path)[1]
+
+
+def _split_aug_path(vhost_path):
+    """Splits an Augeas path into a file path and an internal path.
+
+    After removing "/files", this function splits vhost_path into the
+    file path and the remaining Augeas path.
+
+    :param str vhost_path: Augeas virtual host path
+
+    :returns: file path and internal Augeas path
+    :rtype: `tuple` of `str`
+
+    """
+    # Strip off /files
+    file_path = vhost_path[6:]
+    internal_path = []
+
+    # Remove components from the end of file_path until it becomes valid
+    while not os.path.exists(file_path):
+        file_path, _, internal_path_part = file_path.rpartition("/")
+        internal_path.append(internal_path_part)
+
+    return file_path, "/".join(reversed(internal_path))
+
+
+def parse_define_file(filepath, varname):
+    """ Parses Defines from a variable in configuration file
+
+    :param str filepath: Path of file to parse
+    :param str varname: Name of the variable
+
+    :returns: Dict of Define:Value pairs
+    :rtype: `dict`
+
+    """
+    return_vars = {}
+    # Get list of words in the variable
+    a_opts = util.get_var_from_file(varname, filepath).split()
+    for i, v in enumerate(a_opts):
+        # Handle Define statements and make sure it has an argument
+        if v == "-D" and len(a_opts) >= i+2:
+            var_parts = a_opts[i+1].partition("=")
+            return_vars[var_parts[0]] = var_parts[2]
+        elif len(v) > 2 and v.startswith("-D"):
+            # Found var with no whitespace separator
+            var_parts = v[2:].partition("=")
+            return_vars[var_parts[0]] = var_parts[2]
+    return return_vars
+
+
+def unique_id():
+    """ Returns an unique id to be used as a VirtualHost identifier"""
+    return binascii.hexlify(os.urandom(16)).decode("utf-8")
+
+
+def included_in_paths(filepath, paths):
+    """
+    Returns true if the filepath is included in the list of paths
+    that may contain full paths or wildcard paths that need to be
+    expanded.
+
+    :param str filepath: Filepath to check
+    :params list paths: List of paths to check against
+
+    :returns: True if included
+    :rtype: bool
+    """
+
+    return any([fnmatch.fnmatch(filepath, path) for path in paths])
+
+
+def parse_defines(apachectl):
+    """
+    Gets Defines from httpd process and returns a dictionary of
+    the defined variables.
+
+    :param str apachectl: Path to apachectl executable
+
+    :returns: dictionary of defined variables
+    :rtype: dict
+    """
+
+    variables = dict()
+    define_cmd = [apachectl, "-t", "-D",
+                  "DUMP_RUN_CFG"]
+    matches = parse_from_subprocess(define_cmd, r"Define: ([^ \n]*)")
+    try:
+        matches.remove("DUMP_RUN_CFG")
+    except ValueError:
+        return {}
+
+    for match in matches:
+        if match.count("=") > 1:
+            logger.error("Unexpected number of equal signs in "
+                         "runtime config dump.")
+            raise errors.PluginError(
+                "Error parsing Apache runtime variables")
+        parts = match.partition("=")
+        variables[parts[0]] = parts[2]
+
+    return variables
+
+
+def parse_includes(apachectl):
+    """
+    Gets Include directives from httpd process and returns a list of
+    their values.
+
+    :param str apachectl: Path to apachectl executable
+
+    :returns: list of found Include directive values
+    :rtype: list of str
+    """
+
+    inc_cmd = [apachectl, "-t", "-D",
+               "DUMP_INCLUDES"]
+    return parse_from_subprocess(inc_cmd, r"\(.*\) (.*)")
+
+
+def parse_modules(apachectl):
+    """
+    Get loaded modules from httpd process, and return the list
+    of loaded module names.
+
+    :param str apachectl: Path to apachectl executable
+
+    :returns: list of found LoadModule module names
+    :rtype: list of str
+    """
+
+    mod_cmd = [apachectl, "-t", "-D",
+               "DUMP_MODULES"]
+    return parse_from_subprocess(mod_cmd, r"(.*)_module")
+
+
+def parse_from_subprocess(command, regexp):
+    """Get values from stdout of subprocess command
+
+    :param list command: Command to run
+    :param str regexp: Regexp for parsing
+
+    :returns: list parsed from command output
+    :rtype: list
+
+    """
+    stdout = _get_runtime_cfg(command)
+    return re.compile(regexp).findall(stdout)
+
+
+def _get_runtime_cfg(command):
+    """
+    Get runtime configuration info.
+
+    :param command: Command to run
+
+    :returns: stdout from command
+
+    """
+    try:
+        proc = subprocess.Popen(
+            command,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            universal_newlines=True)
+        stdout, stderr = proc.communicate()
+
+    except (OSError, ValueError):
+        logger.error(
+            "Error running command %s for runtime parameters!%s",
+            command, os.linesep)
+        raise errors.MisconfigurationError(
+            "Error accessing loaded Apache parameters: {0}".format(
+                command))
+    # Small errors that do not impede
+    if proc.returncode != 0:
+        logger.warning("Error in checking parameter list: %s", stderr)
+        raise errors.MisconfigurationError(
+            "Apache is unable to check whether or not the module is "
+            "loaded because Apache is misconfigured.")
+
+    return stdout

certbot-apache/certbot_apache/_internal/apacheparser.py

@@ -0,0 +1,169 @@
+""" apacheconfig implementation of the ParserNode interfaces """
+
+from certbot_apache._internal import assertions
+from certbot_apache._internal import interfaces
+from certbot_apache._internal import parsernode_util as util
+
+
+class ApacheParserNode(interfaces.ParserNode):
+    """ apacheconfig implementation of ParserNode interface.
+
+        Expects metadata `ac_ast` to be passed in, where `ac_ast` is the AST provided
+        by parsing the equivalent configuration text using the apacheconfig library.
+    """
+
+    def __init__(self, **kwargs):
+        ancestor, dirty, filepath, metadata = util.parsernode_kwargs(kwargs)  # pylint: disable=unused-variable
+        super(ApacheParserNode, self).__init__(**kwargs)
+        self.ancestor = ancestor
+        self.filepath = filepath
+        self.dirty = dirty
+        self.metadata = metadata
+        self._raw = self.metadata["ac_ast"]
+
+    def save(self, msg): # pragma: no cover
+        pass
+
+    def find_ancestors(self, name):  # pylint: disable=unused-variable
+        """Find ancestor BlockNodes with a given name"""
+        return [ApacheBlockNode(name=assertions.PASS,
+                                parameters=assertions.PASS,
+                                ancestor=self,
+                                filepath=assertions.PASS,
+                                metadata=self.metadata)]
+
+
+class ApacheCommentNode(ApacheParserNode):
+    """ apacheconfig implementation of CommentNode interface """
+
+    def __init__(self, **kwargs):
+        comment, kwargs = util.commentnode_kwargs(kwargs)  # pylint: disable=unused-variable
+        super(ApacheCommentNode, self).__init__(**kwargs)
+        self.comment = comment
+
+    def __eq__(self, other):  # pragma: no cover
+        if isinstance(other, self.__class__):
+            return (self.comment == other.comment and
+                    self.dirty == other.dirty and
+                    self.ancestor == other.ancestor and
+                    self.metadata == other.metadata and
+                    self.filepath == other.filepath)
+        return False
+
+
+class ApacheDirectiveNode(ApacheParserNode):
+    """ apacheconfig implementation of DirectiveNode interface """
+
+    def __init__(self, **kwargs):
+        name, parameters, enabled, kwargs = util.directivenode_kwargs(kwargs)
+        super(ApacheDirectiveNode, self).__init__(**kwargs)
+        self.name = name
+        self.parameters = parameters
+        self.enabled = enabled
+        self.include = None
+
+    def __eq__(self, other):  # pragma: no cover
+        if isinstance(other, self.__class__):
+            return (self.name == other.name and
+                    self.filepath == other.filepath and
+                    self.parameters == other.parameters and
+                    self.enabled == other.enabled and
+                    self.dirty == other.dirty and
+                    self.ancestor == other.ancestor and
+                    self.metadata == other.metadata)
+        return False
+
+    def set_parameters(self, _parameters):
+        """Sets the parameters for DirectiveNode"""
+        return
+
+
+class ApacheBlockNode(ApacheDirectiveNode):
+    """ apacheconfig implementation of BlockNode interface """
+
+    def __init__(self, **kwargs):
+        super(ApacheBlockNode, self).__init__(**kwargs)
+        self.children = ()
+
+    def __eq__(self, other):  # pragma: no cover
+        if isinstance(other, self.__class__):
+            return (self.name == other.name and
+                    self.filepath == other.filepath and
+                    self.parameters == other.parameters and
+                    self.children == other.children and
+                    self.enabled == other.enabled and
+                    self.dirty == other.dirty and
+                    self.ancestor == other.ancestor and
+                    self.metadata == other.metadata)
+        return False
+
+    def add_child_block(self, name, parameters=None, position=None):  # pylint: disable=unused-argument
+        """Adds a new BlockNode to the sequence of children"""
+        new_block = ApacheBlockNode(name=assertions.PASS,
+                                    parameters=assertions.PASS,
+                                    ancestor=self,
+                                    filepath=assertions.PASS,
+                                    metadata=self.metadata)
+        self.children += (new_block,)
+        return new_block
+
+    def add_child_directive(self, name, parameters=None, position=None):  # pylint: disable=unused-argument
+        """Adds a new DirectiveNode to the sequence of children"""
+        new_dir = ApacheDirectiveNode(name=assertions.PASS,
+                                      parameters=assertions.PASS,
+                                      ancestor=self,
+                                      filepath=assertions.PASS,
+                                      metadata=self.metadata)
+        self.children += (new_dir,)
+        return new_dir
+
+    # pylint: disable=unused-argument
+    def add_child_comment(self, comment="", position=None):  # pragma: no cover
+
+        """Adds a new CommentNode to the sequence of children"""
+        new_comment = ApacheCommentNode(comment=assertions.PASS,
+                                        ancestor=self,
+                                        filepath=assertions.PASS,
+                                        metadata=self.metadata)
+        self.children += (new_comment,)
+        return new_comment
+
+    def find_blocks(self, name, exclude=True): # pylint: disable=unused-argument
+        """Recursive search of BlockNodes from the sequence of children"""
+        return [ApacheBlockNode(name=assertions.PASS,
+                                parameters=assertions.PASS,
+                                ancestor=self,
+                                filepath=assertions.PASS,
+                                metadata=self.metadata)]
+
+    def find_directives(self, name, exclude=True): # pylint: disable=unused-argument
+        """Recursive search of DirectiveNodes from the sequence of children"""
+        return [ApacheDirectiveNode(name=assertions.PASS,
+                                    parameters=assertions.PASS,
+                                    ancestor=self,
+                                    filepath=assertions.PASS,
+                                    metadata=self.metadata)]
+
+    def find_comments(self, comment, exact=False): # pylint: disable=unused-argument
+        """Recursive search of DirectiveNodes from the sequence of children"""
+        return [ApacheCommentNode(comment=assertions.PASS,
+                                  ancestor=self,
+                                  filepath=assertions.PASS,
+                                  metadata=self.metadata)]
+
+    def delete_child(self, child):  # pragma: no cover
+        """Deletes a ParserNode from the sequence of children"""
+        return
+
+    def unsaved_files(self):  # pragma: no cover
+        """Returns a list of unsaved filepaths"""
+        return [assertions.PASS]
+
+    def parsed_paths(self):  # pragma: no cover
+        """Returns a list of parsed configuration file paths"""
+        return [assertions.PASS]
+
+
+interfaces.CommentNode.register(ApacheCommentNode)
+interfaces.DirectiveNode.register(ApacheDirectiveNode)
+interfaces.BlockNode.register(ApacheBlockNode)

certbot-apache/certbot_apache/_internal/assertions.py

@@ -0,0 +1,142 @@
+"""Dual parser node assertions"""
+import fnmatch
+
+from certbot_apache._internal import interfaces
+
+
+PASS = "CERTBOT_PASS_ASSERT"
+
+
+def assertEqual(first, second):
+    """ Equality assertion """
+
+    if isinstance(first, interfaces.CommentNode):
+        assertEqualComment(first, second)
+    elif isinstance(first, interfaces.DirectiveNode):
+        assertEqualDirective(first, second)
+
+    # Do an extra interface implementation assertion, as the contents were
+    # already checked for BlockNode in the assertEqualDirective
+    if isinstance(first, interfaces.BlockNode):
+        assert isinstance(second, interfaces.BlockNode)
+
+    # Skip tests if filepath includes the pass value. This is done
+    # because filepath is variable of the base ParserNode interface, and
+    # unless the implementation is actually done, we cannot assume getting
+    # correct results from boolean assertion for dirty
+    if not isPass(first.filepath) and not isPass(second.filepath):
+        assert first.dirty == second.dirty
+        # We might want to disable this later if testing with two separate
+        # (but identical) directory structures.
+        assert first.filepath == second.filepath
+
+def assertEqualComment(first, second): # pragma: no cover
+    """ Equality assertion for CommentNode """
+
+    assert isinstance(first, interfaces.CommentNode)
+    assert isinstance(second, interfaces.CommentNode)
+
+    if not isPass(first.comment) and not isPass(second.comment):  # type: ignore
+        assert first.comment == second.comment  # type: ignore
+
+def _assertEqualDirectiveComponents(first, second): # pragma: no cover
+    """ Handles assertion for instance variables for DirectiveNode and BlockNode"""
+
+    # Enabled value cannot be asserted, because Augeas implementation
+    # is unable to figure that out.
+    # assert first.enabled == second.enabled
+    if not isPass(first.name) and not isPass(second.name):
+        assert first.name == second.name
+
+    if not isPass(first.parameters) and not isPass(second.parameters):
+        assert first.parameters == second.parameters
+
+def assertEqualDirective(first, second):
+    """ Equality assertion for DirectiveNode """
+
+    assert isinstance(first, interfaces.DirectiveNode)
+    assert isinstance(second, interfaces.DirectiveNode)
+    _assertEqualDirectiveComponents(first, second)
+
+def isPass(value): # pragma: no cover
+    """Checks if the value is set to PASS"""
+    if isinstance(value, bool):
+        return True
+    return PASS in value
+
+def isPassDirective(block):
+    """ Checks if BlockNode or DirectiveNode should pass the assertion """
+
+    if isPass(block.name):
+        return True
+    if isPass(block.parameters): # pragma: no cover
+        return True
+    if isPass(block.filepath): # pragma: no cover
+        return True
+    return False
+
+def isPassComment(comment):
+    """ Checks if CommentNode should pass the assertion """
+
+    if isPass(comment.comment):
+        return True
+    if isPass(comment.filepath): # pragma: no cover
+        return True
+    return False
+
+def isPassNodeList(nodelist): # pragma: no cover
+    """ Checks if a ParserNode in the nodelist should pass the assertion,
+    this function is used for results of find_* methods. Unimplemented find_*
+    methods should return a sequence containing a single ParserNode instance
+    with assertion pass string."""
+
+    try:
+        node = nodelist[0]
+    except IndexError:
+        node = None
+
+    if not node: # pragma: no cover
+        return False
+
+    if isinstance(node, interfaces.DirectiveNode):
+        return isPassDirective(node)
+    return isPassComment(node)
+
+def assertEqualSimple(first, second):
+    """ Simple assertion """
+    if not isPass(first) and not isPass(second):
+        assert first == second
+
+def isEqualVirtualHost(first, second):
+    """
+    Checks that two VirtualHost objects are similar. There are some built
+    in differences with the implementations: VirtualHost created by ParserNode
+    implementation doesn't have "path" defined, as it was used for Augeas path
+    and that cannot obviously be used in the future. Similarly the legacy
+    version lacks "node" variable, that has a reference to the BlockNode for the
+    VirtualHost.
+    """
+    return (
+        first.name == second.name and
+        first.aliases == second.aliases and
+        first.filep == second.filep and
+        first.addrs == second.addrs and
+        first.ssl == second.ssl and
+        first.enabled == second.enabled and
+        first.modmacro == second.modmacro and
+        first.ancestor == second.ancestor
+    )
+
+def assertEqualPathsList(first, second):  # pragma: no cover
+    """
+    Checks that the two lists of file paths match. This assertion allows for wildcard
+    paths.
+    """
+    if any([isPass(path) for path in first]):
+        return
+    if any([isPass(path) for path in second]):
+        return
+    for fpath in first:
+        assert any([fnmatch.fnmatch(fpath, spath) for spath in second])
+    for spath in second:
+        assert any([fnmatch.fnmatch(fpath, spath) for fpath in first])

certbot-apache/certbot_apache/_internal/augeasparser.py

@@ -0,0 +1,538 @@
+"""
+Augeas implementation of the ParserNode interfaces.
+
+Augeas works internally by using XPATH notation. The following is a short example
+of how this all works internally, to better understand what's going on under the
+hood.
+
+A configuration file /etc/apache2/apache2.conf with the following content:
+
+    # First comment line
+    # Second comment line
+    WhateverDirective whatevervalue
+    <ABlock>
+        DirectiveInABlock dirvalue
+    </ABlock>
+    SomeDirective somedirectivevalue
+    <ABlock>
+        AnotherDirectiveInABlock dirvalue
+    </ABlock>
+    # Yet another comment
+
+
+Translates over to Augeas path notation (of immediate children), when calling
+for example: aug.match("/files/etc/apache2/apache2.conf/*")
+
+[
+    "/files/etc/apache2/apache2.conf/#comment[1]",
+    "/files/etc/apache2/apache2.conf/#comment[2]",
+    "/files/etc/apache2/apache2.conf/directive[1]",
+    "/files/etc/apache2/apache2.conf/ABlock[1]",
+    "/files/etc/apache2/apache2.conf/directive[2]",
+    "/files/etc/apache2/apache2.conf/ABlock[2]",
+    "/files/etc/apache2/apache2.conf/#comment[3]"
+]
+
+Regardless of directives name, its key in the Augeas tree is always "directive",
+with index where needed of course. Comments work similarly, while blocks
+have their own key in the Augeas XPATH notation.
+
+It's important to note that all of the unique keys have their own indices.
+
+Augeas paths are case sensitive, while Apache configuration is case insensitive.
+It looks like this:
+
+    <block>
+        directive value
+    </block>
+    <Block>
+        Directive Value
+    </Block>
+    <block>
+        directive value
+    </block>
+    <bLoCk>
+        DiReCtiVe VaLuE
+    </bLoCk>
+
+Translates over to:
+
+[
+    "/files/etc/apache2/apache2.conf/block[1]",
+    "/files/etc/apache2/apache2.conf/Block[1]",
+    "/files/etc/apache2/apache2.conf/block[2]",
+    "/files/etc/apache2/apache2.conf/bLoCk[1]",
+]
+"""
+from acme.magic_typing import Set  # pylint: disable=unused-import, no-name-in-module
+from certbot import errors
+from certbot.compat import os
+
+from certbot_apache._internal import apache_util
+from certbot_apache._internal import assertions
+from certbot_apache._internal import interfaces
+from certbot_apache._internal import parser
+from certbot_apache._internal import parsernode_util as util
+
+
+class AugeasParserNode(interfaces.ParserNode):
+    """ Augeas implementation of ParserNode interface """
+
+    def __init__(self, **kwargs):
+        ancestor, dirty, filepath, metadata = util.parsernode_kwargs(kwargs)  # pylint: disable=unused-variable
+        super(AugeasParserNode, self).__init__(**kwargs)
+        self.ancestor = ancestor
+        self.filepath = filepath
+        self.dirty = dirty
+        self.metadata = metadata
+        self.parser = self.metadata.get("augeasparser")
+        try:
+            if self.metadata["augeaspath"].endswith("/"):
+                raise errors.PluginError(
+                    "Augeas path: {} has a trailing slash".format(
+                        self.metadata["augeaspath"]
+                    )
+                )
+        except KeyError:
+            raise errors.PluginError("Augeas path is required")
+
+    def save(self, msg):
+        self.parser.save(msg)
+
+    def find_ancestors(self, name):
+        """
+        Searches for ancestor BlockNodes with a given name.
+
+        :param str name: Name of the BlockNode parent to search for
+
+        :returns: List of matching ancestor nodes.
+        :rtype: list of AugeasBlockNode
+        """
+
+        ancestors = []
+
+        parent = self.metadata["augeaspath"]
+        while True:
+            # Get the path of ancestor node
+            parent = parent.rpartition("/")[0]
+            # Root of the tree
+            if not parent or parent == "/files":
+                break
+            anc = self._create_blocknode(parent)
+            if anc.name.lower() == name.lower():
+                ancestors.append(anc)
+
+        return ancestors
+
+    def _create_blocknode(self, path):
+        """
+        Helper function to create a BlockNode from Augeas path. This is used by
+        AugeasParserNode.find_ancestors and AugeasBlockNode.
+        and AugeasBlockNode.find_blocks
+
+        """
+
+        name = self._aug_get_name(path)
+        metadata = {"augeasparser": self.parser, "augeaspath": path}
+
+        # Check if the file was included from the root config or initial state
+        enabled = self.parser.parsed_in_original(
+            apache_util.get_file_path(path)
+        )
+
+        return AugeasBlockNode(name=name,
+                               enabled=enabled,
+                               ancestor=assertions.PASS,
+                               filepath=apache_util.get_file_path(path),
+                               metadata=metadata)
+
+    def _aug_get_name(self, path):
+        """
+        Helper function to get name of a configuration block or variable from path.
+        """
+
+        # Remove the ending slash if any
+        if path[-1] == "/":  # pragma: no cover
+            path = path[:-1]
+
+        # Get the block name
+        name = path.split("/")[-1]
+
+        # remove [...], it's not allowed in Apache configuration and is used
+        # for indexing within Augeas
+        name = name.split("[")[0]
+        return name
+
+
+class AugeasCommentNode(AugeasParserNode):
+    """ Augeas implementation of CommentNode interface """
+
+    def __init__(self, **kwargs):
+        comment, kwargs = util.commentnode_kwargs(kwargs)  # pylint: disable=unused-variable
+        super(AugeasCommentNode, self).__init__(**kwargs)
+        # self.comment = comment
+        self.comment = comment
+
+    def __eq__(self, other):
+        if isinstance(other, self.__class__):
+            return (self.comment == other.comment and
+                    self.filepath == other.filepath and
+                    self.dirty == other.dirty and
+                    self.ancestor == other.ancestor and
+                    self.metadata == other.metadata)
+        return False
+
+
+class AugeasDirectiveNode(AugeasParserNode):
+    """ Augeas implementation of DirectiveNode interface """
+
+    def __init__(self, **kwargs):
+        name, parameters, enabled, kwargs = util.directivenode_kwargs(kwargs)
+        super(AugeasDirectiveNode, self).__init__(**kwargs)
+        self.name = name
+        self.enabled = enabled
+        if parameters:
+            self.set_parameters(parameters)
+
+    def __eq__(self, other):
+        if isinstance(other, self.__class__):
+            return (self.name == other.name and
+                    self.filepath == other.filepath and
+                    self.parameters == other.parameters and
+                    self.enabled == other.enabled and
+                    self.dirty == other.dirty and
+                    self.ancestor == other.ancestor and
+                    self.metadata == other.metadata)
+        return False
+
+    def set_parameters(self, parameters):
+        """
+        Sets parameters of a DirectiveNode or BlockNode object.
+
+        :param list parameters: List of all parameters for the node to set.
+        """
+        orig_params = self._aug_get_params(self.metadata["augeaspath"])
+
+        # Clear out old parameters
+        for _ in orig_params:
+            # When the first parameter is removed, the indices get updated
+            param_path = "{}/arg[1]".format(self.metadata["augeaspath"])
+            self.parser.aug.remove(param_path)
+        # Insert new ones
+        for pi, param in enumerate(parameters):
+            param_path = "{}/arg[{}]".format(self.metadata["augeaspath"], pi+1)
+            self.parser.aug.set(param_path, param)
+
+    @property
+    def parameters(self):
+        """
+        Fetches the parameters from Augeas tree, ensuring that the sequence always
+        represents the current state
+
+        :returns: Tuple of parameters for this DirectiveNode
+        :rtype: tuple:
+        """
+        return tuple(self._aug_get_params(self.metadata["augeaspath"]))
+
+    def _aug_get_params(self, path):
+        """Helper function to get parameters for DirectiveNodes and BlockNodes"""
+
+        arg_paths = self.parser.aug.match(path + "/arg")
+        return [self.parser.get_arg(apath) for apath in arg_paths]
+
+
+class AugeasBlockNode(AugeasDirectiveNode):
+    """ Augeas implementation of BlockNode interface """
+
+    def __init__(self, **kwargs):
+        super(AugeasBlockNode, self).__init__(**kwargs)
+        self.children = ()
+
+    def __eq__(self, other):
+        if isinstance(other, self.__class__):
+            return (self.name == other.name and
+                    self.filepath == other.filepath and
+                    self.parameters == other.parameters and
+                    self.children == other.children and
+                    self.enabled == other.enabled and
+                    self.dirty == other.dirty and
+                    self.ancestor == other.ancestor and
+                    self.metadata == other.metadata)
+        return False
+
+    # pylint: disable=unused-argument
+    def add_child_block(self, name, parameters=None, position=None):  # pragma: no cover
+        """Adds a new BlockNode to the sequence of children"""
+
+        insertpath, realpath, before = self._aug_resolve_child_position(
+            name,
+            position
+        )
+        new_metadata = {"augeasparser": self.parser, "augeaspath": realpath}
+
+        # Create the new block
+        self.parser.aug.insert(insertpath, name, before)
+        # Check if the file was included from the root config or initial state
+        enabled = self.parser.parsed_in_original(
+            apache_util.get_file_path(realpath)
+        )
+
+        # Parameters will be set at the initialization of the new object
+        new_block = AugeasBlockNode(name=name,
+                                    parameters=parameters,
+                                    enabled=enabled,
+                                    ancestor=assertions.PASS,
+                                    filepath=apache_util.get_file_path(realpath),
+                                    metadata=new_metadata)
+        return new_block
+
+    # pylint: disable=unused-argument
+    def add_child_directive(self, name, parameters=None, position=None):  # pragma: no cover
+        """Adds a new DirectiveNode to the sequence of children"""
+
+        if not parameters:
+            raise errors.PluginError("Directive requires parameters and none were set.")
+
+        insertpath, realpath, before = self._aug_resolve_child_position(
+            "directive",
+            position
+        )
+        new_metadata = {"augeasparser": self.parser, "augeaspath": realpath}
+
+        # Create the new directive
+        self.parser.aug.insert(insertpath, "directive", before)
+        # Set the directive key
+        self.parser.aug.set(realpath, name)
+        # Check if the file was included from the root config or initial state
+        enabled = self.parser.parsed_in_original(
+            apache_util.get_file_path(realpath)
+        )
+
+        new_dir = AugeasDirectiveNode(name=name,
+                                      parameters=parameters,
+                                      enabled=enabled,
+                                      ancestor=assertions.PASS,
+                                      filepath=apache_util.get_file_path(realpath),
+                                      metadata=new_metadata)
+        return new_dir
+
+    def add_child_comment(self, comment="", position=None):
+        """Adds a new CommentNode to the sequence of children"""
+
+        insertpath, realpath, before = self._aug_resolve_child_position(
+            "#comment",
+            position
+        )
+        new_metadata = {"augeasparser": self.parser, "augeaspath": realpath}
+
+        # Create the new comment
+        self.parser.aug.insert(insertpath, "#comment", before)
+        # Set the comment content
+        self.parser.aug.set(realpath, comment)
+
+        new_comment = AugeasCommentNode(comment=comment,
+                                        ancestor=assertions.PASS,
+                                        filepath=apache_util.get_file_path(realpath),
+                                        metadata=new_metadata)
+        return new_comment
+
+    def find_blocks(self, name, exclude=True):
+        """Recursive search of BlockNodes from the sequence of children"""
+
+        nodes = list()
+        paths = self._aug_find_blocks(name)
+        if exclude:
+            paths = self.parser.exclude_dirs(paths)
+        for path in paths:
+            nodes.append(self._create_blocknode(path))
+
+        return nodes
+
+    def find_directives(self, name, exclude=True):
+        """Recursive search of DirectiveNodes from the sequence of children"""
+
+        nodes = list()
+        ownpath = self.metadata.get("augeaspath")
+
+        directives = self.parser.find_dir(name, start=ownpath, exclude=exclude)
+        already_parsed = set()  # type: Set[str]
+        for directive in directives:
+            # Remove the /arg part from the Augeas path
+            directive = directive.partition("/arg")[0]
+            # find_dir returns an object for each _parameter_ of a directive
+            # so we need to filter out duplicates.
+            if directive not in already_parsed:
+                nodes.append(self._create_directivenode(directive))
+                already_parsed.add(directive)
+
+        return nodes
+
+    def find_comments(self, comment):
+        """
+        Recursive search of DirectiveNodes from the sequence of children.
+
+        :param str comment: Comment content to search for.
+        """
+
+        nodes = list()
+        ownpath = self.metadata.get("augeaspath")
+
+        comments = self.parser.find_comments(comment, start=ownpath)
+        for com in comments:
+            nodes.append(self._create_commentnode(com))
+
+        return nodes
+
+    def delete_child(self, child):
+        """
+        Deletes a ParserNode from the sequence of children, and raises an
+        exception if it's unable to do so.
+        :param AugeasParserNode: child: A node to delete.
+        """
+        if not self.parser.aug.remove(child.metadata["augeaspath"]):
+
+            raise errors.PluginError(
+                ("Could not delete child node, the Augeas path: {} doesn't " +
+                 "seem to exist.").format(child.metadata["augeaspath"])
+            )
+
+    def unsaved_files(self):
+        """Returns a list of unsaved filepaths"""
+        return self.parser.unsaved_files()
+
+    def parsed_paths(self):
+        """
+        Returns a list of file paths that have currently been parsed into the parser
+        tree. The returned list may include paths with wildcard characters, for
+        example: ['/etc/apache2/conf.d/*.load']
+
+        This is typically called on the root node of the ParserNode tree.
+
+        :returns: list of file paths of files that have been parsed
+        """
+
+        res_paths = []
+
+        paths = self.parser.existing_paths
+        for directory in paths:
+            for filename in paths[directory]:
+                res_paths.append(os.path.join(directory, filename))
+
+        return res_paths
+
+    def _create_commentnode(self, path):
+        """Helper function to create a CommentNode from Augeas path"""
+
+        comment = self.parser.aug.get(path)
+        metadata = {"augeasparser": self.parser, "augeaspath": path}
+
+        # Because of the dynamic nature of AugeasParser and the fact that we're
+        # not populating the complete node tree, the ancestor has a dummy value
+        return AugeasCommentNode(comment=comment,
+                                 ancestor=assertions.PASS,
+                                 filepath=apache_util.get_file_path(path),
+                                 metadata=metadata)
+
+    def _create_directivenode(self, path):
+        """Helper function to create a DirectiveNode from Augeas path"""
+
+        name = self.parser.get_arg(path)
+        metadata = {"augeasparser": self.parser, "augeaspath": path}
+
+        # Check if the file was included from the root config or initial state
+        enabled = self.parser.parsed_in_original(
+            apache_util.get_file_path(path)
+        )
+        return AugeasDirectiveNode(name=name,
+                                   ancestor=assertions.PASS,
+                                   enabled=enabled,
+                                   filepath=apache_util.get_file_path(path),
+                                   metadata=metadata)
+
+    def _aug_find_blocks(self, name):
+        """Helper function to perform a search to Augeas DOM tree to search
+        configuration blocks with a given name"""
+
+        # The code here is modified from configurator.get_virtual_hosts()
+        blk_paths = set()
+        for vhost_path in list(self.parser.parser_paths):
+            paths = self.parser.aug.match(
+                ("/files%s//*[label()=~regexp('%s')]" %
+                    (vhost_path, parser.case_i(name))))
+            blk_paths.update([path for path in paths if
+                              name.lower() in os.path.basename(path).lower()])
+        return blk_paths
+
+    def _aug_resolve_child_position(self, name, position):
+        """
+        Helper function that iterates through the immediate children and figures
+        out the insertion path for a new AugeasParserNode.
+
+        Augeas also generalizes indices for directives and comments, simply by
+        using "directive" or "comment" respectively as their names.
+
+        This function iterates over the existing children of the AugeasBlockNode,
+        returning their insertion path, resulting Augeas path and if the new node
+        should be inserted before or after the returned insertion path.
+
+        Note: while Apache is case insensitive, Augeas is not, and blocks like
+        Nameofablock and NameOfABlock have different indices.
+
+        :param str name: Name of the AugeasBlockNode to insert, "directive" for
+            AugeasDirectiveNode or "comment" for AugeasCommentNode
+        :param int position: The position to insert the child AugeasParserNode to
+
+        :returns: Tuple of insert path, resulting path and a boolean if the new
+            node should be inserted before it.
+        :rtype: tuple of str, str, bool
+        """
+
+        # Default to appending
+        before = False
+
+        all_children = self.parser.aug.match("{}/*".format(
+            self.metadata["augeaspath"])
+        )
+
+        # Calculate resulting_path
+        # Augeas indices start at 1. We use counter to calculate the index to
+        # be used in resulting_path.
+        counter = 1
+        for i, child in enumerate(all_children):
+            if position is not None and i >= position:
+                # We're not going to insert the new node to an index after this
+                break
+            childname = self._aug_get_name(child)
+            if name == childname:
+                counter += 1
+
+        resulting_path = "{}/{}[{}]".format(
+            self.metadata["augeaspath"],
+            name,
+            counter
+        )
+
+        # Form the correct insert_path
+        # Inserting the only child and appending as the last child work
+        # similarly in Augeas.
+        append = not all_children or position is None or position >= len(all_children)
+        if append:
+            insert_path = "{}/*[last()]".format(
+                self.metadata["augeaspath"]
+            )
+        elif position == 0:
+            # Insert as the first child, before the current first one.
+            insert_path = all_children[0]
+            before = True
+        else:
+            insert_path = "{}/*[{}]".format(
+                self.metadata["augeaspath"],
+                position
+            )
+
+        return (insert_path, resulting_path, before)
+
+
+interfaces.CommentNode.register(AugeasCommentNode)
+interfaces.DirectiveNode.register(AugeasDirectiveNode)
+interfaces.BlockNode.register(AugeasBlockNode)

certbot-apache/certbot_apache/_internal/configurator.py

@@ -1,5 +1,6 @@
 """Apache Configurator."""
 # pylint: disable=too-many-lines
+from collections import defaultdict
 import copy
 import fnmatch
 import logging
@@ -7,33 +8,34 @@ import re
 import socket
 import time
 
-from collections import defaultdict
-
 import pkg_resources
 import six
-
 import zope.component
 import zope.interface
 
 from acme import challenges
-from acme.magic_typing import DefaultDict, Dict, List, Set, Union  # pylint: disable=unused-import, no-name-in-module
-
+from acme.magic_typing import DefaultDict
+from acme.magic_typing import Dict
+from acme.magic_typing import List
+from acme.magic_typing import Set
+from acme.magic_typing import Union
 from certbot import errors
 from certbot import interfaces
 from certbot import util
-
 from certbot.achallenges import KeyAuthorizationAnnotatedChallenge  # pylint: disable=unused-import
+from certbot.compat import filesystem
 from certbot.compat import os
 from certbot.plugins import common
-from certbot.plugins.util import path_surgery
 from certbot.plugins.enhancements import AutoHSTSEnhancement
-
-from certbot_apache import apache_util
-from certbot_apache import constants
-from certbot_apache import display_ops
-from certbot_apache import http_01
-from certbot_apache import obj
-from certbot_apache import parser
+from certbot.plugins.util import path_surgery
+from certbot_apache._internal import apache_util
+from certbot_apache._internal import assertions
+from certbot_apache._internal import constants
+from certbot_apache._internal import display_ops
+from certbot_apache._internal import dualparser
+from certbot_apache._internal import http_01
+from certbot_apache._internal import obj
+from certbot_apache._internal import parser
 
 logger = logging.getLogger(__name__)
 
@@ -70,18 +72,17 @@ logger = logging.getLogger(__name__)
 @zope.interface.implementer(interfaces.IAuthenticator, interfaces.IInstaller)
 @zope.interface.provider(interfaces.IPluginFactory)
 class ApacheConfigurator(common.Installer):
-    # pylint: disable=too-many-instance-attributes,too-many-public-methods
     """Apache configurator.
 
     :ivar config: Configuration.
     :type config: :class:`~certbot.interfaces.IConfig`
 
     :ivar parser: Handles low level parsing
-    :type parser: :class:`~certbot_apache.parser`
+    :type parser: :class:`~certbot_apache._internal.parser`
 
     :ivar tup version: version of Apache
     :ivar list vhosts: All vhosts found in the configuration
-        (:class:`list` of :class:`~certbot_apache.obj.VirtualHost`)
+        (:class:`list` of :class:`~certbot_apache._internal.obj.VirtualHost`)
 
     :ivar dict assoc: Mapping between domains and vhosts
 
@@ -110,7 +111,7 @@ class ApacheConfigurator(common.Installer):
         handle_sites=False,
         challenge_location="/etc/apache2",
         MOD_SSL_CONF_SRC=pkg_resources.resource_filename(
-            "certbot_apache", "options-ssl-apache.conf")
+            "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf"))
     )
 
     def option(self, key):
@@ -173,8 +174,6 @@ class ApacheConfigurator(common.Installer):
                  "(Only Ubuntu/Debian currently)")
         add("ctl", default=DEFAULTS["ctl"],
             help="Full path to Apache control script")
-        util.add_deprecated_argument(
-            add, argument_name="init-script", nargs=1)
 
     def __init__(self, *args, **kwargs):
         """Initialize an Apache Configurator.
@@ -184,6 +183,7 @@ class ApacheConfigurator(common.Installer):
 
         """
         version = kwargs.pop("version", None)
+        use_parsernode = kwargs.pop("use_parsernode", False)
         super(ApacheConfigurator, self).__init__(*args, **kwargs)
 
         # Add name_server association dict
@@ -199,10 +199,15 @@ class ApacheConfigurator(common.Installer):
         self._autohsts = {}  # type: Dict[str, Dict[str, Union[int, float]]]
         # Reverter save notes
         self.save_notes = ""
-
+        # Should we use ParserNode implementation instead of the old behavior
+        self.USE_PARSERNODE = use_parsernode
+        # Saves the list of file paths that were parsed initially, and
+        # not added to parser tree by self.conf("vhost-root") for example.
+        self.parsed_paths = []  # type: List[str]
         # These will be set in the prepare function
         self._prepared = False
         self.parser = None
+        self.parser_root = None
         self.version = version
         self.vhosts = None
         self.options = copy.deepcopy(self.OS_DEFAULTS)
@@ -252,6 +257,14 @@ class ApacheConfigurator(common.Installer):
         # Perform the actual Augeas initialization to be able to react
         self.parser = self.get_parser()
 
+        # Set up ParserNode root
+        pn_meta = {"augeasparser": self.parser,
+                   "augeaspath": self.parser.get_root_augpath(),
+                   "ac_ast": None}
+        if self.USE_PARSERNODE:
+            self.parser_root = self.get_parsernode_root(pn_meta)
+            self.parsed_paths = self.parser_root.parsed_paths()
+
         # Check for errors in parsing files with Augeas
         self.parser.check_parsing_errors("httpd.aug")
 
@@ -347,6 +360,22 @@ class ApacheConfigurator(common.Installer):
             self.option("server_root"), self.conf("vhost-root"),
             self.version, configurator=self)
 
+    def get_parsernode_root(self, metadata):
+        """Initializes the ParserNode parser root instance."""
+
+        apache_vars = dict()
+        apache_vars["defines"] = apache_util.parse_defines(self.option("ctl"))
+        apache_vars["includes"] = apache_util.parse_includes(self.option("ctl"))
+        apache_vars["modules"] = apache_util.parse_modules(self.option("ctl"))
+        metadata["apache_vars"] = apache_vars
+
+        return dualparser.DualBlockNode(
+            name=assertions.PASS,
+            ancestor=None,
+            filepath=self.parser.loc["root"],
+            metadata=metadata
+        )
+
     def _wildcard_domain(self, domain):
         """
         Checks if domain is a wildcard domain
@@ -393,7 +422,7 @@ class ApacheConfigurator(common.Installer):
             counterpart, should one get created
 
         :returns: List of VirtualHosts or None
-        :rtype: `list` of :class:`~certbot_apache.obj.VirtualHost`
+        :rtype: `list` of :class:`~certbot_apache._internal.obj.VirtualHost`
         """
 
         if self._wildcard_domain(domain):
@@ -452,7 +481,7 @@ class ApacheConfigurator(common.Installer):
                     filtered_vhosts[name] = vhost
 
         # Only unique VHost objects
-        dialog_input = set([vhost for vhost in filtered_vhosts.values()])
+        dialog_input = set(filtered_vhosts.values())
 
         # Ask the user which of names to enable, expect list of names back
         dialog_output = display_ops.select_vhost_multiple(list(dialog_input))
@@ -568,7 +597,7 @@ class ApacheConfigurator(common.Installer):
             counterpart, should one get created
 
         :returns: vhost associated with name
-        :rtype: :class:`~certbot_apache.obj.VirtualHost`
+        :rtype: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :raises .errors.PluginError: If no vhost is available or chosen
 
@@ -603,9 +632,9 @@ class ApacheConfigurator(common.Installer):
                 "in the Apache config.",
                 target_name)
             raise errors.PluginError("No vhost selected")
-        elif temp:
+        if temp:
             return vhost
-        elif not vhost.ssl:
+        if not vhost.ssl:
             addrs = self._get_proposed_addrs(vhost, "443")
             # TODO: Conflicts is too conservative
             if not any(vhost.enabled and vhost.conflicts(addrs) for
@@ -671,7 +700,7 @@ class ApacheConfigurator(common.Installer):
 
         :param str target_name: domain handled by the desired vhost
         :param vhosts: vhosts to consider
-        :type vhosts: `collections.Iterable` of :class:`~certbot_apache.obj.VirtualHost`
+        :type vhosts: `collections.Iterable` of :class:`~certbot_apache._internal.obj.VirtualHost`
         :param bool filter_defaults: whether a vhost with a _default_
             addr is acceptable
 
@@ -763,7 +792,7 @@ class ApacheConfigurator(common.Installer):
 
         return util.get_filtered_names(all_names)
 
-    def get_name_from_ip(self, addr):  # pylint: disable=no-self-use
+    def get_name_from_ip(self, addr):
         """Returns a reverse dns name if available.
 
         :param addr: IP Address
@@ -813,7 +842,7 @@ class ApacheConfigurator(common.Installer):
         """Helper function for get_virtual_hosts().
 
         :param host: In progress vhost whose names will be added
-        :type host: :class:`~certbot_apache.obj.VirtualHost`
+        :type host: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         """
 
@@ -832,7 +861,7 @@ class ApacheConfigurator(common.Installer):
         :param str path: Augeas path to virtual host
 
         :returns: newly created vhost
-        :rtype: :class:`~certbot_apache.obj.VirtualHost`
+        :rtype: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         """
         addrs = set()
@@ -871,9 +900,32 @@ class ApacheConfigurator(common.Installer):
         return vhost
 
     def get_virtual_hosts(self):
+        """
+        Temporary wrapper for legacy and ParserNode version for
+        get_virtual_hosts. This should be replaced with the ParserNode
+        implementation when ready.
+        """
+
+        v1_vhosts = self.get_virtual_hosts_v1()
+        if self.USE_PARSERNODE:
+            v2_vhosts = self.get_virtual_hosts_v2()
+
+            for v1_vh in v1_vhosts:
+                found = False
+                for v2_vh in v2_vhosts:
+                    if assertions.isEqualVirtualHost(v1_vh, v2_vh):
+                        found = True
+                        break
+                if not found:
+                    raise AssertionError("Equivalent for {} was not found".format(v1_vh.path))
+
+            return v2_vhosts
+        return v1_vhosts
+
+    def get_virtual_hosts_v1(self):
         """Returns list of virtual hosts found in the Apache configuration.
 
-        :returns: List of :class:`~certbot_apache.obj.VirtualHost`
+        :returns: List of :class:`~certbot_apache._internal.obj.VirtualHost`
             objects found in configuration
         :rtype: list
 
@@ -895,7 +947,7 @@ class ApacheConfigurator(common.Installer):
                 if not new_vhost:
                     continue
                 internal_path = apache_util.get_internal_aug_path(new_vhost.path)
-                realpath = os.path.realpath(new_vhost.filep)
+                realpath = filesystem.realpath(new_vhost.filep)
                 if realpath not in file_paths:
                     file_paths[realpath] = new_vhost.filep
                     internal_paths[realpath].add(internal_path)
@@ -923,6 +975,80 @@ class ApacheConfigurator(common.Installer):
                     vhs.append(new_vhost)
         return vhs
 
+    def get_virtual_hosts_v2(self):
+        """Returns list of virtual hosts found in the Apache configuration using
+        ParserNode interface.
+        :returns: List of :class:`~certbot_apache.obj.VirtualHost`
+            objects found in configuration
+        :rtype: list
+        """
+
+        vhs = []
+        vhosts = self.parser_root.find_blocks("VirtualHost", exclude=False)
+        for vhblock in vhosts:
+            vhs.append(self._create_vhost_v2(vhblock))
+        return vhs
+
+    def _create_vhost_v2(self, node):
+        """Used by get_virtual_hosts_v2 to create vhost objects using ParserNode
+        interfaces.
+        :param interfaces.BlockNode node: The BlockNode object of VirtualHost block
+        :returns: newly created vhost
+        :rtype: :class:`~certbot_apache.obj.VirtualHost`
+        """
+        addrs = set()
+        for param in node.parameters:
+            addrs.add(obj.Addr.fromstring(param))
+
+        is_ssl = False
+        # Exclusion to match the behavior in get_virtual_hosts_v2
+        sslengine = node.find_directives("SSLEngine", exclude=False)
+        if sslengine:
+            for directive in sslengine:
+                if directive.parameters[0].lower() == "on":
+                    is_ssl = True
+                    break
+
+        # "SSLEngine on" might be set outside of <VirtualHost>
+        # Treat vhosts with port 443 as ssl vhosts
+        for addr in addrs:
+            if addr.get_port() == "443":
+                is_ssl = True
+
+        enabled = apache_util.included_in_paths(node.filepath, self.parsed_paths)
+
+        macro = False
+        # Check if the VirtualHost is contained in a mod_macro block
+        if node.find_ancestors("Macro"):
+            macro = True
+        vhost = obj.VirtualHost(
+            node.filepath, None, addrs, is_ssl, enabled, modmacro=macro, node=node
+        )
+        self._populate_vhost_names_v2(vhost)
+        return vhost
+
+    def _populate_vhost_names_v2(self, vhost):
+        """Helper function that populates the VirtualHost names.
+        :param host: In progress vhost whose names will be added
+        :type host: :class:`~certbot_apache.obj.VirtualHost`
+        """
+
+        servername_match = vhost.node.find_directives("ServerName",
+                                                      exclude=False)
+        serveralias_match = vhost.node.find_directives("ServerAlias",
+                                                       exclude=False)
+
+        servername = None
+        if servername_match:
+            servername = servername_match[-1].parameters[-1]
+
+        if not vhost.modmacro:
+            for alias in serveralias_match:
+                for serveralias in alias.parameters:
+                    vhost.aliases.add(serveralias)
+            vhost.name = servername
+
+
     def is_name_vhost(self, target_addr):
         """Returns if vhost is a name based vhost
 
@@ -930,7 +1056,7 @@ class ApacheConfigurator(common.Installer):
         now NameVirtualHosts. If version is earlier than 2.4, check if addr
         has a NameVirtualHost directive in the Apache config
 
-        :param certbot_apache.obj.Addr target_addr: vhost address
+        :param certbot_apache._internal.obj.Addr target_addr: vhost address
 
         :returns: Success
         :rtype: bool
@@ -948,19 +1074,18 @@ class ApacheConfigurator(common.Installer):
         """Adds NameVirtualHost directive for given address.
 
         :param addr: Address that will be added as NameVirtualHost directive
-        :type addr: :class:`~certbot_apache.obj.Addr`
+        :type addr: :class:`~certbot_apache._internal.obj.Addr`
 
         """
 
         loc = parser.get_aug_path(self.parser.loc["name"])
         if addr.get_port() == "443":
-            path = self.parser.add_dir_to_ifmodssl(
+            self.parser.add_dir_to_ifmodssl(
                 loc, "NameVirtualHost", [str(addr)])
         else:
-            path = self.parser.add_dir(loc, "NameVirtualHost", [str(addr)])
+            self.parser.add_dir(loc, "NameVirtualHost", [str(addr)])
 
-        msg = ("Setting %s to be NameBasedVirtualHost\n"
-               "\tDirective added to %s\n" % (addr, path))
+        msg = "Setting {0} to be NameBasedVirtualHost\n".format(addr)
         logger.debug(msg)
         self.save_notes += msg
 
@@ -1117,7 +1242,7 @@ class ApacheConfigurator(common.Installer):
             if "ssl_module" not in self.parser.modules:
                 self.enable_mod("ssl", temp=temp)
 
-    def make_vhost_ssl(self, nonssl_vhost):  # pylint: disable=too-many-locals
+    def make_vhost_ssl(self, nonssl_vhost):
         """Makes an ssl_vhost version of a nonssl_vhost.
 
         Duplicates vhost and adds default ssl options
@@ -1127,10 +1252,10 @@ class ApacheConfigurator(common.Installer):
         .. note:: This function saves the configuration
 
         :param nonssl_vhost: Valid VH that doesn't have SSLEngine on
-        :type nonssl_vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type nonssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :returns: SSL vhost
-        :rtype: :class:`~certbot_apache.obj.VirtualHost`
+        :rtype: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :raises .errors.PluginError: If more than one virtual host is in
             the file or if plugin is unable to write/read vhost files.
@@ -1221,11 +1346,11 @@ class ApacheConfigurator(common.Installer):
         """
 
         if self.conf("vhost-root") and os.path.exists(self.conf("vhost-root")):
-            fp = os.path.join(os.path.realpath(self.option("vhost_root")),
+            fp = os.path.join(filesystem.realpath(self.option("vhost_root")),
                               os.path.basename(non_ssl_vh_fp))
         else:
             # Use non-ssl filepath
-            fp = os.path.realpath(non_ssl_vh_fp)
+            fp = filesystem.realpath(non_ssl_vh_fp)
 
         if fp.endswith(".conf"):
             return fp[:-(len(".conf"))] + self.option("le_vhost_ext")
@@ -1368,12 +1493,9 @@ class ApacheConfigurator(common.Installer):
                         result.append(comment)
                         sift = True
 
-                    result.append('\n'.join(
-                        ['# ' + l for l in chunk]))
-                    continue
+                    result.append('\n'.join(['# ' + l for l in chunk]))
                 else:
                     result.append('\n'.join(chunk))
-                    continue
         return result, sift
 
     def _get_vhost_block(self, vhost):
@@ -1501,7 +1623,7 @@ class ApacheConfigurator(common.Installer):
         https://httpd.apache.org/docs/2.2/mod/core.html#namevirtualhost
 
         :param vhost: New virtual host that was recently created.
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         """
         need_to_save = False
@@ -1536,7 +1658,7 @@ class ApacheConfigurator(common.Installer):
         :param str id_str: Id string for matching
 
         :returns: The matched VirtualHost or None
-        :rtype: :class:`~certbot_apache.obj.VirtualHost` or None
+        :rtype: :class:`~certbot_apache._internal.obj.VirtualHost` or None
 
         :raises .errors.PluginError: If no VirtualHost is found
         """
@@ -1553,7 +1675,7 @@ class ApacheConfigurator(common.Installer):
         used for keeping track of VirtualHost directive over time.
 
         :param vhost: Virtual host to add the id
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :returns: The unique ID or None
         :rtype: str or None
@@ -1575,7 +1697,7 @@ class ApacheConfigurator(common.Installer):
         If ID already exists, returns that instead.
 
         :param vhost: Virtual host to add or find the id
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :returns: The unique ID for vhost
         :rtype: str or None
@@ -1604,7 +1726,7 @@ class ApacheConfigurator(common.Installer):
     ######################################################################
     # Enhancements
     ######################################################################
-    def supported_enhancements(self):  # pylint: disable=no-self-use
+    def supported_enhancements(self):
         """Returns currently supported enhancements."""
         return ["redirect", "ensure-http-header", "staple-ocsp"]
 
@@ -1613,9 +1735,9 @@ class ApacheConfigurator(common.Installer):
 
         :param str domain: domain to enhance
         :param str enhancement: enhancement type defined in
-            :const:`~certbot.constants.ENHANCEMENTS`
+            :const:`~certbot.plugins.enhancements.ENHANCEMENTS`
         :param options: options for the enhancement
-            See :const:`~certbot.constants.ENHANCEMENTS`
+            See :const:`~certbot.plugins.enhancements.ENHANCEMENTS`
             documentation for appropriate parameter.
 
         :raises .errors.PluginError: If Enhancement is not supported, or if
@@ -1653,7 +1775,7 @@ class ApacheConfigurator(common.Installer):
         """Increase the AutoHSTS max-age value
 
         :param vhost: Virtual host object to modify
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :param str id_str: The unique ID string of VirtualHost
 
@@ -1737,13 +1859,13 @@ class ApacheConfigurator(common.Installer):
         .. note:: This function saves the configuration
 
         :param ssl_vhost: Destination of traffic, an ssl enabled vhost
-        :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :param unused_options: Not currently used
         :type unused_options: Not Available
 
         :returns: Success, general_vhost (HTTP vhost)
-        :rtype: (bool, :class:`~certbot_apache.obj.VirtualHost`)
+        :rtype: (bool, :class:`~certbot_apache._internal.obj.VirtualHost`)
 
         """
         min_apache_ver = (2, 3, 3)
@@ -1793,14 +1915,14 @@ class ApacheConfigurator(common.Installer):
         .. note:: This function saves the configuration
 
         :param ssl_vhost: Destination of traffic, an ssl enabled vhost
-        :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :param header_substring: string that uniquely identifies a header.
                 e.g: Strict-Transport-Security, Upgrade-Insecure-Requests.
         :type str
 
         :returns: Success, general_vhost (HTTP vhost)
-        :rtype: (bool, :class:`~certbot_apache.obj.VirtualHost`)
+        :rtype: (bool, :class:`~certbot_apache._internal.obj.VirtualHost`)
 
         :raises .errors.PluginError: If no viable HTTP host can be created or
             set with header header_substring.
@@ -1824,11 +1946,11 @@ class ApacheConfigurator(common.Installer):
                     ssl_vhost.filep)
 
     def _verify_no_matching_http_header(self, ssl_vhost, header_substring):
-        """Checks to see if an there is an existing Header directive that
+        """Checks to see if there is an existing Header directive that
         contains the string header_substring.
 
         :param ssl_vhost: vhost to check
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :param header_substring: string that uniquely identifies a header.
                 e.g: Strict-Transport-Security, Upgrade-Insecure-Requests.
@@ -1865,7 +1987,7 @@ class ApacheConfigurator(common.Installer):
         .. note:: This function saves the configuration
 
         :param ssl_vhost: Destination of traffic, an ssl enabled vhost
-        :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :param unused_options: Not currently used
         :type unused_options: Not Available
@@ -1950,7 +2072,7 @@ class ApacheConfigurator(common.Installer):
         delete certbot's old rewrite rules and set the new one instead.
 
         :param vhost: vhost to check
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :raises errors.PluginEnhancementAlreadyPresent: When the exact
                 certbot redirection WriteRule exists in virtual host.
@@ -1992,7 +2114,7 @@ class ApacheConfigurator(common.Installer):
         """Checks if there exists a RewriteRule directive in vhost
 
         :param vhost: vhost to check
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :returns: True if a RewriteRule directive exists.
         :rtype: bool
@@ -2006,7 +2128,7 @@ class ApacheConfigurator(common.Installer):
         """Checks if a RewriteEngine directive is on
 
         :param vhost: vhost to check
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         """
         rewrite_engine_path_list = self.parser.find_dir("RewriteEngine", "on",
@@ -2023,10 +2145,10 @@ class ApacheConfigurator(common.Installer):
         """Creates an http_vhost specifically to redirect for the ssl_vhost.
 
         :param ssl_vhost: ssl vhost
-        :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :returns: tuple of the form
-            (`success`, :class:`~certbot_apache.obj.VirtualHost`)
+            (`success`, :class:`~certbot_apache._internal.obj.VirtualHost`)
         :rtype: tuple
 
         """
@@ -2152,7 +2274,7 @@ class ApacheConfigurator(common.Installer):
                   of this method where available.
 
         :param vhost: vhost to enable
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :raises .errors.NotSupportedError: If filesystem layout is not
             supported.
@@ -2170,7 +2292,7 @@ class ApacheConfigurator(common.Installer):
             vhost.enabled = True
         return
 
-    def enable_mod(self, mod_name, temp=False):  # pylint: disable=unused-argument
+    def enable_mod(self, mod_name, temp=False):
         """Enables module in Apache.
 
         Both enables and reloads Apache so module is active.
@@ -2228,7 +2350,7 @@ class ApacheConfigurator(common.Installer):
                 error = str(err)
             raise errors.MisconfigurationError(error)
 
-    def config_test(self):  # pylint: disable=no-self-use
+    def config_test(self):
         """Check the configuration of Apache for errors.
 
         :raises .errors.MisconfigurationError: If config_test fails
@@ -2278,7 +2400,7 @@ class ApacheConfigurator(common.Installer):
     ###########################################################################
     # Challenges Section
     ###########################################################################
-    def get_chall_pref(self, unused_domain):  # pylint: disable=no-self-use
+    def get_chall_pref(self, unused_domain):
         """Return list of challenge preferences."""
         return [challenges.HTTP01]
 
@@ -2346,10 +2468,10 @@ class ApacheConfigurator(common.Installer):
         Enable the AutoHSTS enhancement for defined domains
 
         :param _unused_lineage: Certificate lineage object, unused
-        :type _unused_lineage: certbot.storage.RenewableCert
+        :type _unused_lineage: certbot._internal.storage.RenewableCert
 
         :param domains: List of domains in certificate to enhance
-        :type domains: str
+        :type domains: `list` of `str`
         """
 
         self._autohsts_fetch_state()
@@ -2389,7 +2511,7 @@ class ApacheConfigurator(common.Installer):
         """Do the initial AutoHSTS deployment to a vhost
 
         :param ssl_vhost: The VirtualHost object to deploy the AutoHSTS
-        :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost` or None
+        :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost` or None
 
         :raises errors.PluginEnhancementAlreadyPresent: When already enhanced
 
@@ -2471,7 +2593,7 @@ class ApacheConfigurator(common.Installer):
         and changes the HSTS max-age to a high value.
 
         :param lineage: Certificate lineage object
-        :type lineage: certbot.storage.RenewableCert
+        :type lineage: certbot._internal.storage.RenewableCert
         """
         self._autohsts_fetch_state()
         if not self._autohsts:
@@ -2516,4 +2638,4 @@ class ApacheConfigurator(common.Installer):
         self._autohsts_save_state()
 
 
-AutoHSTSEnhancement.register(ApacheConfigurator)  # pylint: disable=no-member
+AutoHSTSEnhancement.register(ApacheConfigurator)

certbot-apache/certbot_apache/_internal/constants.py

@@ -1,6 +1,7 @@
 """Apache plugin constants."""
 import pkg_resources
 
+from certbot.compat import os
 
 MOD_SSL_CONF_DEST = "options-ssl-apache.conf"
 """Name of the mod_ssl config file as saved in `IConfig.config_dir`."""
@@ -9,6 +10,7 @@ MOD_SSL_CONF_DEST = "options-ssl-apache.conf"
 UPDATED_MOD_SSL_CONF_DIGEST = ".updated-options-ssl-apache-conf-digest.txt"
 """Name of the hash of the updated or informed mod_ssl_conf as saved in `IConfig.config_dir`."""
 
+# NEVER REMOVE A SINGLE HASH FROM THIS LIST UNLESS YOU KNOW EXACTLY WHAT YOU ARE DOING!
 ALL_SSL_OPTIONS_HASHES = [
     '2086bca02db48daf93468332543c60ac6acdb6f0b58c7bfdf578a5d47092f82a',
     '4844d36c9a0f587172d9fa10f4f1c9518e3bcfa1947379f155e16a70a728c21a',
@@ -18,11 +20,17 @@ ALL_SSL_OPTIONS_HASHES = [
     'cfdd7c18d2025836ea3307399f509cfb1ebf2612c87dd600a65da2a8e2f2797b',
     '80720bd171ccdc2e6b917ded340defae66919e4624962396b992b7218a561791',
     'c0c022ea6b8a51ecc8f1003d0a04af6c3f2bc1c3ce506b3c2dfc1f11ef931082',
+    '717b0a89f5e4c39b09a42813ac6e747cfbdeb93439499e73f4f70a1fe1473f20',
+    '0fcdc81280cd179a07ec4d29d3595068b9326b455c488de4b09f585d5dafc137',
+    '86cc09ad5415cd6d5f09a947fe2501a9344328b1e8a8b458107ea903e80baa6c',
+    '06675349e457eae856120cdebb564efe546f0b87399f2264baeb41e442c724c7',
+    '5cc003edd93fb9cd03d40c7686495f8f058f485f75b5e764b789245a386e6daf',
+    '007cd497a56a3bb8b6a2c1aeb4997789e7e38992f74e44cc5d13a625a738ac73',
 ]
 """SHA256 hashes of the contents of previous versions of all versions of MOD_SSL_CONF_SRC"""
 
 AUGEAS_LENS_DIR = pkg_resources.resource_filename(
-    "certbot_apache", "augeas_lens")
+    "certbot_apache", os.path.join("_internal", "augeas_lens"))
 """Path to the Augeas lens directory"""
 
 REWRITE_HTTPS_ARGS = [

certbot-apache/certbot_apache/_internal/display_ops.py

@@ -3,10 +3,10 @@ import logging
 
 import zope.component
 
-import certbot.display.util as display_util
 from certbot import errors
 from certbot import interfaces
 from certbot.compat import os
+import certbot.display.util as display_util
 
 logger = logging.getLogger(__name__)
 
@@ -77,7 +77,7 @@ def _vhost_menu(domain, vhosts):
 
     if free_chars < 2:
         logger.debug("Display size is too small for "
-                     "certbot_apache.display_ops._vhost_menu()")
+                     "certbot_apache._internal.display_ops._vhost_menu()")
         # This runs the edge off the screen, but it doesn't cause an "error"
         filename_size = 1
         disp_name_size = 1

certbot-apache/certbot_apache/_internal/dualparser.py

@@ -0,0 +1,306 @@
+""" Dual ParserNode implementation """
+from certbot_apache._internal import assertions
+from certbot_apache._internal import augeasparser
+from certbot_apache._internal import apacheparser
+
+
+class DualNodeBase(object):
+    """ Dual parser interface for in development testing. This is used as the
+    base class for dual parser interface classes. This class handles runtime
+    attribute value assertions."""
+
+    def save(self, msg):  # pragma: no cover
+        """ Call save for both parsers """
+        self.primary.save(msg)
+        self.secondary.save(msg)
+
+    def __getattr__(self, aname):
+        """ Attribute value assertion """
+        firstval = getattr(self.primary, aname)
+        secondval = getattr(self.secondary, aname)
+        exclusions = [
+            # Metadata will inherently be different, as ApacheParserNode does
+            # not have Augeas paths and so on.
+            aname == "metadata",
+            callable(firstval)
+        ]
+        if not any(exclusions):
+            assertions.assertEqualSimple(firstval, secondval)
+        return firstval
+
+    def find_ancestors(self, name):
+        """ Traverses the ancestor tree and returns ancestors matching name """
+        return self._find_helper(DualBlockNode, "find_ancestors", name)
+
+    def _find_helper(self, nodeclass, findfunc, search, **kwargs):
+        """A helper for find_* functions. The function specific attributes should
+        be passed as keyword arguments.
+
+        :param interfaces.ParserNode nodeclass: The node class for results.
+        :param str findfunc: Name of the find function to call
+        :param str search: The search term
+        """
+
+        primary_res = getattr(self.primary, findfunc)(search, **kwargs)
+        secondary_res = getattr(self.secondary, findfunc)(search, **kwargs)
+
+        # The order of search results for Augeas implementation cannot be
+        # assured.
+
+        pass_primary = assertions.isPassNodeList(primary_res)
+        pass_secondary = assertions.isPassNodeList(secondary_res)
+        new_nodes = list()
+
+        if pass_primary and pass_secondary:
+            # Both unimplemented
+            new_nodes.append(nodeclass(primary=primary_res[0],
+                                       secondary=secondary_res[0]))  # pragma: no cover
+        elif pass_primary:
+            for c in secondary_res:
+                new_nodes.append(nodeclass(primary=primary_res[0],
+                                           secondary=c))
+        elif pass_secondary:
+            for c in primary_res:
+                new_nodes.append(nodeclass(primary=c,
+                                           secondary=secondary_res[0]))
+        else:
+            assert len(primary_res) == len(secondary_res)
+            matches = self._create_matching_list(primary_res, secondary_res)
+            for p, s in matches:
+                new_nodes.append(nodeclass(primary=p, secondary=s))
+
+        return new_nodes
+
+
+class DualCommentNode(DualNodeBase):
+    """ Dual parser implementation of CommentNode interface """
+
+    def __init__(self, **kwargs):
+        """ This initialization implementation allows ordinary initialization
+        of CommentNode objects as well as creating a DualCommentNode object
+        using precreated or fetched CommentNode objects if provided as optional
+        arguments primary and secondary.
+
+        Parameters other than the following are from interfaces.CommentNode:
+
+        :param CommentNode primary: Primary pre-created CommentNode, mainly
+            used when creating new DualParser nodes using add_* methods.
+        :param CommentNode secondary: Secondary pre-created CommentNode
+        """
+
+        kwargs.setdefault("primary", None)
+        kwargs.setdefault("secondary", None)
+        primary = kwargs.pop("primary")
+        secondary = kwargs.pop("secondary")
+
+        if primary or secondary:
+            assert primary and secondary
+            self.primary = primary
+            self.secondary = secondary
+        else:
+            self.primary = augeasparser.AugeasCommentNode(**kwargs)
+            self.secondary = apacheparser.ApacheCommentNode(**kwargs)
+
+        assertions.assertEqual(self.primary, self.secondary)
+
+
+class DualDirectiveNode(DualNodeBase):
+    """ Dual parser implementation of DirectiveNode interface """
+
+    def __init__(self, **kwargs):
+        """ This initialization implementation allows ordinary initialization
+        of DirectiveNode objects as well as creating a DualDirectiveNode object
+        using precreated or fetched DirectiveNode objects if provided as optional
+        arguments primary and secondary.
+
+        Parameters other than the following are from interfaces.DirectiveNode:
+
+        :param DirectiveNode primary: Primary pre-created DirectiveNode, mainly
+            used when creating new DualParser nodes using add_* methods.
+        :param DirectiveNode secondary: Secondary pre-created DirectiveNode
+
+
+        """
+
+        kwargs.setdefault("primary", None)
+        kwargs.setdefault("secondary", None)
+        primary = kwargs.pop("primary")
+        secondary = kwargs.pop("secondary")
+
+        if primary or secondary:
+            assert primary and secondary
+            self.primary = primary
+            self.secondary = secondary
+        else:
+            self.primary = augeasparser.AugeasDirectiveNode(**kwargs)
+            self.secondary = apacheparser.ApacheDirectiveNode(**kwargs)
+
+        assertions.assertEqual(self.primary, self.secondary)
+
+    def set_parameters(self, parameters):
+        """ Sets parameters and asserts that both implementation successfully
+        set the parameter sequence """
+
+        self.primary.set_parameters(parameters)
+        self.secondary.set_parameters(parameters)
+        assertions.assertEqual(self.primary, self.secondary)
+
+
+class DualBlockNode(DualNodeBase):
+    """ Dual parser implementation of BlockNode interface """
+
+    def __init__(self, **kwargs):
+        """ This initialization implementation allows ordinary initialization
+        of BlockNode objects as well as creating a DualBlockNode object
+        using precreated or fetched BlockNode objects if provided as optional
+        arguments primary and secondary.
+
+        Parameters other than the following are from interfaces.BlockNode:
+
+        :param BlockNode primary: Primary pre-created BlockNode, mainly
+            used when creating new DualParser nodes using add_* methods.
+        :param BlockNode secondary: Secondary pre-created BlockNode
+        """
+
+        kwargs.setdefault("primary", None)
+        kwargs.setdefault("secondary", None)
+        primary = kwargs.pop("primary")
+        secondary = kwargs.pop("secondary")
+
+        if primary or secondary:
+            assert primary and secondary
+            self.primary = primary
+            self.secondary = secondary
+        else:
+            self.primary = augeasparser.AugeasBlockNode(**kwargs)
+            self.secondary = apacheparser.ApacheBlockNode(**kwargs)
+
+        assertions.assertEqual(self.primary, self.secondary)
+
+    def add_child_block(self, name, parameters=None, position=None):
+        """ Creates a new child BlockNode, asserts that both implementations
+        did it in a similar way, and returns a newly created DualBlockNode object
+        encapsulating both of the newly created objects """
+
+        primary_new = self.primary.add_child_block(name, parameters, position)
+        secondary_new = self.secondary.add_child_block(name, parameters, position)
+        assertions.assertEqual(primary_new, secondary_new)
+        new_block = DualBlockNode(primary=primary_new, secondary=secondary_new)
+        return new_block
+
+    def add_child_directive(self, name, parameters=None, position=None):
+        """ Creates a new child DirectiveNode, asserts that both implementations
+        did it in a similar way, and returns a newly created DualDirectiveNode
+        object encapsulating both of the newly created objects """
+
+        primary_new = self.primary.add_child_directive(name, parameters, position)
+        secondary_new = self.secondary.add_child_directive(name, parameters, position)
+        assertions.assertEqual(primary_new, secondary_new)
+        new_dir = DualDirectiveNode(primary=primary_new, secondary=secondary_new)
+        return new_dir
+
+    def add_child_comment(self, comment="", position=None):
+        """ Creates a new child CommentNode, asserts that both implementations
+        did it in a similar way, and returns a newly created DualCommentNode
+        object encapsulating both of the newly created objects """
+
+        primary_new = self.primary.add_child_comment(comment, position)
+        secondary_new = self.secondary.add_child_comment(comment, position)
+        assertions.assertEqual(primary_new, secondary_new)
+        new_comment = DualCommentNode(primary=primary_new, secondary=secondary_new)
+        return new_comment
+
+    def _create_matching_list(self, primary_list, secondary_list):
+        """ Matches the list of primary_list to a list of secondary_list and
+        returns a list of tuples. This is used to create results for find_
+        methods.
+
+        This helper function exists, because we cannot ensure that the list of
+        search results returned by primary.find_* and secondary.find_* are ordered
+        in a same way. The function pairs the same search results from both
+        implementations to a list of tuples.
+        """
+
+        matched = list()
+        for p in primary_list:
+            match = None
+            for s in secondary_list:
+                try:
+                    assertions.assertEqual(p, s)
+                    match = s
+                    break
+                except AssertionError:
+                    continue
+            if match:
+                matched.append((p, match))
+            else:
+                raise AssertionError("Could not find a matching node.")
+        return matched
+
+    def find_blocks(self, name, exclude=True):
+        """
+        Performs a search for BlockNodes using both implementations and does simple
+        checks for results. This is built upon the assumption that unimplemented
+        find_* methods return a list with a single assertion passing object.
+        After the assertion, it creates a list of newly created DualBlockNode
+        instances that encapsulate the pairs of returned BlockNode objects.
+        """
+
+        return self._find_helper(DualBlockNode, "find_blocks", name,
+                                 exclude=exclude)
+
+    def find_directives(self, name, exclude=True):
+        """
+        Performs a search for DirectiveNodes using both implementations and
+        checks the results. This is built upon the assumption that unimplemented
+        find_* methods return a list with a single assertion passing object.
+        After the assertion, it creates a list of newly created DualDirectiveNode
+        instances that encapsulate the pairs of returned DirectiveNode objects.
+        """
+
+        return self._find_helper(DualDirectiveNode, "find_directives", name,
+                                 exclude=exclude)
+
+    def find_comments(self, comment):
+        """
+        Performs a search for CommentNodes using both implementations and
+        checks the results. This is built upon the assumption that unimplemented
+        find_* methods return a list with a single assertion passing object.
+        After the assertion, it creates a list of newly created DualCommentNode
+        instances that encapsulate the pairs of returned CommentNode objects.
+        """
+
+        return self._find_helper(DualCommentNode, "find_comments", comment)
+
+    def delete_child(self, child):
+        """Deletes a child from the ParserNode implementations. The actual
+        ParserNode implementations are used here directly in order to be able
+        to match a child to the list of children."""
+
+        self.primary.delete_child(child.primary)
+        self.secondary.delete_child(child.secondary)
+
+    def unsaved_files(self):
+        """ Fetches the list of unsaved file paths and asserts that the lists
+        match """
+        primary_files = self.primary.unsaved_files()
+        secondary_files = self.secondary.unsaved_files()
+        assertions.assertEqualSimple(primary_files, secondary_files)
+
+        return primary_files
+
+    def parsed_paths(self):
+        """
+        Returns a list of file paths that have currently been parsed into the parser
+        tree. The returned list may include paths with wildcard characters, for
+        example: ['/etc/apache2/conf.d/*.load']
+
+        This is typically called on the root node of the ParserNode tree.
+
+        :returns: list of file paths of files that have been parsed
+        """
+
+        primary_paths = self.primary.parsed_paths()
+        secondary_paths = self.secondary.parsed_paths()
+        assertions.assertEqualPathsList(primary_paths, secondary_paths)
+        return primary_paths

certbot-apache/certbot_apache/_internal/entrypoint.py

@@ -1,21 +1,21 @@
 """ Entry point for Apache Plugin """
 # Pylint does not like disutils.version when running inside a venv.
 # See: https://github.com/PyCQA/pylint/issues/73
-from distutils.version import LooseVersion  # pylint: disable=no-name-in-module,import-error
+from distutils.version import LooseVersion
 
 from certbot import util
-
-from certbot_apache import configurator
-from certbot_apache import override_arch
-from certbot_apache import override_fedora
-from certbot_apache import override_darwin
-from certbot_apache import override_debian
-from certbot_apache import override_centos
-from certbot_apache import override_gentoo
-from certbot_apache import override_suse
+from certbot_apache._internal import configurator
+from certbot_apache._internal import override_arch
+from certbot_apache._internal import override_centos
+from certbot_apache._internal import override_darwin
+from certbot_apache._internal import override_debian
+from certbot_apache._internal import override_fedora
+from certbot_apache._internal import override_gentoo
+from certbot_apache._internal import override_suse
 
 OVERRIDE_CLASSES = {
     "arch": override_arch.ArchConfigurator,
+    "cloudlinux": override_centos.CentOSConfigurator,
     "darwin": override_darwin.DarwinConfigurator,
     "debian": override_debian.DebianConfigurator,
     "ubuntu": override_debian.DebianConfigurator,
@@ -23,7 +23,10 @@ OVERRIDE_CLASSES = {
     "centos linux": override_centos.CentOSConfigurator,
     "fedora_old": override_centos.CentOSConfigurator,
     "fedora": override_fedora.FedoraConfigurator,
+    "linuxmint": override_debian.DebianConfigurator,
     "ol": override_centos.CentOSConfigurator,
+    "oracle": override_centos.CentOSConfigurator,
+    "redhatenterpriseserver": override_centos.CentOSConfigurator,
     "red hat enterprise linux server": override_centos.CentOSConfigurator,
     "rhel": override_centos.CentOSConfigurator,
     "amazon": override_centos.CentOSConfigurator,
@@ -31,6 +34,9 @@ OVERRIDE_CLASSES = {
     "gentoo base system": override_gentoo.GentooConfigurator,
     "opensuse": override_suse.OpenSUSEConfigurator,
     "suse": override_suse.OpenSUSEConfigurator,
+    "sles": override_suse.OpenSUSEConfigurator,
+    "scientific": override_centos.CentOSConfigurator,
+    "scientific linux": override_centos.CentOSConfigurator,
 }
 
 

certbot-apache/certbot_apache/_internal/http_01.py

@@ -1,20 +1,20 @@
 """A class that performs HTTP-01 challenges for Apache"""
 import logging
+import errno
 
-from acme.magic_typing import List, Set  # pylint: disable=unused-import, no-name-in-module
-
+from acme.magic_typing import List
+from acme.magic_typing import Set
 from certbot import errors
-from certbot.compat import os
 from certbot.compat import filesystem
+from certbot.compat import os
 from certbot.plugins import common
-
-from certbot_apache.obj import VirtualHost  # pylint: disable=unused-import
-from certbot_apache.parser import get_aug_path
+from certbot_apache._internal.obj import VirtualHost  # pylint: disable=unused-import
+from certbot_apache._internal.parser import get_aug_path
 
 logger = logging.getLogger(__name__)
 
 
-class ApacheHttp01(common.TLSSNI01):
+class ApacheHttp01(common.ChallengePerformer):
     """Class that performs HTTP-01 challenges within the Apache configurator."""
 
     CONFIG_TEMPLATE22_PRE = """\
@@ -169,7 +169,15 @@ class ApacheHttp01(common.TLSSNI01):
 
     def _set_up_challenges(self):
         if not os.path.isdir(self.challenge_dir):
-            filesystem.makedirs(self.challenge_dir, 0o755)
+            old_umask = os.umask(0o022)
+            try:
+                filesystem.makedirs(self.challenge_dir, 0o755)
+            except OSError as exception:
+                if exception.errno not in (errno.EEXIST, errno.EISDIR):
+                    raise errors.PluginError(
+                        "Couldn't create root for http-01 challenge")
+            finally:
+                os.umask(old_umask)
 
         responses = []
         for achall in self.achalls:
@@ -195,8 +203,8 @@ class ApacheHttp01(common.TLSSNI01):
 
         if vhost not in self.moded_vhosts:
             logger.debug(
-                "Adding a temporary challenge validation Include for name: %s " +
-                "in: %s", vhost.name, vhost.filep)
+                "Adding a temporary challenge validation Include for name: %s in: %s",
+                vhost.name, vhost.filep)
             self.configurator.parser.add_dir_beginning(
                 vhost.path, "Include", self.challenge_conf_pre)
             self.configurator.parser.add_dir(

certbot-apache/certbot_apache/_internal/interfaces.py

@@ -0,0 +1,516 @@
+"""ParserNode interface for interacting with configuration tree.
+
+General description
+-------------------
+
+The ParserNode interfaces are designed to be able to contain all the parsing logic,
+while allowing their users to interact with the configuration tree in a Pythonic
+and well structured manner.
+
+The structure allows easy traversal of the tree of ParserNodes. Each ParserNode
+stores a reference to its ancestor and immediate children, allowing the user to
+traverse the tree using built in interface methods as well as accessing the interface
+properties directly.
+
+ParserNode interface implementation should stand between the actual underlying
+parser functionality and the business logic within Configurator code, interfacing
+with both. The ParserNode tree is a result of configuration parsing action.
+
+ParserNode tree will be in charge of maintaining the parser state and hence the
+abstract syntax tree (AST). Interactions between ParserNode tree and underlying
+parser should involve only parsing the configuration files to this structure, and
+writing it back to the filesystem - while preserving the format including whitespaces.
+
+For some implementations (Apache for example) it's important to keep track of and
+to use state information while parsing conditional blocks and directives. This
+allows the implementation to set a flag to parts of the parsed configuration
+structure as not being in effect in a case of unmatched conditional block. It's
+important to store these blocks in the tree as well in order to not to conduct
+destructive actions (failing to write back parts of the configuration) while writing
+the AST back to the filesystem.
+
+The ParserNode tree is in charge of maintaining the its own structure while every
+child node fetched with find - methods or by iterating its list of children can be
+changed in place. When making changes the affected nodes should be flagged as "dirty"
+in order for the parser implementation to figure out the parts of the configuration
+that need to be written back to disk during the save() operation.
+
+
+Metadata
+--------
+
+The metadata holds all the implementation specific attributes of the ParserNodes -
+things like the positional information related to the AST, file paths, whitespacing,
+and any other information relevant to the underlying parser engine.
+
+Access to the metadata should be handled by implementation specific methods, allowing
+the Configurator functionality to access the underlying information where needed.
+
+For some implementations the node can be initialized using the information carried
+in metadata alone. This is useful especially when populating the ParserNode tree
+while parsing the configuration.
+
+
+Apache implementation
+---------------------
+
+The Apache implementation of ParserNode interface requires some implementation
+specific functionalities that are not described by the interface itself.
+
+Initialization
+
+When the user of a ParserNode class is creating these objects, they must specify
+the parameters as described in the documentation for the __init__ methods below.
+When these objects are created internally, however, some parameters may not be
+needed because (possibly more detailed) information is included in the metadata
+parameter. In this case, implementations can deviate from the required parameters
+from __init__, however, they should still behave the same when metadata is not
+provided.
+
+For consistency internally, if an argument is provided directly in the ParserNode
+initialization parameters as well as within metadata it's recommended to establish
+clear behavior around this scenario within the implementation.
+
+Conditional blocks
+
+Apache configuration can have conditional blocks, for example: <IfModule ...>,
+resulting the directives and subblocks within it being either enabled or disabled.
+While find_* interface methods allow including the disabled parts of the configuration
+tree in searches a special care needs to be taken while parsing the structure in
+order to reflect the active state of configuration.
+
+Whitespaces
+
+Each ParserNode object is responsible of storing its prepending whitespace characters
+in order to be able to write the AST back to filesystem like it was, preserving the
+format, this applies for parameters of BlockNode and DirectiveNode as well.
+When parameters of ParserNode are changed, the pre-existing whitespaces in the
+parameter sequence are discarded, as the general reason for storing them is to
+maintain the ability to write the configuration back to filesystem exactly like
+it was. This loses its meaning when we have to change the directives or blocks
+parameters for other reasons.
+
+Searches and matching
+
+Apache configuration is largely case insensitive, so the Apache implementation of
+ParserNode interface needs to provide the user means to match block and directive
+names and parameters in case insensitive manner. This does not apply to everything
+however, for example the parameters of a conditional statement may be case sensitive.
+For this reason the internal representation of data should not ignore the case.
+"""
+
+import abc
+import six
+
+from acme.magic_typing import Any, Dict, Optional, Tuple  # pylint: disable=unused-import, no-name-in-module
+
+
+@six.add_metaclass(abc.ABCMeta)
+class ParserNode(object):
+    """
+    ParserNode is the basic building block of the tree of such nodes,
+    representing the structure of the configuration. It is largely meant to keep
+    the structure information intact and idiomatically accessible.
+
+    The root node as well as the child nodes of it should be instances of ParserNode.
+    Nodes keep track of their differences to on-disk representation of configuration
+    by marking modified ParserNodes as dirty to enable partial write-to-disk for
+    different files in the configuration structure.
+
+    While for the most parts the usage and the child types are obvious, "include"-
+    and similar directives are an exception to this rule. This is because of the
+    nature of include directives - which unroll the contents of another file or
+    configuration block to their place. While we could unroll the included nodes
+    to the parent tree, it remains important to keep the context of include nodes
+    separate in order to write back the original configuration as it was.
+
+    For parsers that require the implementation to keep track of the whitespacing,
+    it's responsibility of each ParserNode object itself to store its prepending
+    whitespaces in order to be able to reconstruct the complete configuration file
+    as it was when originally read from the disk.
+
+    ParserNode objects should have the following attributes:
+
+    # Reference to ancestor node, or None if the node is the root node of the
+    # configuration tree.
+    ancestor: Optional[ParserNode]
+
+    # True if this node has been modified since last save.
+    dirty: bool
+
+    # Filepath of the file where the configuration element for this ParserNode
+    # object resides. For root node, the value for filepath is the httpd root
+    # configuration file. Filepath can be None if a configuration directive is
+    # defined in for example the httpd command line.
+    filepath: Optional[str]
+
+    # Metadata dictionary holds all the implementation specific key-value pairs
+    # for the ParserNode instance.
+    metadata: Dict[str, Any]
+    """
+
+    @abc.abstractmethod
+    def __init__(self, **kwargs):
+        """
+        Initializes the ParserNode instance, and sets the ParserNode specific
+        instance variables. This is not meant to be used directly, but through
+        specific classes implementing ParserNode interface.
+
+        :param ancestor: BlockNode ancestor for this CommentNode. Required.
+        :type ancestor: BlockNode or None
+
+        :param filepath: Filesystem path for the file where this CommentNode
+            does or should exist in the filesystem. Required.
+        :type filepath: str or None
+
+        :param dirty: Boolean flag for denoting if this CommentNode has been
+            created or changed after the last save. Default: False.
+        :type dirty: bool
+
+        :param metadata: Dictionary of metadata values for this ParserNode object.
+            Metadata information should be used only internally in the implementation.
+            Default: {}
+        :type metadata: dict
+        """
+
+    @abc.abstractmethod
+    def save(self, msg):
+        """
+        Save traverses the children, and attempts to write the AST to disk for
+        all the objects that are marked dirty. The actual operation of course
+        depends on the underlying implementation. save() shouldn't be called
+        from the Configurator outside of its designated save() method in order
+        to ensure that the Reverter checkpoints are created properly.
+
+        Note: this approach of keeping internal structure of the configuration
+        within the ParserNode tree does not represent the file inclusion structure
+        of actual configuration files that reside in the filesystem. To handle
+        file writes properly, the file specific temporary trees should be extracted
+        from the full ParserNode tree where necessary when writing to disk.
+
+        :param str msg: Message describing the reason for the save.
+
+        """
+
+    @abc.abstractmethod
+    def find_ancestors(self, name):
+        """
+        Traverses the ancestor tree up, searching for BlockNodes with a specific
+        name.
+
+        :param str name: Name of the ancestor BlockNode to search for
+
+        :returns: A list of ancestor BlockNodes that match the name
+        :rtype: list of BlockNode
+        """
+
+
+# Linter rule exclusion done because of https://github.com/PyCQA/pylint/issues/179
+@six.add_metaclass(abc.ABCMeta)  # pylint: disable=abstract-method
+class CommentNode(ParserNode):
+    """
+    CommentNode class is used for representation of comments within the parsed
+    configuration structure. Because of the nature of comments, it is not able
+    to have child nodes and hence it is always treated as a leaf node.
+
+    CommentNode stores its contents in class variable 'comment' and does not
+    have a specific name.
+
+    CommentNode objects should have the following attributes in addition to
+    the ones described in ParserNode:
+
+    # Contains the contents of the comment without the directive notation
+    # (typically # or /* ... */).
+    comment: str
+
+    """
+
+    @abc.abstractmethod
+    def __init__(self, **kwargs):
+        """
+        Initializes the CommentNode instance and sets its instance variables.
+
+        :param comment: Contents of the comment. Required.
+        :type comment: str
+
+        :param ancestor: BlockNode ancestor for this CommentNode. Required.
+        :type ancestor: BlockNode or None
+
+        :param filepath: Filesystem path for the file where this CommentNode
+            does or should exist in the filesystem. Required.
+        :type filepath: str or None
+
+        :param dirty: Boolean flag for denoting if this CommentNode has been
+            created or changed after the last save. Default: False.
+        :type dirty: bool
+        """
+        super(CommentNode, self).__init__(ancestor=kwargs['ancestor'],
+                                          dirty=kwargs.get('dirty', False),
+                                          filepath=kwargs['filepath'],
+                                          metadata=kwargs.get('metadata', {}))  # pragma: no cover
+
+
+@six.add_metaclass(abc.ABCMeta)
+class DirectiveNode(ParserNode):
+    """
+    DirectiveNode class represents a configuration directive within the configuration.
+    It can have zero or more parameters attached to it. Because of the nature of
+    single directives, it is not able to have child nodes and hence it is always
+    treated as a leaf node.
+
+    If a this directive was defined on the httpd command line, the ancestor instance
+    variable for this DirectiveNode should be None, and it should be inserted to the
+    beginning of root BlockNode children sequence.
+
+    DirectiveNode objects should have the following attributes in addition to
+    the ones described in ParserNode:
+
+    # True if this DirectiveNode is enabled and False if it is inside of an
+    # inactive conditional block.
+    enabled: bool
+
+    # Name, or key of the configuration directive. If BlockNode subclass of
+    # DirectiveNode is the root configuration node, the name should be None.
+    name: Optional[str]
+
+    # Tuple of parameters of this ParserNode object, excluding whitespaces.
+    parameters: Tuple[str, ...]
+
+    """
+
+    @abc.abstractmethod
+    def __init__(self, **kwargs):
+        """
+        Initializes the DirectiveNode instance and sets its instance variables.
+
+        :param name: Name or key of the DirectiveNode object. Required.
+        :type name: str or None
+
+        :param tuple parameters: Tuple of str parameters for this DirectiveNode.
+            Default: ().
+        :type parameters: tuple
+
+        :param ancestor: BlockNode ancestor for this DirectiveNode, or None for
+            root configuration node. Required.
+        :type ancestor: BlockNode or None
+
+        :param filepath: Filesystem path for the file where this DirectiveNode
+            does or should exist in the filesystem, or None for directives introduced
+            in the httpd command line. Required.
+        :type filepath: str or None
+
+        :param dirty: Boolean flag for denoting if this DirectiveNode has been
+            created or changed after the last save. Default: False.
+        :type dirty: bool
+
+        :param enabled: True if this DirectiveNode object is parsed in the active
+            configuration of the httpd. False if the DirectiveNode exists within a
+            unmatched conditional configuration block. Default: True.
+        :type enabled: bool
+
+        """
+        super(DirectiveNode, self).__init__(ancestor=kwargs['ancestor'],
+                                            dirty=kwargs.get('dirty', False),
+                                            filepath=kwargs['filepath'],
+                                            metadata=kwargs.get('metadata', {}))  # pragma: no cover
+
+    @abc.abstractmethod
+    def set_parameters(self, parameters):
+        """
+        Sets the sequence of parameters for this ParserNode object without
+        whitespaces. While the whitespaces for parameters are discarded when using
+        this method, the whitespacing preceeding the ParserNode itself should be
+        kept intact.
+
+        :param list parameters: sequence of parameters
+        """
+
+
+@six.add_metaclass(abc.ABCMeta)
+class BlockNode(DirectiveNode):
+    """
+    BlockNode class represents a block of nested configuration directives, comments
+    and other blocks as its children. A BlockNode can have zero or more parameters
+    attached to it.
+
+    Configuration blocks typically consist of one or more child nodes of all possible
+    types. Because of this, the BlockNode class has various discovery and structure
+    management methods.
+
+    Lists of parameters used as an optional argument for some of the methods should
+    be lists of strings that are applicable parameters for each specific BlockNode
+    or DirectiveNode type. As an example, for a following configuration example:
+
+        <VirtualHost *:80>
+           ...
+        </VirtualHost>
+
+    The node type would be BlockNode, name would be 'VirtualHost' and its parameters
+    would be: ['*:80'].
+
+    While for the following example:
+
+        LoadModule alias_module /usr/lib/apache2/modules/mod_alias.so
+
+    The node type would be DirectiveNode, name would be 'LoadModule' and its
+    parameters would be: ['alias_module', '/usr/lib/apache2/modules/mod_alias.so']
+
+    The applicable parameters are dependent on the underlying configuration language
+    and its grammar.
+
+    BlockNode objects should have the following attributes in addition to
+    the ones described in DirectiveNode:
+
+    # Tuple of direct children of this BlockNode object. The order of children
+    # in this tuple retain the order of elements in the parsed configuration
+    # block.
+    children: Tuple[ParserNode, ...]
+
+    """
+
+    @abc.abstractmethod
+    def add_child_block(self, name, parameters=None, position=None):
+        """
+        Adds a new BlockNode child node with provided values and marks the callee
+        BlockNode dirty. This is used to add new children to the AST. The preceeding
+        whitespaces should not be added based on the ancestor or siblings for the
+        newly created object. This is to match the current behavior of the legacy
+        parser implementation.
+
+        :param str name: The name of the child node to add
+        :param list parameters: list of parameters for the node
+        :param int position: Position in the list of children to add the new child
+            node to. Defaults to None, which appends the newly created node to the list.
+            If an integer is given, the child is inserted before that index in the
+            list similar to list().insert.
+
+        :returns: BlockNode instance of the created child block
+
+        """
+
+    @abc.abstractmethod
+    def add_child_directive(self, name, parameters=None, position=None):
+        """
+        Adds a new DirectiveNode child node with provided values and marks the
+        callee BlockNode dirty. This is used to add new children to the AST. The
+        preceeding whitespaces should not be added based on the ancestor or siblings
+        for the newly created object. This is to match the current behavior of the
+        legacy parser implementation.
+
+
+        :param str name: The name of the child node to add
+        :param list parameters: list of parameters for the node
+        :param int position: Position in the list of children to add the new child
+            node to. Defaults to None, which appends the newly created node to the list.
+            If an integer is given, the child is inserted before that index in the
+            list similar to list().insert.
+
+        :returns: DirectiveNode instance of the created child directive
+
+        """
+
+    @abc.abstractmethod
+    def add_child_comment(self, comment="", position=None):
+        """
+        Adds a new CommentNode child node with provided value and marks the
+        callee BlockNode dirty. This is used to add new children to the AST. The
+        preceeding whitespaces should not be added based on the ancestor or siblings
+        for the newly created object. This is to match the current behavior of the
+        legacy parser implementation.
+
+
+        :param str comment: Comment contents
+        :param int position: Position in the list of children to add the new child
+            node to. Defaults to None, which appends the newly created node to the list.
+            If an integer is given, the child is inserted before that index in the
+            list similar to list().insert.
+
+        :returns: CommentNode instance of the created child comment
+
+        """
+
+    @abc.abstractmethod
+    def find_blocks(self, name, exclude=True):
+        """
+        Find a configuration block by name. This method walks the child tree of
+        ParserNodes under the instance it was called from. This way it is possible
+        to search for the whole configuration tree, when starting from root node or
+        to do a partial search when starting from a specified branch. The lookup
+        should be case insensitive.
+
+        :param str name: The name of the directive to search for
+        :param bool exclude: If the search results should exclude the contents of
+            ParserNode objects that reside within conditional blocks and because
+            of current state are not enabled.
+
+        :returns: A list of found BlockNode objects.
+        """
+
+    @abc.abstractmethod
+    def find_directives(self, name, exclude=True):
+        """
+        Find a directive by name. This method walks the child tree of ParserNodes
+        under the instance it was called from. This way it is possible to search
+        for the whole configuration tree, when starting from root node, or to do
+        a partial search when starting from a specified branch. The lookup should
+        be case insensitive.
+
+        :param str name: The name of the directive to search for
+        :param bool exclude: If the search results should exclude the contents of
+            ParserNode objects that reside within conditional blocks and because
+            of current state are not enabled.
+
+        :returns: A list of found DirectiveNode objects.
+
+        """
+
+    @abc.abstractmethod
+    def find_comments(self, comment):
+        """
+        Find comments with value containing the search term.
+
+        This method walks the child tree of ParserNodes under the instance it was
+        called from. This way it is possible to search for the whole configuration
+        tree, when starting from root node, or to do a partial search when starting
+        from a specified branch. The lookup should be case sensitive.
+
+        :param str comment: The content of comment to search for
+
+        :returns: A list of found CommentNode objects.
+
+        """
+
+    @abc.abstractmethod
+    def delete_child(self, child):
+        """
+        Remove a specified child node from the list of children of the called
+        BlockNode object.
+
+        :param ParserNode child: Child ParserNode object to remove from the list
+            of children of the callee.
+        """
+
+    @abc.abstractmethod
+    def unsaved_files(self):
+        """
+        Returns a list of file paths that have been changed since the last save
+        (or the initial configuration parse). The intended use for this method
+        is to tell the Reverter which files need to be included in a checkpoint.
+
+        This is typically called for the root of the ParserNode tree.
+
+        :returns: list of file paths of files that have been changed but not yet
+            saved to disk.
+        """
+
+    @abc.abstractmethod
+    def parsed_paths(self):
+        """
+        Returns a list of file paths that have currently been parsed into the parser
+        tree. The returned list may include paths with wildcard characters, for
+        example: ['/etc/apache2/conf.d/*.load']
+
+        This is typically called on the root node of the ParserNode tree.
+
+        :returns: list of file paths of files that have been parsed
+        """

certbot-apache/certbot_apache/_internal/obj.py

@@ -1,7 +1,7 @@
 """Module contains classes used by the Apache Configurator."""
 import re
 
-from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module
+from acme.magic_typing import Set
 from certbot.plugins import common
 
 
@@ -24,7 +24,7 @@ class Addr(common.Addr):
         return not self.__eq__(other)
 
     def __repr__(self):
-        return "certbot_apache.obj.Addr(" + repr(self.tup) + ")"
+        return "certbot_apache._internal.obj.Addr(" + repr(self.tup) + ")"
 
     def __hash__(self):  # pylint: disable=useless-super-delegation
         # Python 3 requires explicit overridden for __hash__ if __eq__ or
@@ -98,7 +98,7 @@ class Addr(common.Addr):
         return self.get_addr_obj(port)
 
 
-class VirtualHost(object):  # pylint: disable=too-few-public-methods
+class VirtualHost(object):
     """Represents an Apache Virtualhost.
 
     :ivar str filep: file path of VH
@@ -124,9 +124,8 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
     strip_name = re.compile(r"^(?:.+://)?([^ :$]*)")
 
     def __init__(self, filep, path, addrs, ssl, enabled, name=None,
-                 aliases=None, modmacro=False, ancestor=None):
+                 aliases=None, modmacro=False, ancestor=None, node=None):
 
-        # pylint: disable=too-many-arguments
         """Initialize a VH."""
         self.filep = filep
         self.path = path
@@ -137,6 +136,7 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
         self.enabled = enabled
         self.modmacro = modmacro
         self.ancestor = ancestor
+        self.node = node
 
     def get_names(self):
         """Return a set of all names."""

certbot-apache/certbot_apache/_internal/options-ssl-apache.conf

@@ -0,0 +1,18 @@
+# This file contains important security parameters. If you modify this file
+# manually, Certbot will be unable to automatically provide future security
+# updates. Instead, Certbot will print and log an error message with a path to
+# the up-to-date file that you will need to refer to when manually updating
+# this file.
+
+SSLEngine on
+
+# Intermediate configuration, tweak to your needs
+SSLProtocol             all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
+SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
+SSLHonorCipherOrder     off
+
+SSLOptions +StrictRequire
+
+# Add vhost name to log entries:
+LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
+LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common

certbot-apache/certbot_apache/_internal/override_arch.py

@@ -1,11 +1,11 @@
 """ Distribution specific override class for Arch Linux """
 import pkg_resources
-
 import zope.interface
 
 from certbot import interfaces
+from certbot.compat import os
+from certbot_apache._internal import configurator
 
-from certbot_apache import configurator
 
 @zope.interface.provider(interfaces.IPluginFactory)
 class ArchConfigurator(configurator.ApacheConfigurator):
@@ -27,5 +27,5 @@ class ArchConfigurator(configurator.ApacheConfigurator):
         handle_sites=False,
         challenge_location="/etc/httpd/conf",
         MOD_SSL_CONF_SRC=pkg_resources.resource_filename(
-            "certbot_apache", "options-ssl-apache.conf")
+            "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf"))
     )

certbot-apache/certbot_apache/_internal/override_centos.py

@@ -4,17 +4,15 @@ import logging
 import pkg_resources
 import zope.interface
 
+from acme.magic_typing import List
 from certbot import errors
 from certbot import interfaces
 from certbot import util
+from certbot.compat import os
 from certbot.errors import MisconfigurationError
-
-from acme.magic_typing import List  # pylint: disable=unused-import, no-name-in-module
-
-from certbot_apache import apache_util
-from certbot_apache import configurator
-from certbot_apache import parser
-
+from certbot_apache._internal import apache_util
+from certbot_apache._internal import configurator
+from certbot_apache._internal import parser
 
 logger = logging.getLogger(__name__)
 
@@ -40,7 +38,7 @@ class CentOSConfigurator(configurator.ApacheConfigurator):
         handle_sites=False,
         challenge_location="/etc/httpd/conf.d",
         MOD_SSL_CONF_SRC=pkg_resources.resource_filename(
-            "certbot_apache", "centos-options-ssl-apache.conf")
+            "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf"))
     )
 
     def config_test(self):

certbot-apache/certbot_apache/_internal/override_darwin.py

@@ -1,11 +1,11 @@
 """ Distribution specific override class for macOS """
 import pkg_resources
-
 import zope.interface
 
 from certbot import interfaces
+from certbot.compat import os
+from certbot_apache._internal import configurator
 
-from certbot_apache import configurator
 
 @zope.interface.provider(interfaces.IPluginFactory)
 class DarwinConfigurator(configurator.ApacheConfigurator):
@@ -27,5 +27,5 @@ class DarwinConfigurator(configurator.ApacheConfigurator):
         handle_sites=False,
         challenge_location="/etc/apache2/other",
         MOD_SSL_CONF_SRC=pkg_resources.resource_filename(
-            "certbot_apache", "options-ssl-apache.conf")
+            "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf"))
     )

certbot-apache/certbot_apache/_internal/override_debian.py

@@ -7,10 +7,10 @@ import zope.interface
 from certbot import errors
 from certbot import interfaces
 from certbot import util
+from certbot.compat import filesystem
 from certbot.compat import os
-
-from certbot_apache import apache_util
-from certbot_apache import configurator
+from certbot_apache._internal import apache_util
+from certbot_apache._internal import configurator
 
 logger = logging.getLogger(__name__)
 
@@ -35,7 +35,7 @@ class DebianConfigurator(configurator.ApacheConfigurator):
         handle_sites=True,
         challenge_location="/etc/apache2",
         MOD_SSL_CONF_SRC=pkg_resources.resource_filename(
-            "certbot_apache", "options-ssl-apache.conf")
+            "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf"))
     )
 
     def enable_site(self, vhost):
@@ -45,7 +45,7 @@ class DebianConfigurator(configurator.ApacheConfigurator):
                   modules are enabled appropriately.
 
         :param vhost: vhost to enable
-        :type vhost: :class:`~certbot_apache.obj.VirtualHost`
+        :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost`
 
         :raises .errors.NotSupportedError: If filesystem layout is not
             supported.
@@ -65,20 +65,19 @@ class DebianConfigurator(configurator.ApacheConfigurator):
         try:
             os.symlink(vhost.filep, enabled_path)
         except OSError as err:
-            if os.path.islink(enabled_path) and os.path.realpath(
+            if os.path.islink(enabled_path) and filesystem.realpath(
                enabled_path) == vhost.filep:
                 # Already in shape
                 vhost.enabled = True
                 return None
-            else:
-                logger.warning(
-                    "Could not symlink %s to %s, got error: %s", enabled_path,
-                    vhost.filep, err.strerror)
-                errstring = ("Encountered error while trying to enable a " +
-                             "newly created VirtualHost located at {0} by " +
-                             "linking to it from {1}")
-                raise errors.NotSupportedError(errstring.format(vhost.filep,
-                                                                enabled_path))
+            logger.warning(
+                "Could not symlink %s to %s, got error: %s", enabled_path,
+                vhost.filep, err.strerror)
+            errstring = ("Encountered error while trying to enable a " +
+                         "newly created VirtualHost located at {0} by " +
+                         "linking to it from {1}")
+            raise errors.NotSupportedError(errstring.format(vhost.filep,
+                                                            enabled_path))
         vhost.enabled = True
         logger.info("Enabling available site: %s", vhost.filep)
         self.save_notes += "Enabled site %s\n" % vhost.filep

certbot-apache/certbot_apache/_internal/override_fedora.py

@@ -5,10 +5,10 @@ import zope.interface
 from certbot import errors
 from certbot import interfaces
 from certbot import util
-
-from certbot_apache import apache_util
-from certbot_apache import configurator
-from certbot_apache import parser
+from certbot.compat import os
+from certbot_apache._internal import apache_util
+from certbot_apache._internal import configurator
+from certbot_apache._internal import parser
 
 
 @zope.interface.provider(interfaces.IPluginFactory)
@@ -33,7 +33,7 @@ class FedoraConfigurator(configurator.ApacheConfigurator):
         challenge_location="/etc/httpd/conf.d",
         MOD_SSL_CONF_SRC=pkg_resources.resource_filename(
             # TODO: eventually newest version of Fedora will need their own config
-            "certbot_apache", "centos-options-ssl-apache.conf")
+            "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf"))
     )
 
     def config_test(self):

certbot-apache/certbot_apache/_internal/override_gentoo.py

@@ -1,13 +1,13 @@
 """ Distribution specific override class for Gentoo Linux """
 import pkg_resources
-
 import zope.interface
 
 from certbot import interfaces
+from certbot.compat import os
+from certbot_apache._internal import apache_util
+from certbot_apache._internal import configurator
+from certbot_apache._internal import parser
 
-from certbot_apache import apache_util
-from certbot_apache import configurator
-from certbot_apache import parser
 
 @zope.interface.provider(interfaces.IPluginFactory)
 class GentooConfigurator(configurator.ApacheConfigurator):
@@ -30,7 +30,7 @@ class GentooConfigurator(configurator.ApacheConfigurator):
         handle_sites=False,
         challenge_location="/etc/apache2/vhosts.d",
         MOD_SSL_CONF_SRC=pkg_resources.resource_filename(
-            "certbot_apache", "options-ssl-apache.conf")
+            "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf"))
     )
 
     def _prepare_options(self):
@@ -70,6 +70,6 @@ class GentooParser(parser.ApacheParser):
     def update_modules(self):
         """Get loaded modules from httpd process, and add them to DOM"""
         mod_cmd = [self.configurator.option("ctl"), "modules"]
-        matches = self.parse_from_subprocess(mod_cmd, r"(.*)_module")
+        matches = apache_util.parse_from_subprocess(mod_cmd, r"(.*)_module")
         for mod in matches:
             self.add_mod(mod.strip())

certbot-apache/certbot_apache/_internal/override_suse.py

@@ -1,11 +1,11 @@
 """ Distribution specific override class for OpenSUSE """
 import pkg_resources
-
 import zope.interface
 
 from certbot import interfaces
+from certbot.compat import os
+from certbot_apache._internal import configurator
 
-from certbot_apache import configurator
 
 @zope.interface.provider(interfaces.IPluginFactory)
 class OpenSUSEConfigurator(configurator.ApacheConfigurator):
@@ -27,5 +27,5 @@ class OpenSUSEConfigurator(configurator.ApacheConfigurator):
         handle_sites=False,
         challenge_location="/etc/apache2/vhosts.d",
         MOD_SSL_CONF_SRC=pkg_resources.resource_filename(
-            "certbot_apache", "options-ssl-apache.conf")
+            "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf"))
     )

certbot-apache/certbot_apache/_internal/parser.py

@@ -3,23 +3,22 @@ import copy
 import fnmatch
 import logging
 import re
-import subprocess
 import sys
 
 import six
 
-from acme.magic_typing import Dict, List, Set  # pylint: disable=unused-import, no-name-in-module
-
+from acme.magic_typing import Dict
+from acme.magic_typing import List
+from acme.magic_typing import Set
 from certbot import errors
 from certbot.compat import os
-
-from certbot_apache import constants
+from certbot_apache._internal import apache_util
+from certbot_apache._internal import constants
 
 logger = logging.getLogger(__name__)
 
 
 class ApacheParser(object):
-    # pylint: disable=too-many-public-methods
     """Class handles the fine details of parsing the Apache Configuration.
 
     .. todo:: Make parsing general... remove sites-available etc...
@@ -285,38 +284,21 @@ class ApacheParser(object):
                     mods.add(mod_name)
                     mods.add(os.path.basename(mod_filename)[:-2] + "c")
                 else:
-                    logger.debug("Could not read LoadModule directive from " +
-                                 "Augeas path: %s", match_name[6:])
+                    logger.debug("Could not read LoadModule directive from Augeas path: %s",
+                                 match_name[6:])
         self.modules.update(mods)
 
     def update_runtime_variables(self):
         """Update Includes, Defines and Includes from httpd config dump data"""
+
         self.update_defines()
         self.update_includes()
         self.update_modules()
 
     def update_defines(self):
-        """Get Defines from httpd process"""
+        """Updates the dictionary of known variables in the configuration"""
 
-        variables = dict()
-        define_cmd = [self.configurator.option("ctl"), "-t", "-D",
-                      "DUMP_RUN_CFG"]
-        matches = self.parse_from_subprocess(define_cmd, r"Define: ([^ \n]*)")
-        try:
-            matches.remove("DUMP_RUN_CFG")
-        except ValueError:
-            return
-
-        for match in matches:
-            if match.count("=") > 1:
-                logger.error("Unexpected number of equal signs in "
-                             "runtime config dump.")
-                raise errors.PluginError(
-                    "Error parsing Apache runtime variables")
-            parts = match.partition("=")
-            variables[parts[0]] = parts[2]
-
-        self.variables = variables
+        self.variables = apache_util.parse_defines(self.configurator.option("ctl"))
 
     def update_includes(self):
         """Get includes from httpd process, and add them to DOM if needed"""
@@ -326,9 +308,7 @@ class ApacheParser(object):
         # configuration files
         _ = self.find_dir("Include")
 
-        inc_cmd = [self.configurator.option("ctl"), "-t", "-D",
-                   "DUMP_INCLUDES"]
-        matches = self.parse_from_subprocess(inc_cmd, r"\(.*\) (.*)")
+        matches = apache_util.parse_includes(self.configurator.option("ctl"))
         if matches:
             for i in matches:
                 if not self.parsed_in_current(i):
@@ -337,57 +317,11 @@ class ApacheParser(object):
     def update_modules(self):
         """Get loaded modules from httpd process, and add them to DOM"""
 
-        mod_cmd = [self.configurator.option("ctl"), "-t", "-D",
-                       "DUMP_MODULES"]
-        matches = self.parse_from_subprocess(mod_cmd, r"(.*)_module")
+        matches = apache_util.parse_modules(self.configurator.option("ctl"))
         for mod in matches:
             self.add_mod(mod.strip())
 
-    def parse_from_subprocess(self, command, regexp):
-        """Get values from stdout of subprocess command
-
-        :param list command: Command to run
-        :param str regexp: Regexp for parsing
-
-        :returns: list parsed from command output
-        :rtype: list
-
-        """
-        stdout = self._get_runtime_cfg(command)
-        return re.compile(regexp).findall(stdout)
-
-    def _get_runtime_cfg(self, command):  # pylint: disable=no-self-use
-        """Get runtime configuration info.
-        :param command: Command to run
-
-        :returns: stdout from command
-
-        """
-        try:
-            proc = subprocess.Popen(
-                command,
-                stdout=subprocess.PIPE,
-                stderr=subprocess.PIPE,
-                universal_newlines=True)
-            stdout, stderr = proc.communicate()
-
-        except (OSError, ValueError):
-            logger.error(
-                "Error running command %s for runtime parameters!%s",
-                command, os.linesep)
-            raise errors.MisconfigurationError(
-                "Error accessing loaded Apache parameters: {0}".format(
-                command))
-        # Small errors that do not impede
-        if proc.returncode != 0:
-            logger.warning("Error in checking parameter list: %s", stderr)
-            raise errors.MisconfigurationError(
-                "Apache is unable to check whether or not the module is "
-                "loaded because Apache is misconfigured.")
-
-        return stdout
-
-    def filter_args_num(self, matches, args):  # pylint: disable=no-self-use
+    def filter_args_num(self, matches, args):
         """Filter out directives with specific number of arguments.
 
         This function makes the assumption that all related arguments are given
@@ -613,7 +547,7 @@ class ApacheParser(object):
             "%s//*[self::directive=~regexp('%s')]" % (start, regex))
 
         if exclude:
-            matches = self._exclude_dirs(matches)
+            matches = self.exclude_dirs(matches)
 
         if arg is None:
             arg_suffix = "/arg"
@@ -626,7 +560,7 @@ class ApacheParser(object):
         # https://httpd.apache.org/docs/2.4/mod/core.html#include
         for match in matches:
             dir_ = self.aug.get(match).lower()
-            if dir_ == "include" or dir_ == "includeoptional":
+            if dir_ in ("include", "includeoptional"):
                 ordered_matches.extend(self.find_dir(
                     directive, arg,
                     self._get_include_path(self.get_arg(match + "/arg")),
@@ -666,8 +600,7 @@ class ApacheParser(object):
         # e.g. strip now, not later
         if not value:
             return None
-        else:
-            value = value.strip("'\"")
+        value = value.strip("'\"")
 
         variables = ApacheParser.arg_var_interpreter.findall(value)
 
@@ -680,7 +613,13 @@ class ApacheParser(object):
 
         return value
 
-    def _exclude_dirs(self, matches):
+    def get_root_augpath(self):
+        """
+        Returns the Augeas path of root configuration.
+        """
+        return get_aug_path(self.loc["root"])
+
+    def exclude_dirs(self, matches):
         """Exclude directives that are not loaded into the configuration."""
         filters = [("ifmodule", self.modules), ("ifdefine", self.variables)]
 
@@ -766,7 +705,7 @@ class ApacheParser(object):
         split_arg = arg.split("/")
         for idx, split in enumerate(split_arg):
             if any(char in ApacheParser.fnmatch_chars for char in split):
-                # Turn it into a augeas regex
+                # Turn it into an augeas regex
                 # TODO: Can this instead be an augeas glob instead of regex
                 split_arg[idx] = ("* [label()=~regexp('%s')]" %
                                   self.fnmatch_to_re(split))
@@ -776,7 +715,7 @@ class ApacheParser(object):
 
         return get_aug_path(arg)
 
-    def fnmatch_to_re(self, clean_fn_match):  # pylint: disable=no-self-use
+    def fnmatch_to_re(self, clean_fn_match):
         """Method converts Apache's basic fnmatch to regular expression.
 
         Assumption - Configs are assumed to be well-formed and only writable by

certbot-apache/certbot_apache/_internal/parsernode_util.py

@@ -0,0 +1,129 @@
+"""ParserNode utils"""
+
+
+def validate_kwargs(kwargs, required_names):
+    """
+    Ensures that the kwargs dict has all the expected values. This function modifies
+    the kwargs dictionary, and hence the returned dictionary should be used instead
+    in the caller function instead of the original kwargs.
+
+    :param dict kwargs: Dictionary of keyword arguments to validate.
+    :param list required_names: List of required parameter names.
+    """
+
+    validated_kwargs = dict()
+    for name in required_names:
+        try:
+            validated_kwargs[name] = kwargs.pop(name)
+        except KeyError:
+            raise TypeError("Required keyword argument: {} undefined.".format(name))
+
+    # Raise exception if unknown key word arguments are found.
+    if kwargs:
+        unknown = ", ".join(kwargs.keys())
+        raise TypeError("Unknown keyword argument(s): {}".format(unknown))
+    return validated_kwargs
+
+
+def parsernode_kwargs(kwargs):
+    """
+    Validates keyword arguments for ParserNode. This function modifies the kwargs
+    dictionary, and hence the returned dictionary should be used instead in the
+    caller function instead of the original kwargs.
+
+    If metadata is provided, the otherwise required argument "filepath" may be
+    omitted if the implementation is able to extract its value from the metadata.
+    This usecase is handled within this function. Filepath defaults to None.
+
+    :param dict kwargs: Keyword argument dictionary to validate.
+
+    :returns: Tuple of validated and prepared arguments.
+    """
+
+    # As many values of ParserNode instances can be derived from the metadata,
+    # (ancestor being a common exception here) make sure we permit it here as well.
+    if "metadata" in kwargs:
+        # Filepath can be derived from the metadata in Augeas implementation.
+        # Default is None, as in this case the responsibility of populating this
+        # variable lies on the implementation.
+        kwargs.setdefault("filepath", None)
+
+    kwargs.setdefault("dirty", False)
+    kwargs.setdefault("metadata", {})
+
+    kwargs = validate_kwargs(kwargs, ["ancestor", "dirty", "filepath", "metadata"])
+    return kwargs["ancestor"], kwargs["dirty"], kwargs["filepath"], kwargs["metadata"]
+
+
+def commentnode_kwargs(kwargs):
+    """
+    Validates keyword arguments for CommentNode and sets the default values for
+    optional kwargs. This function modifies the kwargs dictionary, and hence the
+    returned dictionary should be used instead in the caller function instead of
+    the original kwargs.
+
+    If metadata is provided, the otherwise required argument "comment" may be
+    omitted if the implementation is able to extract its value from the metadata.
+    This usecase is handled within this function.
+
+    :param dict kwargs: Keyword argument dictionary to validate.
+
+    :returns: Tuple of validated and prepared arguments and ParserNode kwargs.
+    """
+
+    # As many values of ParserNode instances can be derived from the metadata,
+    # (ancestor being a common exception here) make sure we permit it here as well.
+    if "metadata" in kwargs:
+        kwargs.setdefault("comment", None)
+        # Filepath can be derived from the metadata in Augeas implementation.
+        # Default is None, as in this case the responsibility of populating this
+        # variable lies on the implementation.
+        kwargs.setdefault("filepath", None)
+
+    kwargs.setdefault("dirty", False)
+    kwargs.setdefault("metadata", {})
+
+    kwargs = validate_kwargs(kwargs, ["ancestor", "dirty", "filepath", "comment",
+                                      "metadata"])
+
+    comment = kwargs.pop("comment")
+    return comment, kwargs
+
+
+def directivenode_kwargs(kwargs):
+    """
+    Validates keyword arguments for DirectiveNode and BlockNode and sets the
+    default values for optional kwargs. This function modifies the kwargs
+    dictionary, and hence the returned dictionary should be used instead in the
+    caller function instead of the original kwargs.
+
+    If metadata is provided, the otherwise required argument "name" may be
+    omitted if the implementation is able to extract its value from the metadata.
+    This usecase is handled within this function.
+
+    :param dict kwargs: Keyword argument dictionary to validate.
+
+    :returns: Tuple of validated and prepared arguments and ParserNode kwargs.
+    """
+
+    # As many values of ParserNode instances can be derived from the metadata,
+    # (ancestor being a common exception here) make sure we permit it here as well.
+    if "metadata" in kwargs:
+        kwargs.setdefault("name", None)
+        # Filepath can be derived from the metadata in Augeas implementation.
+        # Default is None, as in this case the responsibility of populating this
+        # variable lies on the implementation.
+        kwargs.setdefault("filepath", None)
+
+    kwargs.setdefault("dirty", False)
+    kwargs.setdefault("enabled", True)
+    kwargs.setdefault("parameters", ())
+    kwargs.setdefault("metadata", {})
+
+    kwargs = validate_kwargs(kwargs, ["ancestor", "dirty", "filepath", "name",
+                                      "parameters", "enabled", "metadata"])
+
+    name = kwargs.pop("name")
+    parameters = kwargs.pop("parameters")
+    enabled = kwargs.pop("enabled")
+    return name, parameters, enabled, kwargs

certbot-apache/certbot_apache/apache_util.py

@@ -1,107 +0,0 @@
-""" Utility functions for certbot-apache plugin """
-import binascii
-
-from certbot import util
-from certbot.compat import os
-
-
-def get_mod_deps(mod_name):
-    """Get known module dependencies.
-
-    .. note:: This does not need to be accurate in order for the client to
-        run.  This simply keeps things clean if the user decides to revert
-        changes.
-    .. warning:: If all deps are not included, it may cause incorrect parsing
-        behavior, due to enable_mod's shortcut for updating the parser's
-        currently defined modules (`.ApacheParser.add_mod`)
-        This would only present a major problem in extremely atypical
-        configs that use ifmod for the missing deps.
-
-    """
-    deps = {
-        "ssl": ["setenvif", "mime"]
-    }
-    return deps.get(mod_name, [])
-
-
-def get_file_path(vhost_path):
-    """Get file path from augeas_vhost_path.
-
-    Takes in Augeas path and returns the file name
-
-    :param str vhost_path: Augeas virtual host path
-
-    :returns: filename of vhost
-    :rtype: str
-
-    """
-    if not vhost_path or not vhost_path.startswith("/files/"):
-        return None
-
-    return _split_aug_path(vhost_path)[0]
-
-
-def get_internal_aug_path(vhost_path):
-    """Get the Augeas path for a vhost with the file path removed.
-
-    :param str vhost_path: Augeas virtual host path
-
-    :returns: Augeas path to vhost relative to the containing file
-    :rtype: str
-
-    """
-    return _split_aug_path(vhost_path)[1]
-
-
-def _split_aug_path(vhost_path):
-    """Splits an Augeas path into a file path and an internal path.
-
-    After removing "/files", this function splits vhost_path into the
-    file path and the remaining Augeas path.
-
-    :param str vhost_path: Augeas virtual host path
-
-    :returns: file path and internal Augeas path
-    :rtype: `tuple` of `str`
-
-    """
-    # Strip off /files
-    file_path = vhost_path[6:]
-    internal_path = []
-
-    # Remove components from the end of file_path until it becomes valid
-    while not os.path.exists(file_path):
-        file_path, _, internal_path_part = file_path.rpartition("/")
-        internal_path.append(internal_path_part)
-
-    return file_path, "/".join(reversed(internal_path))
-
-
-def parse_define_file(filepath, varname):
-    """ Parses Defines from a variable in configuration file
-
-    :param str filepath: Path of file to parse
-    :param str varname: Name of the variable
-
-    :returns: Dict of Define:Value pairs
-    :rtype: `dict`
-
-    """
-    return_vars = {}
-    # Get list of words in the variable
-    a_opts = util.get_var_from_file(varname, filepath).split()
-    for i, v in enumerate(a_opts):
-        # Handle Define statements and make sure it has an argument
-        if v == "-D" and len(a_opts) >= i+2:
-            var_parts = a_opts[i+1].partition("=")
-            return_vars[var_parts[0]] = var_parts[2]
-        elif len(v) > 2 and v.startswith("-D"):
-            # Found var with no whitespace separator
-            var_parts = v[2:].partition("=")
-            return_vars[var_parts[0]] = var_parts[2]
-    return return_vars
-
-
-def unique_id():
-    """ Returns an unique id to be used as a VirtualHost identifier"""
-    return binascii.hexlify(os.urandom(16)).decode("utf-8")

certbot-apache/certbot_apache/centos-options-ssl-apache.conf

@@ -1,25 +0,0 @@
-# This file contains important security parameters. If you modify this file
-# manually, Certbot will be unable to automatically provide future security
-# updates. Instead, Certbot will print and log an error message with a path to
-# the up-to-date file that you will need to refer to when manually updating
-# this file.
-
-SSLEngine on
-
-# Intermediate configuration, tweak to your needs
-SSLProtocol             all -SSLv2 -SSLv3
-SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
-SSLHonorCipherOrder     on
-
-SSLOptions +StrictRequire
-
-# Add vhost name to log entries:
-LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
-LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common
-
-#CustomLog /var/log/apache2/access.log vhost_combined
-#LogLevel warn
-#ErrorLog /var/log/apache2/error.log
-
-# Always ensure Cookies have "Secure" set (JAH 2012/1)
-#Header edit Set-Cookie (?i)^(.*)(;\s*secure)??((\s*;)?(.*)) "$1; Secure$3$4"

certbot-apache/certbot_apache/options-ssl-apache.conf

@@ -1,26 +0,0 @@
-# This file contains important security parameters. If you modify this file
-# manually, Certbot will be unable to automatically provide future security
-# updates. Instead, Certbot will print and log an error message with a path to
-# the up-to-date file that you will need to refer to when manually updating
-# this file.
-
-SSLEngine on
-
-# Intermediate configuration, tweak to your needs
-SSLProtocol             all -SSLv2 -SSLv3
-SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
-SSLHonorCipherOrder     on
-SSLCompression          off
-
-SSLOptions +StrictRequire
-
-# Add vhost name to log entries:
-LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
-LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common
-
-#CustomLog /var/log/apache2/access.log vhost_combined
-#LogLevel warn
-#ErrorLog /var/log/apache2/error.log
-
-# Always ensure Cookies have "Secure" set (JAH 2012/1)
-#Header edit Set-Cookie (?i)^(.*)(;\s*secure)??((\s*;)?(.*)) "$1; Secure$3$4"

certbot-apache/certbot_apache/tests/__init__.py

@@ -1 +0,0 @@
-"""Certbot Apache Tests"""