Testing

court-scraper is developed primarily on Python 3.7 and uses the pytest library for unit testing. We use pytest-vcr for scrapers that use the requests library (e.g. oscn.Site). Scrapers that use Selenium should include a minimal set of live webtests to ensure correct functionality and guard against regressions.

Note

Selenium and other long-running tests should be marked as slow to enable optional running of these tests.

Install and run tests

Assuming you’ve cloned this repo locally and installed test and application dependencies, you can run tests by executing pytest in an active virtual environment:

cd court-scraper/
pipenv install --dev

# Execute tests
pipenv run pytest

Slow tests

Slow-running tests should be marked as such:

@pytest.mark.slow
def test_something_slow():
    ...

Slow tests are skipped by default. To run them, pass the --runslow flag when invoking pytest:

pytest --runslow

Live tests

Tests that hit live web sites should be marked as webtest, allowing them to be executed selectively:

@pytest.mark.webtest
def test_that_hits_live_website():
    ...

# On the command line, run only tests marked as "webtest"
pytest -m webtest

In many cases, tests that hit live websites should be marked as both webtest and slow:

@pytest.mark.webtest
@pytest.mark.slow
def test_that_hits_live_website():
    ...

# On the command line, use both flags to target long-running webtests
pytest --runslow -m webtest

Live web tests of Selenium-based scrapers will open a web browser by default. All tests of Selenium scrapers should use the headless fixture in order to provide the ability to disable running tests in browser.

These tests should typically be marked as slow and webtest as well.:

@pytest.mark.webtest
@pytest.mark.slow
def test_selenium_scrape(headless):

You can activate headless mode when running pytest by using the --headless flag:

pytest --headless --runslow

Test login credentials

Tests that hit live web sites may require authentication, as in the case of some Odyssey sites such as Dekalb and Chatham counties in Georgia.

Such tests require creating user accounts and adding login credentials to a local YAML configuration file.