Testing¶
court-scraper is developed primarily on Python 3.7 and uses the pytest library for unit testing.
We use pytest-vcr for scrapers that use the requests
library
(e.g. oscn.Site
). Scrapers that
use Selenium should include a minimal set of live webtests to ensure
correct functionality and guard against regressions.
Note
Selenium and other long-running tests should be marked as slow to enable optional running of these tests.
Install and run tests¶
Assuming you’ve cloned this repo locally and installed test and application dependencies, you can run tests by executing pytest in an active virtual environment:
cd court-scraper/
pipenv install --dev
# Execute tests
pipenv run pytest
Slow tests¶
Slow-running tests should be marked as such:
@pytest.mark.slow
def test_something_slow():
...
Slow tests are skipped by default. To run them, pass the --runslow
flag
when invoking pytest:
pytest --runslow
Live tests¶
Tests that hit live web sites should be marked as webtest
, allowing them to be executed selectively:
@pytest.mark.webtest
def test_that_hits_live_website():
...
# On the command line, run only tests marked as "webtest"
pytest -m webtest
In many cases, tests that hit live websites should be marked as both webtest
and slow
:
@pytest.mark.webtest
@pytest.mark.slow
def test_that_hits_live_website():
...
# On the command line, use both flags to target long-running webtests
pytest --runslow -m webtest
Live web tests of Selenium-based scrapers will open a web browser by default.
All tests of Selenium scrapers should use the headless
fixture in order to provide
the ability to disable running tests in browser.
These tests should typically be marked as slow
and webtest
as well.:
@pytest.mark.webtest
@pytest.mark.slow
def test_selenium_scrape(headless):
You can activate headless mode when running pytest by using the --headless
flag:
pytest --headless --runslow
Test login credentials¶
Tests that hit live web sites may require authentication, as in the case of some Odyssey sites such as Dekalb and Chatham counties in Georgia.
Such tests require creating user accounts and adding login credentials to a local YAML configuration file.