head 1.17; access; symbols pkgsrc-2026Q1:1.16.0.2 pkgsrc-2026Q1-base:1.16 pkgsrc-2025Q4:1.15.0.6 pkgsrc-2025Q4-base:1.15 pkgsrc-2025Q3:1.15.0.4 pkgsrc-2025Q3-base:1.15 pkgsrc-2025Q2:1.15.0.2 pkgsrc-2025Q2-base:1.15 pkgsrc-2025Q1:1.13.0.4 pkgsrc-2025Q1-base:1.13 pkgsrc-2024Q4:1.13.0.2 pkgsrc-2024Q4-base:1.13 pkgsrc-2024Q3:1.12.0.4 pkgsrc-2024Q3-base:1.12 pkgsrc-2024Q2:1.12.0.2 pkgsrc-2024Q2-base:1.12 pkgsrc-2024Q1:1.11.0.2 pkgsrc-2024Q1-base:1.11 pkgsrc-2023Q4:1.10.0.2 pkgsrc-2023Q4-base:1.10 pkgsrc-2023Q3:1.9.0.4 pkgsrc-2023Q3-base:1.9 pkgsrc-2023Q2:1.9.0.2 pkgsrc-2023Q2-base:1.9 pkgsrc-2023Q1:1.8.0.18 pkgsrc-2023Q1-base:1.8 pkgsrc-2022Q4:1.8.0.16 pkgsrc-2022Q4-base:1.8 pkgsrc-2022Q3:1.8.0.14 pkgsrc-2022Q3-base:1.8 pkgsrc-2022Q2:1.8.0.12 pkgsrc-2022Q2-base:1.8 pkgsrc-2022Q1:1.8.0.10 pkgsrc-2022Q1-base:1.8 pkgsrc-2021Q4:1.8.0.8 pkgsrc-2021Q4-base:1.8 pkgsrc-2021Q3:1.8.0.6 pkgsrc-2021Q3-base:1.8 pkgsrc-2021Q2:1.8.0.4 pkgsrc-2021Q2-base:1.8 pkgsrc-2021Q1:1.8.0.2 pkgsrc-2021Q1-base:1.8 pkgsrc-2020Q4:1.7.0.8 pkgsrc-2020Q4-base:1.7 pkgsrc-2020Q3:1.7.0.6 pkgsrc-2020Q3-base:1.7 pkgsrc-2020Q2:1.7.0.4 pkgsrc-2020Q2-base:1.7 pkgsrc-2020Q1:1.7.0.2 pkgsrc-2020Q1-base:1.7 pkgsrc-2019Q4:1.6.0.6 pkgsrc-2019Q4-base:1.6 pkgsrc-2019Q3:1.6.0.2 pkgsrc-2019Q3-base:1.6 pkgsrc-2019Q2:1.5.0.4 pkgsrc-2019Q2-base:1.5 pkgsrc-2019Q1:1.5.0.2 pkgsrc-2019Q1-base:1.5 pkgsrc-2018Q4:1.3.0.8 pkgsrc-2018Q4-base:1.3 pkgsrc-2018Q3:1.3.0.6 pkgsrc-2018Q3-base:1.3 pkgsrc-2018Q2:1.3.0.4 pkgsrc-2018Q2-base:1.3 pkgsrc-2018Q1:1.3.0.2 pkgsrc-2018Q1-base:1.3 pkgsrc-2017Q4:1.2.0.8 pkgsrc-2017Q4-base:1.2 pkgsrc-2017Q3:1.2.0.6 pkgsrc-2017Q3-base:1.2 pkgsrc-2017Q2:1.2.0.2 pkgsrc-2017Q2-base:1.2 pkgsrc-2017Q1:1.1.0.2 pkgsrc-2017Q1-base:1.1; locks; strict; comment @# @; 1.17 date 2026.04.22.11.13.56; author adam; state Exp; branches; next 1.16; commitid JI7mRIWsAR79HWCG; 1.16 date 2026.01.08.16.07.35; author adam; state Exp; branches; next 1.15; commitid oreeHmbxAFj7FBpG; 1.15 date 2025.05.15.09.21.59; author adam; state Exp; branches; next 1.14; commitid HdAhaaOhettd4ZUF; 1.14 date 2025.04.14.20.28.02; author adam; state Exp; branches; next 1.13; commitid UEGjPtvhiyhEK3RF; 1.13 date 2024.11.30.06.56.49; author adam; state Exp; branches; next 1.12; commitid wokNRJGH9n1dBDzF; 1.12 date 2024.05.14.19.15.59; author adam; state Exp; branches; next 1.11; commitid 4S5NYv2usH3n70aF; 1.11 date 2024.02.16.19.02.45; author adam; state Exp; branches; next 1.10; commitid i6P7L6ng8lxhSGYE; 1.10 date 2023.10.10.17.18.23; author triaxx; state Exp; branches; next 1.9; commitid tLzh8iEqeU6qr6IE; 1.9 date 2023.04.27.09.33.44; author adam; state Exp; branches; next 1.8; commitid tlgNed5lNpikcJmE; 1.8 date 2021.03.22.08.56.56; author triaxx; state Exp; branches; next 1.7; commitid Do2yerH4BnWuChMC; 1.7 date 2020.01.29.22.06.30; author adam; state Exp; branches; next 1.6; commitid 6gSvVBpWVN1PqDUB; 1.6 date 2019.08.22.08.21.11; author adam; state Exp; branches; next 1.5; commitid 4ykVR3aKtVPZZZzB; 1.5 date 2019.01.31.09.07.46; author adam; state Exp; branches; next 1.4; commitid YqCcULhRsPC1NU9B; 1.4 date 2019.01.24.14.11.48; author adam; state Exp; branches; next 1.3; commitid D549NF1qLDkjH29B; 1.3 date 2018.01.04.21.31.41; author adam; state Exp; branches; next 1.2; commitid AYVt976oH416vBlA; 1.2 date 2017.05.20.06.25.36; author adam; state Exp; branches; next 1.1; commitid SHXh0bHW2C98R5Sz; 1.1 date 2017.02.13.21.25.33; author adam; state Exp; branches; next ; commitid RFPMXVz9YZD0VPFz; desc @@ 1.17 log @py-scrapy: updated to 2.15.0 Scrapy 2.15.0 (2026-04-09) Highlights: - Experimental support for running without a Twisted reactor - Experimental ``httpx``-based download handler @ text @@@comment $NetBSD: PLIST,v 1.16 2026/01/08 16:07:35 adam Exp $ bin/scrapy-${PYVERSSUFFIX} ${PYSITELIB}/${WHEEL_INFODIR}/METADATA ${PYSITELIB}/${WHEEL_INFODIR}/RECORD ${PYSITELIB}/${WHEEL_INFODIR}/WHEEL ${PYSITELIB}/${WHEEL_INFODIR}/entry_points.txt ${PYSITELIB}/${WHEEL_INFODIR}/licenses/AUTHORS ${PYSITELIB}/${WHEEL_INFODIR}/licenses/LICENSE ${PYSITELIB}/scrapy/VERSION ${PYSITELIB}/scrapy/__init__.py ${PYSITELIB}/scrapy/__init__.pyc ${PYSITELIB}/scrapy/__init__.pyo ${PYSITELIB}/scrapy/__main__.py ${PYSITELIB}/scrapy/__main__.pyc ${PYSITELIB}/scrapy/__main__.pyo ${PYSITELIB}/scrapy/addons.py ${PYSITELIB}/scrapy/addons.pyc ${PYSITELIB}/scrapy/addons.pyo ${PYSITELIB}/scrapy/cmdline.py ${PYSITELIB}/scrapy/cmdline.pyc ${PYSITELIB}/scrapy/cmdline.pyo ${PYSITELIB}/scrapy/commands/__init__.py ${PYSITELIB}/scrapy/commands/__init__.pyc ${PYSITELIB}/scrapy/commands/__init__.pyo ${PYSITELIB}/scrapy/commands/bench.py ${PYSITELIB}/scrapy/commands/bench.pyc ${PYSITELIB}/scrapy/commands/bench.pyo ${PYSITELIB}/scrapy/commands/check.py ${PYSITELIB}/scrapy/commands/check.pyc ${PYSITELIB}/scrapy/commands/check.pyo ${PYSITELIB}/scrapy/commands/crawl.py ${PYSITELIB}/scrapy/commands/crawl.pyc ${PYSITELIB}/scrapy/commands/crawl.pyo ${PYSITELIB}/scrapy/commands/edit.py ${PYSITELIB}/scrapy/commands/edit.pyc ${PYSITELIB}/scrapy/commands/edit.pyo ${PYSITELIB}/scrapy/commands/fetch.py ${PYSITELIB}/scrapy/commands/fetch.pyc ${PYSITELIB}/scrapy/commands/fetch.pyo ${PYSITELIB}/scrapy/commands/genspider.py ${PYSITELIB}/scrapy/commands/genspider.pyc ${PYSITELIB}/scrapy/commands/genspider.pyo ${PYSITELIB}/scrapy/commands/list.py ${PYSITELIB}/scrapy/commands/list.pyc ${PYSITELIB}/scrapy/commands/list.pyo ${PYSITELIB}/scrapy/commands/parse.py ${PYSITELIB}/scrapy/commands/parse.pyc ${PYSITELIB}/scrapy/commands/parse.pyo ${PYSITELIB}/scrapy/commands/runspider.py ${PYSITELIB}/scrapy/commands/runspider.pyc ${PYSITELIB}/scrapy/commands/runspider.pyo ${PYSITELIB}/scrapy/commands/settings.py ${PYSITELIB}/scrapy/commands/settings.pyc ${PYSITELIB}/scrapy/commands/settings.pyo ${PYSITELIB}/scrapy/commands/shell.py ${PYSITELIB}/scrapy/commands/shell.pyc ${PYSITELIB}/scrapy/commands/shell.pyo ${PYSITELIB}/scrapy/commands/startproject.py ${PYSITELIB}/scrapy/commands/startproject.pyc ${PYSITELIB}/scrapy/commands/startproject.pyo ${PYSITELIB}/scrapy/commands/version.py ${PYSITELIB}/scrapy/commands/version.pyc ${PYSITELIB}/scrapy/commands/version.pyo ${PYSITELIB}/scrapy/commands/view.py ${PYSITELIB}/scrapy/commands/view.pyc ${PYSITELIB}/scrapy/commands/view.pyo ${PYSITELIB}/scrapy/contracts/__init__.py ${PYSITELIB}/scrapy/contracts/__init__.pyc ${PYSITELIB}/scrapy/contracts/__init__.pyo ${PYSITELIB}/scrapy/contracts/default.py ${PYSITELIB}/scrapy/contracts/default.pyc ${PYSITELIB}/scrapy/contracts/default.pyo ${PYSITELIB}/scrapy/core/__init__.py ${PYSITELIB}/scrapy/core/__init__.pyc ${PYSITELIB}/scrapy/core/__init__.pyo ${PYSITELIB}/scrapy/core/downloader/__init__.py ${PYSITELIB}/scrapy/core/downloader/__init__.pyc ${PYSITELIB}/scrapy/core/downloader/__init__.pyo ${PYSITELIB}/scrapy/core/downloader/contextfactory.py ${PYSITELIB}/scrapy/core/downloader/contextfactory.pyc ${PYSITELIB}/scrapy/core/downloader/contextfactory.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/__init__.py ${PYSITELIB}/scrapy/core/downloader/handlers/__init__.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/__init__.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/_httpx.py ${PYSITELIB}/scrapy/core/downloader/handlers/_httpx.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/_httpx.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/base.py ${PYSITELIB}/scrapy/core/downloader/handlers/base.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/base.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/datauri.py ${PYSITELIB}/scrapy/core/downloader/handlers/datauri.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/datauri.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/file.py ${PYSITELIB}/scrapy/core/downloader/handlers/file.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/file.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/ftp.py ${PYSITELIB}/scrapy/core/downloader/handlers/ftp.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/ftp.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/http.py ${PYSITELIB}/scrapy/core/downloader/handlers/http.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/http.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/http10.py ${PYSITELIB}/scrapy/core/downloader/handlers/http10.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/http10.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/http11.py ${PYSITELIB}/scrapy/core/downloader/handlers/http11.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/http11.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/http2.py ${PYSITELIB}/scrapy/core/downloader/handlers/http2.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/http2.pyo ${PYSITELIB}/scrapy/core/downloader/handlers/s3.py ${PYSITELIB}/scrapy/core/downloader/handlers/s3.pyc ${PYSITELIB}/scrapy/core/downloader/handlers/s3.pyo ${PYSITELIB}/scrapy/core/downloader/middleware.py ${PYSITELIB}/scrapy/core/downloader/middleware.pyc ${PYSITELIB}/scrapy/core/downloader/middleware.pyo ${PYSITELIB}/scrapy/core/downloader/tls.py ${PYSITELIB}/scrapy/core/downloader/tls.pyc ${PYSITELIB}/scrapy/core/downloader/tls.pyo ${PYSITELIB}/scrapy/core/downloader/webclient.py ${PYSITELIB}/scrapy/core/downloader/webclient.pyc ${PYSITELIB}/scrapy/core/downloader/webclient.pyo ${PYSITELIB}/scrapy/core/engine.py ${PYSITELIB}/scrapy/core/engine.pyc ${PYSITELIB}/scrapy/core/engine.pyo ${PYSITELIB}/scrapy/core/http2/__init__.py ${PYSITELIB}/scrapy/core/http2/__init__.pyc ${PYSITELIB}/scrapy/core/http2/__init__.pyo ${PYSITELIB}/scrapy/core/http2/agent.py ${PYSITELIB}/scrapy/core/http2/agent.pyc ${PYSITELIB}/scrapy/core/http2/agent.pyo ${PYSITELIB}/scrapy/core/http2/protocol.py ${PYSITELIB}/scrapy/core/http2/protocol.pyc ${PYSITELIB}/scrapy/core/http2/protocol.pyo ${PYSITELIB}/scrapy/core/http2/stream.py ${PYSITELIB}/scrapy/core/http2/stream.pyc ${PYSITELIB}/scrapy/core/http2/stream.pyo ${PYSITELIB}/scrapy/core/scheduler.py ${PYSITELIB}/scrapy/core/scheduler.pyc ${PYSITELIB}/scrapy/core/scheduler.pyo ${PYSITELIB}/scrapy/core/scraper.py ${PYSITELIB}/scrapy/core/scraper.pyc ${PYSITELIB}/scrapy/core/scraper.pyo ${PYSITELIB}/scrapy/core/spidermw.py ${PYSITELIB}/scrapy/core/spidermw.pyc ${PYSITELIB}/scrapy/core/spidermw.pyo ${PYSITELIB}/scrapy/crawler.py ${PYSITELIB}/scrapy/crawler.pyc ${PYSITELIB}/scrapy/crawler.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/__init__.py ${PYSITELIB}/scrapy/downloadermiddlewares/__init__.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/__init__.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/ajaxcrawl.py ${PYSITELIB}/scrapy/downloadermiddlewares/ajaxcrawl.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/ajaxcrawl.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/cookies.py ${PYSITELIB}/scrapy/downloadermiddlewares/cookies.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/cookies.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/defaultheaders.py ${PYSITELIB}/scrapy/downloadermiddlewares/defaultheaders.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/defaultheaders.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/downloadtimeout.py ${PYSITELIB}/scrapy/downloadermiddlewares/downloadtimeout.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/downloadtimeout.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.py ${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.py ${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.py ${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.py ${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/offsite.py ${PYSITELIB}/scrapy/downloadermiddlewares/offsite.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/offsite.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/redirect.py ${PYSITELIB}/scrapy/downloadermiddlewares/redirect.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/redirect.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/retry.py ${PYSITELIB}/scrapy/downloadermiddlewares/retry.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/retry.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.py ${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/stats.py ${PYSITELIB}/scrapy/downloadermiddlewares/stats.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/stats.pyo ${PYSITELIB}/scrapy/downloadermiddlewares/useragent.py ${PYSITELIB}/scrapy/downloadermiddlewares/useragent.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/useragent.pyo ${PYSITELIB}/scrapy/dupefilters.py ${PYSITELIB}/scrapy/dupefilters.pyc ${PYSITELIB}/scrapy/dupefilters.pyo ${PYSITELIB}/scrapy/exceptions.py ${PYSITELIB}/scrapy/exceptions.pyc ${PYSITELIB}/scrapy/exceptions.pyo ${PYSITELIB}/scrapy/exporters.py ${PYSITELIB}/scrapy/exporters.pyc ${PYSITELIB}/scrapy/exporters.pyo ${PYSITELIB}/scrapy/extension.py ${PYSITELIB}/scrapy/extension.pyc ${PYSITELIB}/scrapy/extension.pyo ${PYSITELIB}/scrapy/extensions/__init__.py ${PYSITELIB}/scrapy/extensions/__init__.pyc ${PYSITELIB}/scrapy/extensions/__init__.pyo ${PYSITELIB}/scrapy/extensions/closespider.py ${PYSITELIB}/scrapy/extensions/closespider.pyc ${PYSITELIB}/scrapy/extensions/closespider.pyo ${PYSITELIB}/scrapy/extensions/corestats.py ${PYSITELIB}/scrapy/extensions/corestats.pyc ${PYSITELIB}/scrapy/extensions/corestats.pyo ${PYSITELIB}/scrapy/extensions/debug.py ${PYSITELIB}/scrapy/extensions/debug.pyc ${PYSITELIB}/scrapy/extensions/debug.pyo ${PYSITELIB}/scrapy/extensions/feedexport.py ${PYSITELIB}/scrapy/extensions/feedexport.pyc ${PYSITELIB}/scrapy/extensions/feedexport.pyo ${PYSITELIB}/scrapy/extensions/httpcache.py ${PYSITELIB}/scrapy/extensions/httpcache.pyc ${PYSITELIB}/scrapy/extensions/httpcache.pyo ${PYSITELIB}/scrapy/extensions/logcount.py ${PYSITELIB}/scrapy/extensions/logcount.pyc ${PYSITELIB}/scrapy/extensions/logcount.pyo ${PYSITELIB}/scrapy/extensions/logstats.py ${PYSITELIB}/scrapy/extensions/logstats.pyc ${PYSITELIB}/scrapy/extensions/logstats.pyo ${PYSITELIB}/scrapy/extensions/memdebug.py ${PYSITELIB}/scrapy/extensions/memdebug.pyc ${PYSITELIB}/scrapy/extensions/memdebug.pyo ${PYSITELIB}/scrapy/extensions/memusage.py ${PYSITELIB}/scrapy/extensions/memusage.pyc ${PYSITELIB}/scrapy/extensions/memusage.pyo ${PYSITELIB}/scrapy/extensions/periodic_log.py ${PYSITELIB}/scrapy/extensions/periodic_log.pyc ${PYSITELIB}/scrapy/extensions/periodic_log.pyo ${PYSITELIB}/scrapy/extensions/postprocessing.py ${PYSITELIB}/scrapy/extensions/postprocessing.pyc ${PYSITELIB}/scrapy/extensions/postprocessing.pyo ${PYSITELIB}/scrapy/extensions/spiderstate.py ${PYSITELIB}/scrapy/extensions/spiderstate.pyc ${PYSITELIB}/scrapy/extensions/spiderstate.pyo ${PYSITELIB}/scrapy/extensions/statsmailer.py ${PYSITELIB}/scrapy/extensions/statsmailer.pyc ${PYSITELIB}/scrapy/extensions/statsmailer.pyo ${PYSITELIB}/scrapy/extensions/telnet.py ${PYSITELIB}/scrapy/extensions/telnet.pyc ${PYSITELIB}/scrapy/extensions/telnet.pyo ${PYSITELIB}/scrapy/extensions/throttle.py ${PYSITELIB}/scrapy/extensions/throttle.pyc ${PYSITELIB}/scrapy/extensions/throttle.pyo ${PYSITELIB}/scrapy/http/__init__.py ${PYSITELIB}/scrapy/http/__init__.pyc ${PYSITELIB}/scrapy/http/__init__.pyo ${PYSITELIB}/scrapy/http/cookies.py ${PYSITELIB}/scrapy/http/cookies.pyc ${PYSITELIB}/scrapy/http/cookies.pyo ${PYSITELIB}/scrapy/http/headers.py ${PYSITELIB}/scrapy/http/headers.pyc ${PYSITELIB}/scrapy/http/headers.pyo ${PYSITELIB}/scrapy/http/request/__init__.py ${PYSITELIB}/scrapy/http/request/__init__.pyc ${PYSITELIB}/scrapy/http/request/__init__.pyo ${PYSITELIB}/scrapy/http/request/form.py ${PYSITELIB}/scrapy/http/request/form.pyc ${PYSITELIB}/scrapy/http/request/form.pyo ${PYSITELIB}/scrapy/http/request/json_request.py ${PYSITELIB}/scrapy/http/request/json_request.pyc ${PYSITELIB}/scrapy/http/request/json_request.pyo ${PYSITELIB}/scrapy/http/request/rpc.py ${PYSITELIB}/scrapy/http/request/rpc.pyc ${PYSITELIB}/scrapy/http/request/rpc.pyo ${PYSITELIB}/scrapy/http/response/__init__.py ${PYSITELIB}/scrapy/http/response/__init__.pyc ${PYSITELIB}/scrapy/http/response/__init__.pyo ${PYSITELIB}/scrapy/http/response/html.py ${PYSITELIB}/scrapy/http/response/html.pyc ${PYSITELIB}/scrapy/http/response/html.pyo ${PYSITELIB}/scrapy/http/response/json.py ${PYSITELIB}/scrapy/http/response/json.pyc ${PYSITELIB}/scrapy/http/response/json.pyo ${PYSITELIB}/scrapy/http/response/text.py ${PYSITELIB}/scrapy/http/response/text.pyc ${PYSITELIB}/scrapy/http/response/text.pyo ${PYSITELIB}/scrapy/http/response/xml.py ${PYSITELIB}/scrapy/http/response/xml.pyc ${PYSITELIB}/scrapy/http/response/xml.pyo ${PYSITELIB}/scrapy/interfaces.py ${PYSITELIB}/scrapy/interfaces.pyc ${PYSITELIB}/scrapy/interfaces.pyo ${PYSITELIB}/scrapy/item.py ${PYSITELIB}/scrapy/item.pyc ${PYSITELIB}/scrapy/item.pyo ${PYSITELIB}/scrapy/link.py ${PYSITELIB}/scrapy/link.pyc ${PYSITELIB}/scrapy/link.pyo ${PYSITELIB}/scrapy/linkextractors/__init__.py ${PYSITELIB}/scrapy/linkextractors/__init__.pyc ${PYSITELIB}/scrapy/linkextractors/__init__.pyo ${PYSITELIB}/scrapy/linkextractors/lxmlhtml.py ${PYSITELIB}/scrapy/linkextractors/lxmlhtml.pyc ${PYSITELIB}/scrapy/linkextractors/lxmlhtml.pyo ${PYSITELIB}/scrapy/loader/__init__.py ${PYSITELIB}/scrapy/loader/__init__.pyc ${PYSITELIB}/scrapy/loader/__init__.pyo ${PYSITELIB}/scrapy/logformatter.py ${PYSITELIB}/scrapy/logformatter.pyc ${PYSITELIB}/scrapy/logformatter.pyo ${PYSITELIB}/scrapy/mail.py ${PYSITELIB}/scrapy/mail.pyc ${PYSITELIB}/scrapy/mail.pyo ${PYSITELIB}/scrapy/middleware.py ${PYSITELIB}/scrapy/middleware.pyc ${PYSITELIB}/scrapy/middleware.pyo ${PYSITELIB}/scrapy/mime.types ${PYSITELIB}/scrapy/pipelines/__init__.py ${PYSITELIB}/scrapy/pipelines/__init__.pyc ${PYSITELIB}/scrapy/pipelines/__init__.pyo ${PYSITELIB}/scrapy/pipelines/files.py ${PYSITELIB}/scrapy/pipelines/files.pyc ${PYSITELIB}/scrapy/pipelines/files.pyo ${PYSITELIB}/scrapy/pipelines/images.py ${PYSITELIB}/scrapy/pipelines/images.pyc ${PYSITELIB}/scrapy/pipelines/images.pyo ${PYSITELIB}/scrapy/pipelines/media.py ${PYSITELIB}/scrapy/pipelines/media.pyc ${PYSITELIB}/scrapy/pipelines/media.pyo ${PYSITELIB}/scrapy/pqueues.py ${PYSITELIB}/scrapy/pqueues.pyc ${PYSITELIB}/scrapy/pqueues.pyo ${PYSITELIB}/scrapy/py.typed ${PYSITELIB}/scrapy/resolver.py ${PYSITELIB}/scrapy/resolver.pyc ${PYSITELIB}/scrapy/resolver.pyo ${PYSITELIB}/scrapy/responsetypes.py ${PYSITELIB}/scrapy/responsetypes.pyc ${PYSITELIB}/scrapy/responsetypes.pyo ${PYSITELIB}/scrapy/robotstxt.py ${PYSITELIB}/scrapy/robotstxt.pyc ${PYSITELIB}/scrapy/robotstxt.pyo ${PYSITELIB}/scrapy/selector/__init__.py ${PYSITELIB}/scrapy/selector/__init__.pyc ${PYSITELIB}/scrapy/selector/__init__.pyo ${PYSITELIB}/scrapy/selector/unified.py ${PYSITELIB}/scrapy/selector/unified.pyc ${PYSITELIB}/scrapy/selector/unified.pyo ${PYSITELIB}/scrapy/settings/__init__.py ${PYSITELIB}/scrapy/settings/__init__.pyc ${PYSITELIB}/scrapy/settings/__init__.pyo ${PYSITELIB}/scrapy/settings/default_settings.py ${PYSITELIB}/scrapy/settings/default_settings.pyc ${PYSITELIB}/scrapy/settings/default_settings.pyo ${PYSITELIB}/scrapy/shell.py ${PYSITELIB}/scrapy/shell.pyc ${PYSITELIB}/scrapy/shell.pyo ${PYSITELIB}/scrapy/signalmanager.py ${PYSITELIB}/scrapy/signalmanager.pyc ${PYSITELIB}/scrapy/signalmanager.pyo ${PYSITELIB}/scrapy/signals.py ${PYSITELIB}/scrapy/signals.pyc ${PYSITELIB}/scrapy/signals.pyo ${PYSITELIB}/scrapy/spiderloader.py ${PYSITELIB}/scrapy/spiderloader.pyc ${PYSITELIB}/scrapy/spiderloader.pyo ${PYSITELIB}/scrapy/spidermiddlewares/__init__.py ${PYSITELIB}/scrapy/spidermiddlewares/__init__.pyc ${PYSITELIB}/scrapy/spidermiddlewares/__init__.pyo ${PYSITELIB}/scrapy/spidermiddlewares/base.py ${PYSITELIB}/scrapy/spidermiddlewares/base.pyc ${PYSITELIB}/scrapy/spidermiddlewares/base.pyo ${PYSITELIB}/scrapy/spidermiddlewares/depth.py ${PYSITELIB}/scrapy/spidermiddlewares/depth.pyc ${PYSITELIB}/scrapy/spidermiddlewares/depth.pyo ${PYSITELIB}/scrapy/spidermiddlewares/httperror.py ${PYSITELIB}/scrapy/spidermiddlewares/httperror.pyc ${PYSITELIB}/scrapy/spidermiddlewares/httperror.pyo ${PYSITELIB}/scrapy/spidermiddlewares/referer.py ${PYSITELIB}/scrapy/spidermiddlewares/referer.pyc ${PYSITELIB}/scrapy/spidermiddlewares/referer.pyo ${PYSITELIB}/scrapy/spidermiddlewares/start.py ${PYSITELIB}/scrapy/spidermiddlewares/start.pyc ${PYSITELIB}/scrapy/spidermiddlewares/start.pyo ${PYSITELIB}/scrapy/spidermiddlewares/urllength.py ${PYSITELIB}/scrapy/spidermiddlewares/urllength.pyc ${PYSITELIB}/scrapy/spidermiddlewares/urllength.pyo ${PYSITELIB}/scrapy/spiders/__init__.py ${PYSITELIB}/scrapy/spiders/__init__.pyc ${PYSITELIB}/scrapy/spiders/__init__.pyo ${PYSITELIB}/scrapy/spiders/crawl.py ${PYSITELIB}/scrapy/spiders/crawl.pyc ${PYSITELIB}/scrapy/spiders/crawl.pyo ${PYSITELIB}/scrapy/spiders/feed.py ${PYSITELIB}/scrapy/spiders/feed.pyc ${PYSITELIB}/scrapy/spiders/feed.pyo ${PYSITELIB}/scrapy/spiders/init.py ${PYSITELIB}/scrapy/spiders/init.pyc ${PYSITELIB}/scrapy/spiders/init.pyo ${PYSITELIB}/scrapy/spiders/sitemap.py ${PYSITELIB}/scrapy/spiders/sitemap.pyc ${PYSITELIB}/scrapy/spiders/sitemap.pyo ${PYSITELIB}/scrapy/squeues.py ${PYSITELIB}/scrapy/squeues.pyc ${PYSITELIB}/scrapy/squeues.pyo ${PYSITELIB}/scrapy/statscollectors.py ${PYSITELIB}/scrapy/statscollectors.pyc ${PYSITELIB}/scrapy/statscollectors.pyo ${PYSITELIB}/scrapy/templates/project/module/__init__.py ${PYSITELIB}/scrapy/templates/project/module/__init__.pyc ${PYSITELIB}/scrapy/templates/project/module/__init__.pyo ${PYSITELIB}/scrapy/templates/project/module/items.py.tmpl ${PYSITELIB}/scrapy/templates/project/module/middlewares.py.tmpl ${PYSITELIB}/scrapy/templates/project/module/pipelines.py.tmpl ${PYSITELIB}/scrapy/templates/project/module/settings.py.tmpl ${PYSITELIB}/scrapy/templates/project/module/spiders/__init__.py ${PYSITELIB}/scrapy/templates/project/module/spiders/__init__.pyc ${PYSITELIB}/scrapy/templates/project/module/spiders/__init__.pyo ${PYSITELIB}/scrapy/templates/project/scrapy.cfg ${PYSITELIB}/scrapy/templates/spiders/basic.tmpl ${PYSITELIB}/scrapy/templates/spiders/crawl.tmpl ${PYSITELIB}/scrapy/templates/spiders/csvfeed.tmpl ${PYSITELIB}/scrapy/templates/spiders/xmlfeed.tmpl ${PYSITELIB}/scrapy/utils/__init__.py ${PYSITELIB}/scrapy/utils/__init__.pyc ${PYSITELIB}/scrapy/utils/__init__.pyo ${PYSITELIB}/scrapy/utils/_compression.py ${PYSITELIB}/scrapy/utils/_compression.pyc ${PYSITELIB}/scrapy/utils/_compression.pyo ${PYSITELIB}/scrapy/utils/_deps_compat.py ${PYSITELIB}/scrapy/utils/_deps_compat.pyc ${PYSITELIB}/scrapy/utils/_deps_compat.pyo ${PYSITELIB}/scrapy/utils/_download_handlers.py ${PYSITELIB}/scrapy/utils/_download_handlers.pyc ${PYSITELIB}/scrapy/utils/_download_handlers.pyo ${PYSITELIB}/scrapy/utils/asyncgen.py ${PYSITELIB}/scrapy/utils/asyncgen.pyc ${PYSITELIB}/scrapy/utils/asyncgen.pyo ${PYSITELIB}/scrapy/utils/asyncio.py ${PYSITELIB}/scrapy/utils/asyncio.pyc ${PYSITELIB}/scrapy/utils/asyncio.pyo ${PYSITELIB}/scrapy/utils/benchserver.py ${PYSITELIB}/scrapy/utils/benchserver.pyc ${PYSITELIB}/scrapy/utils/benchserver.pyo ${PYSITELIB}/scrapy/utils/boto.py ${PYSITELIB}/scrapy/utils/boto.pyc ${PYSITELIB}/scrapy/utils/boto.pyo ${PYSITELIB}/scrapy/utils/conf.py ${PYSITELIB}/scrapy/utils/conf.pyc ${PYSITELIB}/scrapy/utils/conf.pyo ${PYSITELIB}/scrapy/utils/console.py ${PYSITELIB}/scrapy/utils/console.pyc ${PYSITELIB}/scrapy/utils/console.pyo ${PYSITELIB}/scrapy/utils/curl.py ${PYSITELIB}/scrapy/utils/curl.pyc ${PYSITELIB}/scrapy/utils/curl.pyo ${PYSITELIB}/scrapy/utils/datatypes.py ${PYSITELIB}/scrapy/utils/datatypes.pyc ${PYSITELIB}/scrapy/utils/datatypes.pyo ${PYSITELIB}/scrapy/utils/decorators.py ${PYSITELIB}/scrapy/utils/decorators.pyc ${PYSITELIB}/scrapy/utils/decorators.pyo ${PYSITELIB}/scrapy/utils/defer.py ${PYSITELIB}/scrapy/utils/defer.pyc ${PYSITELIB}/scrapy/utils/defer.pyo ${PYSITELIB}/scrapy/utils/deprecate.py ${PYSITELIB}/scrapy/utils/deprecate.pyc ${PYSITELIB}/scrapy/utils/deprecate.pyo ${PYSITELIB}/scrapy/utils/display.py ${PYSITELIB}/scrapy/utils/display.pyc ${PYSITELIB}/scrapy/utils/display.pyo ${PYSITELIB}/scrapy/utils/engine.py ${PYSITELIB}/scrapy/utils/engine.pyc ${PYSITELIB}/scrapy/utils/engine.pyo ${PYSITELIB}/scrapy/utils/ftp.py ${PYSITELIB}/scrapy/utils/ftp.pyc ${PYSITELIB}/scrapy/utils/ftp.pyo ${PYSITELIB}/scrapy/utils/gz.py ${PYSITELIB}/scrapy/utils/gz.pyc ${PYSITELIB}/scrapy/utils/gz.pyo ${PYSITELIB}/scrapy/utils/httpobj.py ${PYSITELIB}/scrapy/utils/httpobj.pyc ${PYSITELIB}/scrapy/utils/httpobj.pyo ${PYSITELIB}/scrapy/utils/iterators.py ${PYSITELIB}/scrapy/utils/iterators.pyc ${PYSITELIB}/scrapy/utils/iterators.pyo ${PYSITELIB}/scrapy/utils/job.py ${PYSITELIB}/scrapy/utils/job.pyc ${PYSITELIB}/scrapy/utils/job.pyo ${PYSITELIB}/scrapy/utils/log.py ${PYSITELIB}/scrapy/utils/log.pyc ${PYSITELIB}/scrapy/utils/log.pyo ${PYSITELIB}/scrapy/utils/misc.py ${PYSITELIB}/scrapy/utils/misc.pyc ${PYSITELIB}/scrapy/utils/misc.pyo ${PYSITELIB}/scrapy/utils/ossignal.py ${PYSITELIB}/scrapy/utils/ossignal.pyc ${PYSITELIB}/scrapy/utils/ossignal.pyo ${PYSITELIB}/scrapy/utils/project.py ${PYSITELIB}/scrapy/utils/project.pyc ${PYSITELIB}/scrapy/utils/project.pyo ${PYSITELIB}/scrapy/utils/python.py ${PYSITELIB}/scrapy/utils/python.pyc ${PYSITELIB}/scrapy/utils/python.pyo ${PYSITELIB}/scrapy/utils/reactor.py ${PYSITELIB}/scrapy/utils/reactor.pyc ${PYSITELIB}/scrapy/utils/reactor.pyo ${PYSITELIB}/scrapy/utils/reactorless.py ${PYSITELIB}/scrapy/utils/reactorless.pyc ${PYSITELIB}/scrapy/utils/reactorless.pyo ${PYSITELIB}/scrapy/utils/request.py ${PYSITELIB}/scrapy/utils/request.pyc ${PYSITELIB}/scrapy/utils/request.pyo ${PYSITELIB}/scrapy/utils/response.py ${PYSITELIB}/scrapy/utils/response.pyc ${PYSITELIB}/scrapy/utils/response.pyo ${PYSITELIB}/scrapy/utils/serialize.py ${PYSITELIB}/scrapy/utils/serialize.pyc ${PYSITELIB}/scrapy/utils/serialize.pyo ${PYSITELIB}/scrapy/utils/signal.py ${PYSITELIB}/scrapy/utils/signal.pyc ${PYSITELIB}/scrapy/utils/signal.pyo ${PYSITELIB}/scrapy/utils/sitemap.py ${PYSITELIB}/scrapy/utils/sitemap.pyc ${PYSITELIB}/scrapy/utils/sitemap.pyo ${PYSITELIB}/scrapy/utils/spider.py ${PYSITELIB}/scrapy/utils/spider.pyc ${PYSITELIB}/scrapy/utils/spider.pyo ${PYSITELIB}/scrapy/utils/ssl.py ${PYSITELIB}/scrapy/utils/ssl.pyc ${PYSITELIB}/scrapy/utils/ssl.pyo ${PYSITELIB}/scrapy/utils/template.py ${PYSITELIB}/scrapy/utils/template.pyc ${PYSITELIB}/scrapy/utils/template.pyo ${PYSITELIB}/scrapy/utils/test.py ${PYSITELIB}/scrapy/utils/test.pyc ${PYSITELIB}/scrapy/utils/test.pyo ${PYSITELIB}/scrapy/utils/testproc.py ${PYSITELIB}/scrapy/utils/testproc.pyc ${PYSITELIB}/scrapy/utils/testproc.pyo ${PYSITELIB}/scrapy/utils/testsite.py ${PYSITELIB}/scrapy/utils/testsite.pyc ${PYSITELIB}/scrapy/utils/testsite.pyo ${PYSITELIB}/scrapy/utils/trackref.py ${PYSITELIB}/scrapy/utils/trackref.pyc ${PYSITELIB}/scrapy/utils/trackref.pyo ${PYSITELIB}/scrapy/utils/url.py ${PYSITELIB}/scrapy/utils/url.pyc ${PYSITELIB}/scrapy/utils/url.pyo ${PYSITELIB}/scrapy/utils/versions.py ${PYSITELIB}/scrapy/utils/versions.pyc ${PYSITELIB}/scrapy/utils/versions.pyo @ 1.16 log @py-scrapy: updated to 2.14.0 Scrapy 2.14.0 (2026-01-05) Highlights: - More coroutine-based replacements for Deferred-based APIs - The default priority queue is now ``DownloaderAwarePriorityQueue`` - Dropped support for Python 3.9 and PyPy 3.10 - Improved and documented the API for custom download handlers @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.15 2025/05/15 09:21:59 adam Exp $ d85 3 d432 6 d510 3 @ 1.15 log @py-scrapy: updated to 2.13.0 Scrapy 2.13.0 (2025-05-08) Highlights: - The asyncio reactor is now enabled by default - Replaced ``start_requests()`` (sync) with :meth:`~scrapy.Spider.start` (async) and changed how it is iterated. - Added the :reqmeta:`allow_offsite` request meta key - :ref:`Spider middlewares that don't support asynchronous spider output ` are deprecated - Added a base class for :ref:`universal spider middlewares ` @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.14 2025/04/14 20:28:02 adam Exp $ d85 3 d223 3 a377 3 ${PYSITELIB}/scrapy/spidermiddlewares/offsite.py ${PYSITELIB}/scrapy/spidermiddlewares/offsite.pyc ${PYSITELIB}/scrapy/spidermiddlewares/offsite.pyo d432 3 @ 1.14 log @Fix PLIST after py-setuptools update; bump depends and revision @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.13 2024/11/30 06:56:49 adam Exp $ a8 1 ${PYSITELIB}/${WHEEL_INFODIR}/top_level.txt d363 3 d378 3 @ 1.13 log @py-scrapy: updated to 2.12.0 Scrapy 2.12.0 (2024-11-18) Highlights: - Dropped support for Python 3.8, added support for Python 3.13 - :meth:`~scrapy.Spider.start_requests` can now yield items - Added :class:`~scrapy.http.JsonResponse` - Added :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM` @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.12 2024/05/14 19:15:59 adam Exp $ a2 2 ${PYSITELIB}/${WHEEL_INFODIR}/AUTHORS ${PYSITELIB}/${WHEEL_INFODIR}/LICENSE d7 2 @ 1.12 log @py-scrapy: updated to 2.11.2 Scrapy 2.11.2 (2024-05-14) -------------------------- Security bug fixes ~~~~~~~~~~~~~~~~~~ - Redirects to non-HTTP protocols are no longer followed. Please, see the `23j4-mw76-5v7h security advisory`_ for more information. (:issue:`457`) .. _23j4-mw76-5v7h security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-23j4-mw76-5v7h - The ``Authorization`` header is now dropped on redirects to a different scheme (``http://`` or ``https://``) or port, even if the domain is the same. Please, see the `4qqq-9vqf-3h3f security advisory`_ for more information. .. _4qqq-9vqf-3h3f security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-4qqq-9vqf-3h3f - When using system proxy settings that are different for ``http://`` and ``https://``, redirects to a different URL scheme will now also trigger the corresponding change in proxy settings for the redirected request. Please, see the `jm3v-qxmh-hxwv security advisory`_ for more information. (:issue:`767`) .. _jm3v-qxmh-hxwv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-jm3v-qxmh-hxwv - :attr:`Spider.allowed_domains ` is now enforced for all requests, and not only requests from spider callbacks. (:issue:`1042`, :issue:`2241`, :issue:`6358`) - :func:`~scrapy.utils.iterators.xmliter_lxml` no longer resolves XML entities. (:issue:`6265`) - defusedxml_ is now used to make :class:`scrapy.http.request.rpc.XmlRpcRequest` more secure. (:issue:`6250`, :issue:`6251`) .. _defusedxml: https://github.com/tiran/defusedxml Bug fixes ~~~~~~~~~ - Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in favor of brotli_. (:issue:`6261`) .. _brotli: https://github.com/google/brotli .. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotli instead if you can. - Make :setting:`METAREFRESH_IGNORE_TAGS` ``["noscript"]`` by default. This prevents :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` from following redirects that would not be followed by web browsers with JavaScript enabled. (:issue:`6342`, :issue:`6347`) - During :ref:`feed export `, do not close the underlying file from :ref:`built-in post-processing plugins `. (:issue:`5932`, :issue:`6178`, :issue:`6239`) - :class:`LinkExtractor ` now properly applies the ``unique`` and ``canonicalize`` parameters. (:issue:`3273`, :issue:`6221`) - Do not initialize the scheduler disk queue if :setting:`JOBDIR` is an empty string. (:issue:`6121`, :issue:`6124`) - Fix :attr:`Spider.logger ` not logging custom extra information. (:issue:`6323`, :issue:`6324`) - ``robots.txt`` files with a non-UTF-8 encoding no longer prevent parsing the UTF-8-compatible (e.g. ASCII) parts of the document. (:issue:`6292`, :issue:`6298`) - :meth:`scrapy.http.cookies.WrappedRequest.get_header` no longer raises an exception if ``default`` is ``None``. (:issue:`6308`, :issue:`6310`) - :class:`~scrapy.selector.Selector` now uses :func:`scrapy.utils.response.get_base_url` to determine the base URL of a given :class:`~scrapy.http.Response`. (:issue:`6265`) - The :meth:`media_to_download` method of :ref:`media pipelines ` now logs exceptions before stripping them. (:issue:`5067`, :issue:`5068`) - When passing a callback to the :command:`parse` command, build the callback callable with the right signature. (:issue:`6182`) Documentation ~~~~~~~~~~~~~ - Add a FAQ entry about :ref:`creating blank requests `. (:issue:`6203`, :issue:`6208`) - Document that :attr:`scrapy.selector.Selector.type` can be ``"json"``. (:issue:`6328`, :issue:`6334`) Quality assurance ~~~~~~~~~~~~~~~~~ - Make builds reproducible. (:issue:`5019`, :issue:`6322`) - Packaging and test fixes. @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.11 2024/02/16 19:02:45 adam Exp $ a250 3 ${PYSITELIB}/scrapy/http/common.py ${PYSITELIB}/scrapy/http/common.pyc ${PYSITELIB}/scrapy/http/common.pyo d275 3 a301 6 ${PYSITELIB}/scrapy/loader/common.py ${PYSITELIB}/scrapy/loader/common.pyc ${PYSITELIB}/scrapy/loader/common.pyo ${PYSITELIB}/scrapy/loader/processors.py ${PYSITELIB}/scrapy/loader/processors.pyc ${PYSITELIB}/scrapy/loader/processors.pyo d327 1 @ 1.11 log @py-scrapy: updated to 2.11.1 Scrapy 2.11.1 (2024-02-14) -------------------------- Highlights: - Security bug fixes. - Support for Twisted >= 23.8.0. - Documentation improvements. Security bug fixes ~~~~~~~~~~~~~~~~~~ - Addressed `ReDoS vulnerabilities`_: - ``scrapy.utils.iterators.xmliter`` is now deprecated in favor of :func:`~scrapy.utils.iterators.xmliter_lxml`, which :class:`~scrapy.spiders.XMLFeedSpider` now uses. To minimize the impact of this change on existing code, :func:`~scrapy.utils.iterators.xmliter_lxml` now supports indicating the node namespace with a prefix in the node name, and big files with highly nested trees when using libxml2 2.7+. - Fixed regular expressions in the implementation of the :func:`~scrapy.utils.response.open_in_browser` function. Please, see the `cc65-xxvf-f7r9 security advisory`_ for more information. .. _ReDoS vulnerabilities: https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS .. _cc65-xxvf-f7r9 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cc65-xxvf-f7r9 - :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security advisory`_ for more information. .. _7j7m-v7m3-jqm7 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-7j7m-v7m3-jqm7 - Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, the deprecated ``scrapy.downloadermiddlewares.decompression`` module has been removed. - The ``Authorization`` header is now dropped on redirects to a different domain. Please, see the `cw9j-q3vf-hrrv security advisory`_ for more information. .. _cw9j-q3vf-hrrv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cw9j-q3vf-hrrv Modified requirements ~~~~~~~~~~~~~~~~~~~~~ - The Twisted dependency is no longer restricted to < 23.8.0. (:issue:`6024`, :issue:`6064`, :issue:`6142`) Bug fixes ~~~~~~~~~ - The OS signal handling code was refactored to no longer use private Twisted functions. (:issue:`6024`, :issue:`6064`, :issue:`6112`) Documentation ~~~~~~~~~~~~~ - Improved documentation for :class:`~scrapy.crawler.Crawler` initialization changes made in the 2.11.0 release. (:issue:`6057`, :issue:`6147`) - Extended documentation for :attr:`Request.meta `. (:issue:`5565`) - Fixed the :reqmeta:`dont_merge_cookies` documentation. (:issue:`5936`, :issue:`6077`) - Added a link to Zyte's export guides to the :ref:`feed exports ` documentation. (:issue:`6183`) - Added a missing note about backward-incompatible changes in :class:`~scrapy.exporters.PythonItemExporter` to the 2.11.0 release notes. (:issue:`6060`, :issue:`6081`) - Added a missing note about removing the deprecated ``scrapy.utils.boto.is_botocore()`` function to the 2.8.0 release notes. (:issue:`6056`, :issue:`6061`) - Other documentation improvements. (:issue:`6128`, :issue:`6144`, :issue:`6163`, :issue:`6190`, :issue:`6192`) Quality assurance ~~~~~~~~~~~~~~~~~ - Added Python 3.12 to the CI configuration, re-enabled tests that were disabled when the pre-release support was added. (:issue:`5985`, :issue:`6083`, :issue:`6098`) - Fixed a test issue on PyPy 7.3.14. (:issue:`6204`, :issue:`6205`) @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.10 2023/10/10 17:18:23 triaxx Exp $ d173 3 @ 1.10 log @py-scrapy: Update to 2.11.0 upstream changes: ----------------- * 2.11.0: https://docs.scrapy.org/en/latest/news.html#scrapy-2-11-0-2023-09-18 * 2.10.0: https://docs.scrapy.org/en/2.10/news.html#scrapy-2-10-0-2023-08-04 @ text @d1 1 a1 1 @@comment $NetBSD$ d3 7 a9 7 ${PYSITELIB}/${EGG_INFODIR}/PKG-INFO ${PYSITELIB}/${EGG_INFODIR}/SOURCES.txt ${PYSITELIB}/${EGG_INFODIR}/dependency_links.txt ${PYSITELIB}/${EGG_INFODIR}/entry_points.txt ${PYSITELIB}/${EGG_INFODIR}/not-zip-safe ${PYSITELIB}/${EGG_INFODIR}/requires.txt ${PYSITELIB}/${EGG_INFODIR}/top_level.txt a154 3 ${PYSITELIB}/scrapy/downloadermiddlewares/decompression.py ${PYSITELIB}/scrapy/downloadermiddlewares/decompression.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/decompression.pyo d420 3 @ 1.9 log @py-scrapy: updated to 2.8.0 Scrapy 2.8.0 (2023-02-02) ------------------------- This is a maintenance release, with minor features, bug fixes, and cleanups. Deprecation removals ~~~~~~~~~~~~~~~~~~~~ - The ``scrapy.utils.gz.read1`` function, deprecated in Scrapy 2.0, has now been removed. Use the :meth:`~io.BufferedIOBase.read1` method of :class:`~gzip.GzipFile` instead. - The ``scrapy.utils.python.to_native_str`` function, deprecated in Scrapy 2.0, has now been removed. Use :func:`scrapy.utils.python.to_unicode` instead. - The ``scrapy.utils.python.MutableChain.next`` method, deprecated in Scrapy 2.0, has now been removed. Use :meth:`~scrapy.utils.python.MutableChain.__next__` instead. - The ``scrapy.linkextractors.FilteringLinkExtractor`` class, deprecated in Scrapy 2.0, has now been removed. Use :class:`LinkExtractor ` instead. - Support for using environment variables prefixed with ``SCRAPY_`` to override settings, deprecated in Scrapy 2.0, has now been removed. - Support for the ``noconnect`` query string argument in proxy URLs, deprecated in Scrapy 2.0, has now been removed. We expect proxies that used to need it to work fine without it. - The ``scrapy.utils.python.retry_on_eintr`` function, deprecated in Scrapy 2.3, has now been removed. - The ``scrapy.utils.python.WeakKeyCache`` class, deprecated in Scrapy 2.4, has now been removed. Deprecations ~~~~~~~~~~~~ - :exc:`scrapy.pipelines.images.NoimagesDrop` is now deprecated. - :meth:`ImagesPipeline.convert_image ` must now accept a ``response_body`` parameter. New features ~~~~~~~~~~~~ - Applied black_ coding style to files generated with the :command:`genspider` and :command:`startproject` commands. .. _black: https://black.readthedocs.io/en/stable/ - :setting:`FEED_EXPORT_ENCODING` is now set to ``"utf-8"`` in the ``settings.py`` file that the :command:`startproject` command generates. With this value, JSON exports won’t force the use of escape sequences for non-ASCII characters. - The :class:`~scrapy.extensions.memusage.MemoryUsage` extension now logs the peak memory usage during checks, and the binary unit MiB is now used to avoid confusion. - The ``callback`` parameter of :class:`~scrapy.http.Request` can now be set to :func:`scrapy.http.request.NO_CALLBACK`, to distinguish it from ``None``, as the latter indicates that the default spider callback (:meth:`~scrapy.Spider.parse`) is to be used. Bug fixes ~~~~~~~~~ - Enabled unsafe legacy SSL renegotiation to fix access to some outdated websites. - Fixed STARTTLS-based email delivery not working with Twisted 21.2.0 and better. - Fixed the :meth:`finish_exporting` method of :ref:`item exporters ` not being called for empty files. - Fixed HTTP/2 responses getting only the last value for a header when multiple headers with the same name are received. - Fixed an exception raised by the :command:`shell` command on some cases when :ref:`using asyncio `. - When using :class:`~scrapy.spiders.CrawlSpider`, callback keyword arguments (``cb_kwargs``) added to a request in the ``process_request`` callback of a :class:`~scrapy.spiders.Rule` will no longer be ignored. - The :ref:`images pipeline ` no longer re-encodes JPEG files. - Fixed the handling of transparent WebP images by the :ref:`images pipeline `. - :func:`scrapy.shell.inspect_response` no longer inhibits ``SIGINT`` (Ctrl+C). - :class:`LinkExtractor ` with ``unique=False`` no longer filters out links that have identical URL *and* text. - :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now ignores URL protocols that do not support ``robots.txt`` (``data://``, ``file://``). - Silenced the ``filelock`` debug log messages introduced in Scrapy 2.6. - Fixed the output of ``scrapy -h`` showing an unintended ``**commands**`` line. - Made the active project indication in the output of :ref:`commands ` more clear. Documentation ~~~~~~~~~~~~~ - Documented how to :ref:`debug spiders from Visual Studio Code `. - Documented how :setting:`DOWNLOAD_DELAY` affects per-domain concurrency. - Improved consistency. - Fixed typos. Quality assurance ~~~~~~~~~~~~~~~~~ - Applied :ref:`black coding style `, sorted import statements, and introduced :ref:`pre-commit `. - Switched from :mod:`os.path` to :mod:`pathlib`. - Addressed many issues reported by Pylint. - Improved code readability. - Improved package metadata. - Removed direct invocations of ``setup.py``. - Removed unnecessary :class:`~collections.OrderedDict` usages. - Removed unnecessary ``__str__`` definitions. - Removed obsolete code and comments. - Fixed test and CI issues. @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.8 2021/03/22 08:56:56 triaxx Exp $ d17 3 d230 3 a491 3 ${PYSITELIB}/scrapy/utils/reqser.py ${PYSITELIB}/scrapy/utils/reqser.pyc ${PYSITELIB}/scrapy/utils/reqser.pyo @ 1.8 log @py-scrapy: Update to 2.4.1 upstream cheanges: ------------------ A lot of changes listed at https://github.com/scrapy/scrapy/blob/master/docs/news.rst @ text @d1 1 a1 1 @@comment $NetBSD$ d101 3 d119 12 d227 3 d417 3 a458 3 ${PYSITELIB}/scrapy/utils/http.py ${PYSITELIB}/scrapy/utils/http.pyc ${PYSITELIB}/scrapy/utils/http.pyo a470 3 ${PYSITELIB}/scrapy/utils/markup.py ${PYSITELIB}/scrapy/utils/markup.pyc ${PYSITELIB}/scrapy/utils/markup.pyo a473 3 ${PYSITELIB}/scrapy/utils/multipart.py ${PYSITELIB}/scrapy/utils/multipart.pyc ${PYSITELIB}/scrapy/utils/multipart.pyo a479 3 ${PYSITELIB}/scrapy/utils/py36.py ${PYSITELIB}/scrapy/utils/py36.pyc ${PYSITELIB}/scrapy/utils/py36.pyo @ 1.7 log @py-scrapy: updated to 1.8.0 Scrapy 1.8.0: Highlights: * Dropped Python 3.4 support and updated minimum requirements; made Python 3.8 support official * New :meth:`Request.from_curl ` class method * New :setting:`ROBOTSTXT_PARSER` and :setting:`ROBOTSTXT_USER_AGENT` settings * New :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` settings @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.6 2019/08/22 08:21:11 adam Exp $ a16 3 ${PYSITELIB}/scrapy/_monkeypatches.py ${PYSITELIB}/scrapy/_monkeypatches.pyc ${PYSITELIB}/scrapy/_monkeypatches.pyo a133 3 ${PYSITELIB}/scrapy/downloadermiddlewares/chunked.py ${PYSITELIB}/scrapy/downloadermiddlewares/chunked.pyc ${PYSITELIB}/scrapy/downloadermiddlewares/chunked.pyo a271 3 ${PYSITELIB}/scrapy/linkextractors/htmlparser.py ${PYSITELIB}/scrapy/linkextractors/htmlparser.pyc ${PYSITELIB}/scrapy/linkextractors/htmlparser.pyo a274 6 ${PYSITELIB}/scrapy/linkextractors/regex.py ${PYSITELIB}/scrapy/linkextractors/regex.pyc ${PYSITELIB}/scrapy/linkextractors/regex.pyo ${PYSITELIB}/scrapy/linkextractors/sgml.py ${PYSITELIB}/scrapy/linkextractors/sgml.pyc ${PYSITELIB}/scrapy/linkextractors/sgml.pyo a329 3 ${PYSITELIB}/scrapy/settings/deprecated.py ${PYSITELIB}/scrapy/settings/deprecated.pyc ${PYSITELIB}/scrapy/settings/deprecated.pyo d468 3 @ 1.6 log @py-scrapy: updated to 1.7.3 Scrapy 1.7.3: Enforce lxml 4.3.5 or lower for Python 3.4 (issue 3912, issue 3918). Scrapy 1.7.2: Fix Python 2 support (issue 3889, issue 3893, issue 3896). Scrapy 1.7.1: Re-packaging of Scrapy 1.7.0, which was missing some changes in PyPI. Scrapy 1.7.0: Highlights: Improvements for crawls targeting multiple domains A cleaner way to pass arguments to callbacks A new class for JSON requests Improvements for rule-based spiders New features for feed exports Backward-incompatible changes 429 is now part of the RETRY_HTTP_CODES setting by default This change is backward incompatible. If you don’t want to retry 429, you must override RETRY_HTTP_CODES accordingly. Crawler, CrawlerRunner.crawl and CrawlerRunner.create_crawler no longer accept a Spider subclass instance, they only accept a Spider subclass now. Spider subclass instances were never meant to work, and they were not working as one would expect: instead of using the passed Spider subclass instance, their from_crawler method was called to generate a new instance. Non-default values for the SCHEDULER_PRIORITY_QUEUE setting may stop working. Scheduler priority queue classes now need to handle Request objects instead of arbitrary Python data structures. New features A new scheduler priority queue, scrapy.pqueues.DownloaderAwarePriorityQueue, may be enabled for a significant scheduling improvement on crawls targetting multiple web domains, at the cost of no CONCURRENT_REQUESTS_PER_IP support (issue 3520) A new Request.cb_kwargs attribute provides a cleaner way to pass keyword arguments to callback methods (issue 1138, issue 3563) A new JSONRequest class offers a more convenient way to build JSON requests (issue 3504, issue 3505) A process_request callback passed to the Rule constructor now receives the Response object that originated the request as its second argument (issue 3682) A new restrict_text parameter for the LinkExtractor constructor allows filtering links by linking text (issue 3622, issue 3635) A new FEED_STORAGE_S3_ACL setting allows defining a custom ACL for feeds exported to Amazon S3 (issue 3607) A new FEED_STORAGE_FTP_ACTIVE setting allows using FTP’s active connection mode for feeds exported to FTP servers (issue 3829) A new METAREFRESH_IGNORE_TAGS setting allows overriding which HTML tags are ignored when searching a response for HTML meta tags that trigger a redirect (issue 1422, issue 3768) A new redirect_reasons request meta key exposes the reason (status code, meta refresh) behind every followed redirect (issue 3581, issue 3687) The SCRAPY_CHECK variable is now set to the true string during runs of the check command, which allows detecting contract check runs from code (issue 3704, issue 3739) A new Item.deepcopy() method makes it easier to deep-copy items (issue 1493, issue 3671) CoreStats also logs elapsed_time_seconds now (issue 3638) Exceptions from ItemLoader input and output processors are now more verbose (issue 3836, issue 3840) Crawler, CrawlerRunner.crawl and CrawlerRunner.create_crawler now fail gracefully if they receive a Spider subclass instance instead of the subclass itself (issue 2283, issue 3610, issue 3872) Bug fixes process_spider_exception() is now also invoked for generators (issue 220, issue 2061) System exceptions like KeyboardInterrupt are no longer caught (issue 3726) ItemLoader.load_item() no longer makes later calls to ItemLoader.get_output_value() or ItemLoader.load_item() return empty data (issue 3804, issue 3819) The images pipeline (ImagesPipeline) no longer ignores these Amazon S3 settings: AWS_ENDPOINT_URL, AWS_REGION_NAME, AWS_USE_SSL, AWS_VERIFY (issue 3625) Fixed a memory leak in MediaPipeline affecting, for example, non-200 responses and exceptions from custom middlewares (issue 3813) Requests with private callbacks are now correctly unserialized from disk (issue 3790) FormRequest.from_response() now handles invalid methods like major web browsers @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.5 2019/01/31 09:07:46 adam Exp $ d330 3 d429 3 d513 3 a536 9 ${PYSITELIB}/scrapy/xlib/__init__.py ${PYSITELIB}/scrapy/xlib/__init__.pyc ${PYSITELIB}/scrapy/xlib/__init__.pyo ${PYSITELIB}/scrapy/xlib/pydispatch.py ${PYSITELIB}/scrapy/xlib/pydispatch.pyc ${PYSITELIB}/scrapy/xlib/pydispatch.pyo ${PYSITELIB}/scrapy/xlib/tx.py ${PYSITELIB}/scrapy/xlib/tx.pyc ${PYSITELIB}/scrapy/xlib/tx.pyo @ 1.5 log @py-scrapy: updated to 1.6.0 Scrapy 1.6.0: Highlights: * better Windows support; * Python 3.7 compatibility; * big documentation improvements, including a switch from .extract_first() + .extract() API to .get() + .getall() API; * feed exports, FilePipeline and MediaPipeline improvements; * better extensibility: :signal:item_error and :signal:request_reached_downloader signals; from_crawler support for feed exporters, feed storages and dupefilters. * scrapy.contracts fixes and new features; * telnet console security improvements, first released as a backport in :ref:release-1.5.2; * clean-up of the deprecated code; * various bug fixes, small new features and usability improvements across the codebase. @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.4 2019/01/24 14:11:48 adam Exp $ a67 3 ${PYSITELIB}/scrapy/conf.py ${PYSITELIB}/scrapy/conf.pyc ${PYSITELIB}/scrapy/conf.pyo d248 3 a298 3 ${PYSITELIB}/scrapy/log.py ${PYSITELIB}/scrapy/log.pyc ${PYSITELIB}/scrapy/log.pyo d321 3 a332 6 ${PYSITELIB}/scrapy/selector/csstranslator.py ${PYSITELIB}/scrapy/selector/csstranslator.pyc ${PYSITELIB}/scrapy/selector/csstranslator.pyo ${PYSITELIB}/scrapy/selector/lxmlsel.py ${PYSITELIB}/scrapy/selector/lxmlsel.pyc ${PYSITELIB}/scrapy/selector/lxmlsel.pyo a395 3 ${PYSITELIB}/scrapy/telnet.py ${PYSITELIB}/scrapy/telnet.pyc ${PYSITELIB}/scrapy/telnet.pyo @ 1.4 log @py-scrapy: updated to 1.5.2 Scrapy 1.5.2: * *Security bugfix*: Telnet console extension can be easily exploited by rogue websites POSTing content to http://localhost:6023, we haven't found a way to exploit it from Scrapy, but it is very easy to trick a browser to do so and elevates the risk for local development environment. *The fix is backwards incompatible*, it enables telnet user-password authentication by default with a random generated password. If you can't upgrade right away, please consider setting :setting:TELNET_CONSOLE_PORT out of its default value. See :ref:telnet console documentation for more info * Backport CI build failure under GCE environemnt due to boto import error. @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.3 2018/01/04 21:31:41 adam Exp $ a22 3 ${PYSITELIB}/scrapy/command.py ${PYSITELIB}/scrapy/command.pyc ${PYSITELIB}/scrapy/command.pyo a76 168 ${PYSITELIB}/scrapy/contrib/__init__.py ${PYSITELIB}/scrapy/contrib/__init__.pyc ${PYSITELIB}/scrapy/contrib/__init__.pyo ${PYSITELIB}/scrapy/contrib/closespider.py ${PYSITELIB}/scrapy/contrib/closespider.pyc ${PYSITELIB}/scrapy/contrib/closespider.pyo ${PYSITELIB}/scrapy/contrib/corestats.py ${PYSITELIB}/scrapy/contrib/corestats.pyc ${PYSITELIB}/scrapy/contrib/corestats.pyo ${PYSITELIB}/scrapy/contrib/debug.py ${PYSITELIB}/scrapy/contrib/debug.pyc ${PYSITELIB}/scrapy/contrib/debug.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/__init__.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/__init__.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/__init__.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/ajaxcrawl.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/ajaxcrawl.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/ajaxcrawl.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/chunked.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/chunked.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/chunked.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/cookies.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/cookies.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/cookies.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/decompression.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/decompression.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/decompression.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/defaultheaders.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/defaultheaders.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/defaultheaders.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/downloadtimeout.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/downloadtimeout.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/downloadtimeout.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpauth.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpauth.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpauth.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcache.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcache.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcache.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcompression.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcompression.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcompression.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpproxy.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpproxy.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpproxy.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/redirect.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/redirect.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/redirect.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/retry.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/retry.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/retry.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/robotstxt.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/robotstxt.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/robotstxt.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/stats.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/stats.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/stats.pyo ${PYSITELIB}/scrapy/contrib/downloadermiddleware/useragent.py ${PYSITELIB}/scrapy/contrib/downloadermiddleware/useragent.pyc ${PYSITELIB}/scrapy/contrib/downloadermiddleware/useragent.pyo ${PYSITELIB}/scrapy/contrib/exporter/__init__.py ${PYSITELIB}/scrapy/contrib/exporter/__init__.pyc ${PYSITELIB}/scrapy/contrib/exporter/__init__.pyo ${PYSITELIB}/scrapy/contrib/feedexport.py ${PYSITELIB}/scrapy/contrib/feedexport.pyc ${PYSITELIB}/scrapy/contrib/feedexport.pyo ${PYSITELIB}/scrapy/contrib/httpcache.py ${PYSITELIB}/scrapy/contrib/httpcache.pyc ${PYSITELIB}/scrapy/contrib/httpcache.pyo ${PYSITELIB}/scrapy/contrib/linkextractors/__init__.py ${PYSITELIB}/scrapy/contrib/linkextractors/__init__.pyc ${PYSITELIB}/scrapy/contrib/linkextractors/__init__.pyo ${PYSITELIB}/scrapy/contrib/linkextractors/htmlparser.py ${PYSITELIB}/scrapy/contrib/linkextractors/htmlparser.pyc ${PYSITELIB}/scrapy/contrib/linkextractors/htmlparser.pyo ${PYSITELIB}/scrapy/contrib/linkextractors/lxmlhtml.py ${PYSITELIB}/scrapy/contrib/linkextractors/lxmlhtml.pyc ${PYSITELIB}/scrapy/contrib/linkextractors/lxmlhtml.pyo ${PYSITELIB}/scrapy/contrib/linkextractors/regex.py ${PYSITELIB}/scrapy/contrib/linkextractors/regex.pyc ${PYSITELIB}/scrapy/contrib/linkextractors/regex.pyo ${PYSITELIB}/scrapy/contrib/linkextractors/sgml.py ${PYSITELIB}/scrapy/contrib/linkextractors/sgml.pyc ${PYSITELIB}/scrapy/contrib/linkextractors/sgml.pyo ${PYSITELIB}/scrapy/contrib/loader/__init__.py ${PYSITELIB}/scrapy/contrib/loader/__init__.pyc ${PYSITELIB}/scrapy/contrib/loader/__init__.pyo ${PYSITELIB}/scrapy/contrib/loader/common.py ${PYSITELIB}/scrapy/contrib/loader/common.pyc ${PYSITELIB}/scrapy/contrib/loader/common.pyo ${PYSITELIB}/scrapy/contrib/loader/processor.py ${PYSITELIB}/scrapy/contrib/loader/processor.pyc ${PYSITELIB}/scrapy/contrib/loader/processor.pyo ${PYSITELIB}/scrapy/contrib/logstats.py ${PYSITELIB}/scrapy/contrib/logstats.pyc ${PYSITELIB}/scrapy/contrib/logstats.pyo ${PYSITELIB}/scrapy/contrib/memdebug.py ${PYSITELIB}/scrapy/contrib/memdebug.pyc ${PYSITELIB}/scrapy/contrib/memdebug.pyo ${PYSITELIB}/scrapy/contrib/memusage.py ${PYSITELIB}/scrapy/contrib/memusage.pyc ${PYSITELIB}/scrapy/contrib/memusage.pyo ${PYSITELIB}/scrapy/contrib/pipeline/__init__.py ${PYSITELIB}/scrapy/contrib/pipeline/__init__.pyc ${PYSITELIB}/scrapy/contrib/pipeline/__init__.pyo ${PYSITELIB}/scrapy/contrib/pipeline/files.py ${PYSITELIB}/scrapy/contrib/pipeline/files.pyc ${PYSITELIB}/scrapy/contrib/pipeline/files.pyo ${PYSITELIB}/scrapy/contrib/pipeline/images.py ${PYSITELIB}/scrapy/contrib/pipeline/images.pyc ${PYSITELIB}/scrapy/contrib/pipeline/images.pyo ${PYSITELIB}/scrapy/contrib/pipeline/media.py ${PYSITELIB}/scrapy/contrib/pipeline/media.pyc ${PYSITELIB}/scrapy/contrib/pipeline/media.pyo ${PYSITELIB}/scrapy/contrib/spidermiddleware/__init__.py ${PYSITELIB}/scrapy/contrib/spidermiddleware/__init__.pyc ${PYSITELIB}/scrapy/contrib/spidermiddleware/__init__.pyo ${PYSITELIB}/scrapy/contrib/spidermiddleware/depth.py ${PYSITELIB}/scrapy/contrib/spidermiddleware/depth.pyc ${PYSITELIB}/scrapy/contrib/spidermiddleware/depth.pyo ${PYSITELIB}/scrapy/contrib/spidermiddleware/httperror.py ${PYSITELIB}/scrapy/contrib/spidermiddleware/httperror.pyc ${PYSITELIB}/scrapy/contrib/spidermiddleware/httperror.pyo ${PYSITELIB}/scrapy/contrib/spidermiddleware/offsite.py ${PYSITELIB}/scrapy/contrib/spidermiddleware/offsite.pyc ${PYSITELIB}/scrapy/contrib/spidermiddleware/offsite.pyo ${PYSITELIB}/scrapy/contrib/spidermiddleware/referer.py ${PYSITELIB}/scrapy/contrib/spidermiddleware/referer.pyc ${PYSITELIB}/scrapy/contrib/spidermiddleware/referer.pyo ${PYSITELIB}/scrapy/contrib/spidermiddleware/urllength.py ${PYSITELIB}/scrapy/contrib/spidermiddleware/urllength.pyc ${PYSITELIB}/scrapy/contrib/spidermiddleware/urllength.pyo ${PYSITELIB}/scrapy/contrib/spiders/__init__.py ${PYSITELIB}/scrapy/contrib/spiders/__init__.pyc ${PYSITELIB}/scrapy/contrib/spiders/__init__.pyo ${PYSITELIB}/scrapy/contrib/spiders/crawl.py ${PYSITELIB}/scrapy/contrib/spiders/crawl.pyc ${PYSITELIB}/scrapy/contrib/spiders/crawl.pyo ${PYSITELIB}/scrapy/contrib/spiders/feed.py ${PYSITELIB}/scrapy/contrib/spiders/feed.pyc ${PYSITELIB}/scrapy/contrib/spiders/feed.pyo ${PYSITELIB}/scrapy/contrib/spiders/init.py ${PYSITELIB}/scrapy/contrib/spiders/init.pyc ${PYSITELIB}/scrapy/contrib/spiders/init.pyo ${PYSITELIB}/scrapy/contrib/spiders/sitemap.py ${PYSITELIB}/scrapy/contrib/spiders/sitemap.pyc ${PYSITELIB}/scrapy/contrib/spiders/sitemap.pyo ${PYSITELIB}/scrapy/contrib/spiderstate.py ${PYSITELIB}/scrapy/contrib/spiderstate.pyc ${PYSITELIB}/scrapy/contrib/spiderstate.pyo ${PYSITELIB}/scrapy/contrib/statsmailer.py ${PYSITELIB}/scrapy/contrib/statsmailer.pyc ${PYSITELIB}/scrapy/contrib/statsmailer.pyo ${PYSITELIB}/scrapy/contrib/throttle.py ${PYSITELIB}/scrapy/contrib/throttle.pyc ${PYSITELIB}/scrapy/contrib/throttle.pyo ${PYSITELIB}/scrapy/contrib_exp/__init__.py ${PYSITELIB}/scrapy/contrib_exp/__init__.pyc ${PYSITELIB}/scrapy/contrib_exp/__init__.pyo ${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/__init__.py ${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/__init__.pyc ${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/__init__.pyo ${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/decompression.py ${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/decompression.pyc ${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/decompression.pyo ${PYSITELIB}/scrapy/contrib_exp/iterators.py ${PYSITELIB}/scrapy/contrib_exp/iterators.pyc ${PYSITELIB}/scrapy/contrib_exp/iterators.pyo a181 3 ${PYSITELIB}/scrapy/dupefilter.py ${PYSITELIB}/scrapy/dupefilter.pyc ${PYSITELIB}/scrapy/dupefilter.pyo a274 3 ${PYSITELIB}/scrapy/linkextractor.py ${PYSITELIB}/scrapy/linkextractor.pyc ${PYSITELIB}/scrapy/linkextractor.pyo a323 3 ${PYSITELIB}/scrapy/project.py ${PYSITELIB}/scrapy/project.pyc ${PYSITELIB}/scrapy/project.pyo a359 3 ${PYSITELIB}/scrapy/spider.py ${PYSITELIB}/scrapy/spider.pyc ${PYSITELIB}/scrapy/spider.pyo a362 3 ${PYSITELIB}/scrapy/spidermanager.py ${PYSITELIB}/scrapy/spidermanager.pyc ${PYSITELIB}/scrapy/spidermanager.pyo a395 3 ${PYSITELIB}/scrapy/squeue.py ${PYSITELIB}/scrapy/squeue.pyc ${PYSITELIB}/scrapy/squeue.pyo a398 6 ${PYSITELIB}/scrapy/stats.py ${PYSITELIB}/scrapy/stats.pyc ${PYSITELIB}/scrapy/stats.pyo ${PYSITELIB}/scrapy/statscol.py ${PYSITELIB}/scrapy/statscol.pyc ${PYSITELIB}/scrapy/statscol.pyo a437 3 ${PYSITELIB}/scrapy/utils/decorator.py ${PYSITELIB}/scrapy/utils/decorator.pyc ${PYSITELIB}/scrapy/utils/decorator.pyo @ 1.3 log @py-scrapy: updated to 1.5.0 Scrapy 1.5.0: This release brings small new features and improvements across the codebase. Some highlights: * Google Cloud Storage is supported in FilesPipeline and ImagesPipeline. * Crawling with proxy servers becomes more efficient, as connections to proxies can be reused now. * Warnings, exception and logging messages are improved to make debugging easier. * scrapy parse command now allows to set custom request meta via --meta argument. * Compatibility with Python 3.6, PyPy and PyPy3 is improved; PyPy and PyPy3 are now supported officially, by running tests on CI. * Better default handling of HTTP 308, 522 and 524 status codes. * Documentation is improved, as usual. Backwards Incompatible Changes * Scrapy 1.5 drops support for Python 3.3. * Default Scrapy User-Agent now uses https link to scrapy.org. **This is technically backwards-incompatible**; override :setting:USER_AGENT if you relied on old value. * Logging of settings overridden by custom_settings is fixed; **this is technically backwards-incompatible** because the logger changes from [scrapy.utils.log] to [scrapy.crawler]. If you're parsing Scrapy logs, please update your log parsers. * LinkExtractor now ignores m4v extension by default, this is change in behavior. * 522 and 524 status codes are added to RETRY_HTTP_CODES New features - Support tags in Response.follow - Support for ptpython REPL - Google Cloud Storage support for FilesPipeline and ImagesPipeline - New --meta option of the "scrapy parse" command allows to pass additional request.meta - Populate spider variable when using shell.inspect_response - Handle HTTP 308 Permanent Redirect - Add 522 and 524 to RETRY_HTTP_CODES - Log versions information at startup - scrapy.mail.MailSender now works in Python 3 (it requires Twisted 17.9.0) - Connections to proxy servers are reused - Add template for a downloader middleware - Explicit message for NotImplementedError when parse callback not defined - CrawlerProcess got an option to disable installation of root log handler - LinkExtractor now ignores m4v extension by default - Better log messages for responses over :setting:DOWNLOAD_WARNSIZE and :setting:DOWNLOAD_MAXSIZE limits - Show warning when a URL is put to Spider.allowed_domains instead of a domain. Bug fixes - Fix logging of settings overridden by custom_settings; **this is technically backwards-incompatible** because the logger changes from [scrapy.utils.log] to [scrapy.crawler], so please update your log parsers if needed - Default Scrapy User-Agent now uses https link to scrapy.org. **This is technically backwards-incompatible**; override :setting:USER_AGENT if you relied on old value. - Fix PyPy and PyPy3 test failures, support them officially - Fix DNS resolver when DNSCACHE_ENABLED=False - Add cryptography for Debian Jessie tox test env - Add verification to check if Request callback is callable - Port extras/qpsclient.py to Python 3 - Use getfullargspec under the scenes for Python 3 to stop DeprecationWarning - Update deprecated test aliases - Fix SitemapSpider support for alternate links @ text @d1 2 a2 2 @@comment $NetBSD: PLIST,v 1.2 2017/05/20 06:25:36 adam Exp $ bin/scrapy${PYVERSSUFFIX} @ 1.2 log @Scrapy 1.4 does not bring that many breathtaking new features but quite a few handy improvements nonetheless. Scrapy now supports anonymous FTP sessions with customizable user and password via the new :setting:`FTP_USER` and :setting:`FTP_PASSWORD` settings. And if you're using Twisted version 17.1.0 or above, FTP is now available with Python 3. There's a new :meth:`response.follow ` method for creating requests; **it is now a recommended way to create Requests in Scrapy spiders**. This method makes it easier to write correct spiders; ``response.follow`` has several advantages over creating ``scrapy.Request`` objects directly: * it handles relative URLs; * it works properly with non-ascii URLs on non-UTF8 pages; * in addition to absolute and relative URLs it supports Selectors; for ```` elements it can also extract their href values. @ text @d1 1 a1 1 @@comment $NetBSD: PLIST,v 1.1 2017/02/13 21:25:33 adam Exp $ d732 3 @ 1.1 log @Added www/py-scrapy version 1.3.2 Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. @ text @d1 2 a2 2 @@comment $NetBSD$ bin/scrapy d14 3 d260 3 @