Head
2026-05-29 23:27:28 [scrapy.utils.log] INFO: Scrapy 2.10.0 started (bot: retriever)
2026-05-29 23:27:28 [scrapy.utils.log] INFO: Versions: lxml 5.1.0.0, libxml2 2.12.3, cssselect 1.2.0, parsel 1.9.0, w3lib 2.0.0, Twisted 22.4.0, Python 3.10.12 (main, Mar 3 2026, 11:56:32) [GCC 11.4.0], pyOpenSSL 24.1.0 (OpenSSL 3.2.1 30 Jan 2024), cryptography 42.0.5, Platform Linux-6.8.0-1030-aws-x86_64-with-glibc2.35
2026-05-29 23:27:28 [scrapy.addons] INFO: Enabled addons:
[]
2026-05-29 23:27:28 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_ENABLED': True,
'BOT_NAME': 'retriever',
'CONCURRENT_REQUESTS_PER_DOMAIN': 3,
'DOWNLOAD_DELAY': 2.5,
'HTTPCACHE_EXPIRATION_SECS': 86400,
'IMAGES_STORE_S3_ACL': 'public-read',
'LOG_FILE': '/home/scrapyd/logs/loknow_spiders_prod/CartercadillacSpider/f31dbb705bb511f1bdc0c3bbae71ecef.log',
'LOG_FORMATTER': 'loknow_spiders.logging.PoliteLogFormatter',
'LOG_LEVEL': 'INFO',
'NEWSPIDER_MODULE': 'loknow_spiders.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['loknow_spiders.spiders'],
'TELNETCONSOLE_ENABLED': False,
'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
'USER_AGENT': 'AdRetriever (https://adretriever.com)'}
2026-05-29 23:27:28 [py.warnings] WARNING: /home/scrapyd/venv/lib/python3.10/site-packages/scrapy/utils/request.py:248: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.
It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy.
See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
return cls(crawler)
2026-05-29 23:27:28 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats',
'scrapy.extensions.throttle.AutoThrottle']
2026-05-29 23:27:28 [CartercadillacSpider] INFO: Current Spider Environment set to: prod
2026-05-29 23:27:28 [CartercadillacSpider] INFO: Retrieving AWS secret: spider_prod_secrets
2026-05-29 23:27:28 [botocore.credentials] INFO: Found credentials in shared credentials file: ~/.aws/credentials
2026-05-29 23:27:28 [CartercadillacSpider] INFO: In base spider ml_kwargs is None and scrape_type is inventory
2026-05-29 23:27:28 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'loknow_spiders.middlewares.ThrottlingRetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2026-05-29 23:27:28 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'loknow_spiders.middlewares.RetrieverSpiderMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2026-05-29 23:27:29 [scrapy.middleware] INFO: Enabled item pipelines:
['loknow_spiders.pipelines.HTMLDetailsDecruftPipeline',
'loknow_spiders.pipelines.ExpungePhoneNumbersPipeline',
'loknow_spiders.pipelines.GPTVehiclePipeline',
'loknow_spiders.pipelines.PostProcessGPTVehiclePipeline',
'loknow_spiders.pipelines.CartercadillacPipeline',
'loknow_spiders.pipelines.RetrieverPipeline']
2026-05-29 23:27:29 [scrapy.core.engine] INFO: Spider opened
2026-05-29 23:27:29 [CartercadillacSpider] INFO: RetrieverAPI is making requests to https://api.adretriever.com with headers {'Authorization': 'Token 96dfb640234f1d676d6fa726f8eae6e7aab44cda'}
2026-05-29 23:27:29 [loknow_spiders] INFO: Scrape 38308f45 has been opened.
2026-05-29 23:27:29 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-05-29 23:27:29 [scrapy-playwright] INFO: Starting download handler
2026-05-29 23:27:29 [scrapy-playwright] INFO: Starting download handler
2026-05-29 23:27:37 [scrapy-playwright] INFO: Launching browser chromium
2026-05-29 23:27:37 [scrapy-playwright] INFO: Browser chromium launched
2026-05-29 23:28:11 [loknow_spiders] INFO: Clicking next page element...
2026-05-29 23:28:15 [loknow_spiders] INFO: Clicking next page element...
2026-05-29 23:28:29 [scrapy.extensions.logstats] INFO: Crawled 4 pages (at 4 pages/min), scraped 0 items (at 0 items/min)
2026-05-29 23:29:03 [loknow_spiders] INFO: Timeout querying next page element (could be the last page)
2026-05-29 23:29:03 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:29:05 [loknow_spiders] INFO: Timeout querying next page element (could be the last page)
2026-05-29 23:29:05 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:29:14 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:29:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-2155' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:29:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-2307' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:29:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-2309' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:29:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-2317' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:29:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-2612' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:29:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-2772' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:29:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-2775' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:29:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-2785' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:29:15 [loknow_spiders] INFO: Cache: missed loknow_spiders:gpt-response-cache:cb41af52f3bb1a55f2bb599b81da923e
2026-05-29 23:29:18 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:29:19 [httpx] INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2026-05-29 23:29:19 [loknow_spiders] INFO: Cache: hit loknow_spiders:gpt-response-cache:fc751227fda326a61528ad3e0f009ac6
2026-05-29 23:29:19 [CartercadillacSpider] INFO: Processed https://cartercadillac.ca/inventory/2026-cadillac-xt5-awd-luxury-V2HAPMB2Q5CDWG2nUNOi3Avdp
2026-05-29 23:29:19 [root] INFO: Making post request to https://api.adretriever.com/api/import/38308f45
2026-05-29 23:29:19 [loknow_spiders] INFO: Created: 1 items out of 1
Tail
task: <Task pending name='Task-110623' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:53:43 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-110618' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:53:43 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-110664' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:53:43 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-110584' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:53:43 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-110612' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:53:44 [loknow_spiders] INFO: Cache: missed loknow_spiders:gpt-response-cache:4192a8133b2bf2aba6e7239283bbc47d
2026-05-29 23:53:46 [httpx] INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2026-05-29 23:53:46 [CartercadillacSpider] INFO: Processed https://cartercadillac.ca/inventory/2026-cadillac-escalade-4wd-sport-h4PQfy8NQaeHwHJmFqezwQvdp
2026-05-29 23:53:47 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-112254' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:53:47 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-112292' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:53:47 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-112302' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:53:47 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-112316' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:53:47 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-112294' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:13 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:54:15 [loknow_spiders] INFO: Cache: hit loknow_spiders:gpt-response-cache:d7e95f3ad752b05234285a97fc096903
2026-05-29 23:54:15 [CartercadillacSpider] INFO: Processed https://cartercadillac.ca/inventory/2026-cadillac-lyriq-lyriq-v-premium-EmWdTqwwQlamdVSedrFhGwvdp
2026-05-29 23:54:29 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:54:29 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-114050' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:29 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-113961' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:29 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-113966' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:29 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-113944' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:29 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-113956' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:30 [scrapy.extensions.logstats] INFO: Crawled 69 pages (at 3 pages/min), scraped 61 items (at 3 items/min)
2026-05-29 23:54:30 [loknow_spiders] INFO: Cache: missed loknow_spiders:gpt-response-cache:b194f3c3afe49a8f2f590f9bbccdeda2
2026-05-29 23:54:33 [httpx] INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2026-05-29 23:54:33 [CartercadillacSpider] INFO: Processed https://cartercadillac.ca/inventory/2026-cadillac-ct4-v-blackwing-vRh94kohQfyrWRJsqfOPFgvdp
2026-05-29 23:54:33 [root] INFO: Making post request to https://api.adretriever.com/api/import/38308f45
2026-05-29 23:54:34 [loknow_spiders] INFO: Created: 4 items out of 4
2026-05-29 23:54:50 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-115607' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:50 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-115694' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:50 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-115639' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:50 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-115681' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:50 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-115683' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:54:51 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:54:53 [loknow_spiders] INFO: Cache: missed loknow_spiders:gpt-response-cache:8cb52baaf84d418028f1a7565f8157c3
2026-05-29 23:54:56 [httpx] INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2026-05-29 23:54:56 [CartercadillacSpider] INFO: Processed https://cartercadillac.ca/inventory/2026-cadillac-optiq-optiq-v-apu1vnIeSlaFgJyZfqEBSQvdp
2026-05-29 23:55:10 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-117403' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:10 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-117377' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:10 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-117387' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:10 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-117392' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:10 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-117382' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:13 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:55:14 [loknow_spiders] INFO: Cache: hit loknow_spiders:gpt-response-cache:ebb8f42b59bfee7dc80e6e7e5e92d6c0
2026-05-29 23:55:14 [CartercadillacSpider] INFO: Processed https://cartercadillac.ca/inventory/2026-cadillac-lyriq-lyriq-v-2f2rDWZxSHalduLw0iLdRAvdp
2026-05-29 23:55:18 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-119158' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:18 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-119155' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:18 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-119067' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:18 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-119060' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:18 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-119160' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:55:29 [scrapy.extensions.logstats] INFO: Crawled 71 pages (at 2 pages/min), scraped 64 items (at 3 items/min)
2026-05-29 23:55:45 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:55:46 [loknow_spiders] INFO: Cache: missed loknow_spiders:gpt-response-cache:a5d97e5d3f3521d214951ff068262262
2026-05-29 23:55:52 [httpx] INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2026-05-29 23:55:52 [CartercadillacSpider] INFO: Processed https://cartercadillac.ca/inventory/2026-cadillac-optiq-optiq-v-eQbwVnzYTLqcB6Iqx2MDTgvdp
2026-05-29 23:55:52 [root] INFO: Making post request to https://api.adretriever.com/api/import/38308f45
2026-05-29 23:55:53 [loknow_spiders] INFO: Created: 3 items out of 3
2026-05-29 23:56:15 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:56:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-120870' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-120837' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-120859' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-120875' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:15 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-120864' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:15 [loknow_spiders] INFO: Cache: hit loknow_spiders:gpt-response-cache:58ebcd48feb44ea5dc582f831d1d327b
2026-05-29 23:56:15 [CartercadillacSpider] INFO: Processed https://cartercadillac.ca/inventory/2026-cadillac-xt5-awd-sport-9wHdoqRBRrSMQGWkITw3uAvdp
2026-05-29 23:56:22 [CartercadillacSpider] WARNING: Got Locator.wait_for: Timeout 30000ms exceeded.
Call log:
- waiting for locator(".primary-carousel [data-slick-index=\"0\"] img").first to be visible
on page https://cartercadillac.ca/inventory/2026-cadillac-lyriq-premium-sport-gM8NMaoMQeK8pTO6sAxZkAvdp -- will try to fetch a version from cache.
2026-05-29 23:56:23 [CartercadillacSpider] INFO: Closed Playwright page
2026-05-29 23:56:23 [loknow_spiders] INFO: Cache: missed loknow_spiders:gpt-response-cache:None
2026-05-29 23:56:23 [scrapy.core.scraper] INFO: Dropped: Item had no text for a prompt, and we missed it in the cache by URL (https://cartercadillac.ca/inventory/2026-cadillac-lyriq-premium-sport-gM8NMaoMQeK8pTO6sAxZkAvdp).
{'item_data': {'url': 'https://cartercadillac.ca/inventory/2026-cadillac-lyriq-premium-sport-gM8N...
2026-05-29 23:56:23 [scrapy.core.engine] INFO: Closing spider (finished)
2026-05-29 23:56:23 [root] INFO: Making post request to https://api.adretriever.com/api/import/38308f45
2026-05-29 23:56:23 [loknow_spiders] INFO: Created: 1 items out of 1
2026-05-29 23:56:23 [loknow_spiders] INFO: GPT cache hit rate: 30.0%
2026-05-29 23:56:23 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (66 items) in: file:///home/scrapyd/items/loknow_spiders_prod/CartercadillacSpider/f31dbb705bb511f1bdc0c3bbae71ecef.jl
2026-05-29 23:56:23 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 40551,
'downloader/request_count': 74,
'downloader/request_method_count/GET': 74,
'downloader/response_bytes': 100450875,
'downloader/response_count': 74,
'downloader/response_status_count/200': 74,
'elapsed_time_seconds': 1734.170171,
'feedexport/success_count/FileFeedStorage': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2026, 5, 29, 23, 56, 23, 460648),
'httpcompression/response_bytes': 1434,
'httpcompression/response_count': 2,
'item_dropped_count': 4,
'item_dropped_reasons_count/DropItem': 4,
'item_scraped_count': 66,
'log_count/ERROR': 355,
'log_count/INFO': 353,
'log_count/WARNING': 5,
'memusage/max': 291627008,
'memusage/startup': 125333504,
'playwright/context_count': 1,
'playwright/context_count/max_concurrent': 1,
'playwright/context_count/persistent/False': 1,
'playwright/context_count/remote/False': 1,
'playwright/page_count': 72,
'playwright/page_count/max_concurrent': 5,
'playwright/request_count': 21435,
'playwright/request_count/method/GET': 19734,
'playwright/request_count/method/POST': 1701,
'playwright/request_count/navigation': 1298,
'playwright/request_count/resource_type/document': 1298,
'playwright/request_count/resource_type/eventsource': 138,
'playwright/request_count/resource_type/fetch': 1587,
'playwright/request_count/resource_type/font': 1492,
'playwright/request_count/resource_type/image': 2802,
'playwright/request_count/resource_type/other': 511,
'playwright/request_count/resource_type/ping': 465,
'playwright/request_count/resource_type/script': 8086,
'playwright/request_count/resource_type/stylesheet': 3518,
'playwright/request_count/resource_type/xhr': 1538,
'playwright/response_count': 20983,
'playwright/response_count/method/GET': 19367,
'playwright/response_count/method/POST': 1616,
'playwright/response_count/resource_type/document': 1296,
'playwright/response_count/resource_type/eventsource': 138,
'playwright/response_count/resource_type/fetch': 1511,
'playwright/response_count/resource_type/font': 1348,
'playwright/response_count/resource_type/image': 2802,
'playwright/response_count/resource_type/other': 502,
'playwright/response_count/resource_type/ping': 463,
'playwright/response_count/resource_type/script': 8014,
'playwright/response_count/resource_type/stylesheet': 3518,
'playwright/response_count/resource_type/xhr': 1391,
'request_depth_max': 1,
'response_received_count': 74,
'robotstxt/request_count': 2,
'robotstxt/response_count': 2,
'robotstxt/response_status_count/200': 2,
'scheduler/dequeued': 72,
'scheduler/dequeued/memory': 72,
'scheduler/enqueued': 72,
'scheduler/enqueued/memory': 72,
'start_time': datetime.datetime(2026, 5, 29, 23, 27, 29, 290477)}
2026-05-29 23:56:23 [scrapy.core.engine] INFO: Spider closed (finished)
2026-05-29 23:56:23 [scrapy-playwright] INFO: Closing download handler
2026-05-29 23:56:23 [scrapy-playwright] INFO: Closing download handler
2026-05-29 23:56:23 [scrapy-playwright] INFO: Closing browser
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-122496' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-122503' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-122505' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-122513' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-122632' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-124290' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-124240' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-124217' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-124242' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>
2026-05-29 23:56:23 [asyncio] ERROR: Task was destroyed but it is pending!
task: <Task pending name='Task-124251' coro=<ScrapyPlaywrightDownloadHandler._get_response_and_download.<locals>._handle_response() running at /home/scrapyd/venv/lib/python3.10/site-packages/scrapy_playwright/handler.py:524> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[AsyncIOEventEmitter._emit_run.<locals>.callback() at /home/scrapyd/venv/lib/python3.10/site-packages/pyee/asyncio.py:97]>