• Using local stats: LogParser v0.8.4, last updated at 2026-05-29 22:50:32, /home/scrapyd/logs/loknow_spiders_prod/MainstSpider/69cb400e5bb011f1bdc0c3bbae71ecef.json

PROJECT (loknow_spiders_prod), SPIDER (MainstSpider)

  • Log analysis
  • Log categorization
  • Progress visualization
  • View log
  • Crawler.stats
  • projectloknow_spiders_prod
    spiderMainstSpider
    job69cb400e5bb011f1bdc0c3bbae71ecef
    first_log_time2026-05-29 22:47:50
    latest_log_time2026-05-29 22:50:22
    runtime0:02:32
    crawled_pages 488
    scraped_items 484
    shutdown_reasonN/A
    finish_reasonfinished
    log_critical_count0
    log_error_count0
    log_warning_count1
    log_redirect_count758
    log_retry_count0
    log_ignore_count1
    latest_crawl
    latest_scrape
    latest_log
    current_time
    latest_itemN/A

    • WARNING+

    • warning_logs
      1 in total 

      2026-05-29 22:47:50 [py.warnings] WARNING: /home/scrapyd/venv/lib/python3.10/site-packages/scrapy/utils/request.py:248: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.

It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy.

See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
  return cls(crawler)

      INFO

    • redirect_logs
      last 0 of 758

    • ignore_logs
      1 in total 

      2026-05-29 22:49:28 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.mainst.biz/bridgeland-hillside-apartments>: HTTP status code is not handled or not allowed

      DEBUG

    • scrapy_version

      2.10.0

    • latest_stat

      2026-05-29 22:49:52 [scrapy.extensions.logstats] INFO: Crawled 474 pages (at 197 pages/min), scraped 470 items (at 208 items/min)

    • Head

      2026-05-29 22:47:50 [scrapy.utils.log] INFO: Scrapy 2.10.0 started (bot: retriever)
2026-05-29 22:47:50 [scrapy.utils.log] INFO: Versions: lxml 5.1.0.0, libxml2 2.12.3, cssselect 1.2.0, parsel 1.9.0, w3lib 2.0.0, Twisted 22.4.0, Python 3.10.12 (main, Mar  3 2026, 11:56:32) [GCC 11.4.0], pyOpenSSL 24.1.0 (OpenSSL 3.2.1 30 Jan 2024), cryptography 42.0.5, Platform Linux-6.8.0-1030-aws-x86_64-with-glibc2.35
2026-05-29 22:47:50 [scrapy.addons] INFO: Enabled addons:
[]
2026-05-29 22:47:50 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_ENABLED': True,
 'BOT_NAME': 'retriever',
 'HTTPCACHE_ENABLED': True,
 'HTTPCACHE_EXPIRATION_SECS': 86400,
 'IMAGES_STORE_S3_ACL': 'public-read',
 'LOG_FILE': '/home/scrapyd/logs/loknow_spiders_prod/MainstSpider/69cb400e5bb011f1bdc0c3bbae71ecef.log',
 'LOG_FORMATTER': 'loknow_spiders.logging.PoliteLogFormatter',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'loknow_spiders.spiders',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['loknow_spiders.spiders'],
 'TELNETCONSOLE_ENABLED': False,
 'USER_AGENT': 'AdRetriever (https://adretriever.com)'}
2026-05-29 22:47:50 [py.warnings] WARNING: /home/scrapyd/venv/lib/python3.10/site-packages/scrapy/utils/request.py:248: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.

It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy.

See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
  return cls(crawler)

2026-05-29 22:47:50 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.throttle.AutoThrottle']
2026-05-29 22:47:50 [MainstSpider] INFO: Current Spider Environment set to: prod
2026-05-29 22:47:50 [MainstSpider] INFO: Retrieving AWS secret: spider_prod_secrets
2026-05-29 22:47:50 [botocore.credentials] INFO: Found credentials in shared credentials file: ~/.aws/credentials
2026-05-29 22:47:50 [MainstSpider] INFO: In base spider ml_kwargs is None and scrape_type is inventory
2026-05-29 22:47:50 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'loknow_spiders.middlewares.ThrottlingRetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats',
 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware']
2026-05-29 22:47:50 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'loknow_spiders.middlewares.RetrieverSpiderMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2026-05-29 22:47:52 [scrapy.middleware] INFO: Enabled item pipelines:
['loknow_spiders.pipelines.MainstPipeline',
 'loknow_spiders.pipelines.RetrieverPipeline']
2026-05-29 22:47:52 [scrapy.core.engine] INFO: Spider opened
2026-05-29 22:47:52 [MainstSpider] INFO: RetrieverAPI is making requests to https://api.adretriever.com with headers {'Authorization': 'Token 96dfb640234f1d676d6fa726f8eae6e7aab44cda'}
2026-05-29 22:47:52 [loknow_spiders] INFO: Scrape 94e27e89 has been opened.

2026-05-29 22:47:52 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/allyn-arms-apartments
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/aquarius-place
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/amantea-tower
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/amera-manor
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/alexander-apartments
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/donna-place
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/dickens-apartments
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/diane-apartments-2
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/diane-apartments-1
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/dhillon-tower
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/deluca-manor
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/delburn-house
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/dartmouth-apartments
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/dahlstrom-manor
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cypress-manor
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/crescent-heights-view
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/crescent-heights-manor
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/crescent-heights-apartments
2026-05-29 22:48:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/courtenay-riverway-apartments
2026-05-29 22:48:10 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/coronation-place
2026-05-29 22:48:10 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cornerstone-apartments

2026-05-29 22:48:10 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/confederation-place
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/grapes-manor
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/granville-place
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/grand-apartments
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/grace-manor
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/glenbow-manor
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/gilford-apartments
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/gemini-apartments
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/gateway-place
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/gateway-manor-apartments
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/gary-manor
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/froom-apartments
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/forestview-heights-apartments
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/floyd-manor
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/flores-manor
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/falconcrest-village
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/fairhaven-apartments
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/evergreen-manor-apartments
2026-05-29 22:48:15 [MainstSpider] INFO: Processed https://www.mainst.biz/apartments/edmonton/evansdale-court-apartments

    • Tail

      2026-05-29 22:49:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-on-fort-saskatchewan
2026-05-29 22:49:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-on-elbow
2026-05-29 22:49:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-on-33rd
2026-05-29 22:49:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-on-124th
2026-05-29 22:49:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-mission-hills
2026-05-29 22:49:09 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-mews
2026-05-29 22:49:10 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-marda-loop-apartments
2026-05-29 22:49:10 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-maple-ridge-apartments
2026-05-29 22:49:10 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-hilltop

2026-05-29 22:49:10 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mainstreet-hart-apartments
2026-05-29 22:49:13 [MainstSpider] INFO: Processed https://www.mainst.biz/apartments/surrey/mainstreet-guildford-townhomes
2026-05-29 22:49:13 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/mack-apartments
2026-05-29 22:49:13 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/lynden-court
2026-05-29 22:49:13 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/lorraine-manor
2026-05-29 22:49:13 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/abbey-tower
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/loftus-manor
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/linde-apartments
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/lincoln-terrace
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/libra-apartments
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/leo-manor-apartments
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/legislative-estates-apartments
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/lawson-place
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/laurentian-apartments
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/lauderdale-manor
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/lancelot-apartments
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kleisinger-45
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kleisinger-3
2026-05-29 22:49:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kleisinger-27
2026-05-29 22:49:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kirkness-park-apartments
2026-05-29 22:49:15 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kingsway-lofts-and-apartments
2026-05-29 22:49:17 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kings-court
2026-05-29 22:49:17 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/king-edward
2026-05-29 22:49:17 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/key-west-apartments
2026-05-29 22:49:17 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kewanee-manor
2026-05-29 22:49:17 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kerpel-apartments
2026-05-29 22:49:17 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kensington-apartments
2026-05-29 22:49:17 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/karen-hall
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kane-apartments
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/kamloops-tranquille-apartments
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/julliard-south
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/julliard-north
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/julia-place
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/joy-manor-apartments
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/john-manor
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/jerry-place
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/jazz-manor
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/jays-place
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/jasper-ave-lookout-2
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/inglewood-124-apartments
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/ingle-apartments
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/imperial-parkside
2026-05-29 22:49:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/huntsville-apartments
2026-05-29 22:49:19 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/greystone-court

2026-05-29 22:49:19 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/grey-apartments
2026-05-29 22:49:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/youngstown-townhomes-and-apartments
2026-05-29 22:49:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/grey-909
2026-05-29 22:49:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/greenwood-gardens
2026-05-29 22:49:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/elbow-mission-apartments
2026-05-29 22:49:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/elbow-court
2026-05-29 22:49:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/eileen-manor
2026-05-29 22:49:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/edmonton-trail-apartments
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/edgar-estates
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/eastwood-manor
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/duchess-manor
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/downtown-edmonton-apartments
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/doverglen-estates
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/dorset-arms
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/confederation-park-apartments
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/compton-court
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/college-manor-mainstreet-apartments
2026-05-29 22:49:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/college-grove
2026-05-29 22:49:22 [root] INFO: Making post request to https://api.adretriever.com/api/import/94e27e89
2026-05-29 22:49:23 [loknow_spiders] INFO: Created: 98 items out of 100
2026-05-29 22:49:23 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/college-1127
2026-05-29 22:49:23 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cochrane-house-apartments
2026-05-29 22:49:23 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cochrane-104
2026-05-29 22:49:23 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/clareview-court
2026-05-29 22:49:23 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/chinook-woods
2026-05-29 22:49:24 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/chinook-place
2026-05-29 22:49:24 [MainstSpider] INFO: Processed https://www.mainst.biz/apartments/calgary/chinook-meadows-apartments
2026-05-29 22:49:25 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/chilliwack-landing-apartments
2026-05-29 22:49:25 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/chilliwack-district-mainstreet-apartments
2026-05-29 22:49:25 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/chilliwack-core-mainstreet-apartments
2026-05-29 22:49:25 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/chilliwack-central-apartments
2026-05-29 22:49:25 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/chelsey-apartments
2026-05-29 22:49:25 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/chelsea-park-gardens
2026-05-29 22:49:25 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cheam-view-apartments-2
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cheam-view-apartments-1
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/charlotte-manor
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/charles-rutherford-apartments
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cesar-place
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/centre-crescent-apartments-2
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/centre-crescent-apartments-1
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/centre-110
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/centre-102
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/central-lrt-apartments
2026-05-29 22:49:26 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/central-beltline-apartments
2026-05-29 22:49:27 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cedarwood-arms
2026-05-29 22:49:27 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cedartree-village
2026-05-29 22:49:27 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cedarbrae-manor
2026-05-29 22:49:27 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cedar-grove
2026-05-29 22:49:27 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cathy-manor
2026-05-29 22:49:27 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cascade-manor
2026-05-29 22:49:27 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/carlson-apartments
2026-05-29 22:49:28 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <404 https://www.mainst.biz/bridgeland-hillside-apartments>: HTTP status code is not handled or not allowed
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/carina-court
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cardona-place
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/capricorn-apartments
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/capital-manor
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/camino-manor
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cameron-manor-1
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/camelia-court
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/cambridge-manor
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/byron-apartments
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/broadview-legislature-apartments
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bridgeport-manor
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bridgeland-apartments
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/apartments/calgary/bowness-village-apartments
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bowness-4347-apartments
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bowness-4315-apartments
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bowness-4311
2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bow-river-townhomes

2026-05-29 22:49:29 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/booth-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/2501-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bankview-place
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bankview-hill-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bankview-1737-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bankview-1711
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/avonhurst-3525
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/avila-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/avenue-tower
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/avenue-houses
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/aurora-place-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/athlone-park-townhomes
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/astoria-manor
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/assiniboine
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/aspen-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/ashley-court
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/ariel-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/argyll-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/arcade-heritage-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/anna-court-apartments
2026-05-29 22:49:48 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/amherst-apartments

2026-05-29 22:49:50 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/airdrie-meadows
2026-05-29 22:49:52 [scrapy.extensions.logstats] INFO: Crawled 474 pages (at 197 pages/min), scraped 470 items (at 208 items/min)
2026-05-29 22:49:59 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/beltline-garden-apartments

2026-05-29 22:50:00 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/beltline-apartments-off-17th

2026-05-29 22:50:10 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/beaumaris-lake-apartments
2026-05-29 22:50:14 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/alex-44
2026-05-29 22:50:17 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/alex-38
2026-05-29 22:50:18 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/alex-20
2026-05-29 22:50:19 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/yarmouth-apartments
2026-05-29 22:50:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bonaventure-apartments
2026-05-29 22:50:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/berkley-manor
2026-05-29 22:50:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/beltline-on-14th
2026-05-29 22:50:21 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/beltline-apartments
2026-05-29 22:50:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/belmae-manor
2026-05-29 22:50:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/beauview-apartments
2026-05-29 22:50:22 [MainstSpider] INFO: Processed https://www.mainst.biz/find-an-apartment/bannerman-apartments
2026-05-29 22:50:22 [scrapy.core.engine] INFO: Closing spider (finished)

2026-05-29 22:50:22 [root] INFO: Making post request to https://api.adretriever.com/api/import/94e27e89
2026-05-29 22:50:22 [loknow_spiders] INFO: Created: 76 items out of 84
2026-05-29 22:50:22 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (484 items) in: file:///home/scrapyd/items/loknow_spiders_prod/MainstSpider/69cb400e5bb011f1bdc0c3bbae71ecef.jl
2026-05-29 22:50:22 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 334188,
 'downloader/request_count': 1246,
 'downloader/request_method_count/GET': 1246,
 'downloader/response_bytes': 24502588,
 'downloader/response_count': 1246,
 'downloader/response_status_count/200': 487,
 'downloader/response_status_count/301': 758,
 'downloader/response_status_count/404': 1,
 'elapsed_time_seconds': 150.141889,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2026, 5, 29, 22, 50, 22, 844787),
 'httpcache/firsthand': 51,
 'httpcache/hit': 1195,
 'httpcache/miss': 51,
 'httpcache/store': 51,
 'httpcompression/response_bytes': 103937490,
 'httpcompression/response_count': 488,
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/404': 1,
 'item_scraped_count': 484,
 'log_count/INFO': 512,
 'log_count/WARNING': 1,
 'memusage/max': 179142656,
 'memusage/startup': 125198336,
 'request_depth_max': 1,
 'response_received_count': 488,
 'robotstxt/request_count': 2,
 'robotstxt/response_count': 2,
 'robotstxt/response_status_count/200': 2,
 'scheduler/dequeued': 1244,
 'scheduler/dequeued/memory': 1244,
 'scheduler/enqueued': 1244,
 'scheduler/enqueued/memory': 1244,
 'start_time': datetime.datetime(2026, 5, 29, 22, 47, 52, 702898)}
2026-05-29 22:50:22 [scrapy.core.engine] INFO: Spider closed (finished)

    • Log

      /1/log/utf8/loknow_spiders_prod/MainstSpider/69cb400e5bb011f1bdc0c3bbae71ecef/?job_finished=True

    • Source

      https://scrapyd.adretriever.com/logs/loknow_spiders_prod/MainstSpider/69cb400e5bb011f1bdc0c3bbae71ecef.log

  • sourcelog
    last_update_time2026-05-29 22:50:22
    last_update_timestamp1780095022
    downloader/request_bytes334188
    downloader/request_count1246
    downloader/request_method_count/GET1246
    downloader/response_bytes24502588
    downloader/response_count1246
    downloader/response_status_count/200487
    downloader/response_status_count/301758
    downloader/response_status_count/4041
    elapsed_time_seconds150.141889
    feedexport/success_count/FileFeedStorage1
    finish_reasonfinished
    finish_timedatetime.datetime(2026, 5, 29, 22, 50, 22, 844787)
    httpcache/firsthand51
    httpcache/hit1195
    httpcache/miss51
    httpcache/store51
    httpcompression/response_bytes103937490
    httpcompression/response_count488
    httperror/response_ignored_count1
    httperror/response_ignored_status_count/4041
    item_scraped_count484
    log_count/INFO512
    log_count/WARNING1
    memusage/max179142656
    memusage/startup125198336
    request_depth_max1
    response_received_count488
    robotstxt/request_count2
    robotstxt/response_count2
    robotstxt/response_status_count/2002
    scheduler/dequeued1244
    scheduler/dequeued/memory1244
    scheduler/enqueued1244
    scheduler/enqueued/memory1244
    start_timedatetime.datetime(2026, 5, 29, 22, 47, 52, 702898)