This commit is contained in:
2025-08-05 09:19:34 +08:00
commit 584548d006
1696 changed files with 53855 additions and 0 deletions
+11
View File
@@ -0,0 +1,11 @@
TODO
====
* Add SCRAPY_JOB global support (jobs sharing same SCRAPY_JOB share same queues).
* Use a spider middleware instead of spider mixin. This will avoid the spider
idle signal hack.
* Allow to use pubsub whenever appropriate.
* Move example project to its own repository. Include different crawling use
cases (i.e.: producer/consumer).
* Add pyrebloom dupefilter.
* Warn and pass unserializable requests.