that all Routes is triggering program backend for scraping some website, this web is working fine. but i wonder how make it more flexible.
i just thought of making auto-run that all routes with one button, and have their own status.
that column will have Status and their status like (running, done, stop, not running etc.) but i cannot imagine that logic.
i already create for auto-run and work fine, and my question just how to know their status is running, done, stop or not running in the background.
any idea really appreciate. this is my own project so i’m so excite to make this work
The simplest way of doing this is by running a log of each stage of the scraping process. So for example:
from selenium import webdriver from selenium.webdriver.common.by import By driver = webdriver.Chrome() driver.get('https://www.google.com/') print("Loaded Google.com") some_task = driver.find_element(By.XPATH, '//button[text()="Some text"]') print("Got some task")
(Locating elements as per: https://selenium-python.readthedocs.io/locating-elements.html)
However, for real-time processing of task status and for more efficiency, you can use Celery.
Celery works well for web scraping tasks as it actually allows you to asynchronously offload work from your Python app to workers and task queues.
You can then retrieve proper status reports from each worker. See: https://docs.celeryq.dev/en/stable/reference/celery.states.html
Answered By – Olney1
Answer Checked By – Dawn Plyler (Easybugfix Volunteer)