Use scrapy framework to scrape the first 500 items (title, image url) from sreality.cz (flats, sell) and save it in the Postgresql database. Implement a simple HTTP server in python and show these 500 items on a simple page (title and image) and put everything to single docker-compose command so that I can just run "docker-compose up" in the Github repository and see the scraped ads on http://127.0.0.1:8080 page.
-
Download and setup selenium chrome browser
-
Start the DB server docker run --name some-postgres -e POSTGRES_PASSWORD=my_super_strong_password -d postgres
-
Run crawler to populate DB cd sreality_scraper scrapy crawl sreality
-
Run the server cd .. python3 web_server.py
Simply run docker-compose up --build
Visit 127.0.0.1:8080
(If nothing shows up, wait for the crawler to end)
docker run --rm -it --entrypoint /bin/bash --volumes-from your-container-name your-image-name
docker build -t your-image-name .
docker rm -f your-container-name # stop and remove old docker container
docker run -p 8080:8080 --name your-container-name your-image-name
docker images # find ID
docker rmi IMG_ID_OR_NAME # force with -f
docker image prune #(remove dangling (~ghosts) imgs)
docker run --name some-postgres -e POSTGRES_PASSWORD=my_super_strong_password -d postgres
psql -U postgres -h 127.0.0.1 password=my_super_strong_password
psql -U postgres -h 127.0.0.1 -c "ALTER USER postgres WITH PASSWORD 'my_super_strong_password';"
su -c "psql -h 127.0.0.1 -c \"ALTER ROLE postgres WITH LOGIN PASSWORD 'my_super_strong_password' \"" postgres
TODO: Add tests, change a bit docker-compose so it uses variables/environment such as DB connections, selenium directory, number of pages to scrape, running one shell file instead of two CMDs, etc., add a button to regenerate listings, generate a listing in browser (framework/js), not the server.