/tech/ - Technology★

As a programmer I'd never personally use any data-scraping-as-a-service bullshit, but companies seem to like them and pay questionably large amounts of money to use some of them.

If you just want to write a simple scraper/bot use python. It is still king of the web scraping world. Maybe include phantom.js or similar if you absolutely need it.

the only scraping things that make enough money to be relevant are nigger shoe bots to get the limited release sneakers and sell them at a huge markup.

awk, sed, grep and curl.

>>828329 (OP)

My dad works at a storage facility and i made a scraper that gets the local competitions prices and emails and excel spreadsheet of it.

I would love to learn of other scraping projects that are available so i could use some of my newly learned skills. Where do you guys stand on the legality of scraping and having a scraper pinned on github for possible employers to see?

>>828404

>Where do you guys stand on the legality of scraping

zero fucks given. just don't be a dick and flood people's servers.

>>828404

I worked on some project before that included scraping prices off of football websites and presenting a list of the 3 most cheapest tickets to football games in UK

>>828443

the only problem is that if they change their site even slightly, it can screw everything up

how would /tech/ make a scraper to categorically scrape for pictures?

>>828453

be more specific

>>828451

Sorry.

Forgot to give detail that those websites were partners of the client, so offered xml/json format of the important details.

But yeah, if scraping off of websites, then what you said is gonna happen.

>>828454

Scrape all new rare pepes from the internet every hour.

apify.com

/thread

>>828530

Fuck off newfag