/tech/ - Technology★

direct twitter api https://developer.twitter.com/en/docs.html

or pick a language and use a real scraping library

https://scrapy.org/

https://jsoup.org/

http://www.nokogiri.org/

writing a program to go off and open a browser to do it is going to be the slowest way possible to do this.

>>999782 (OP)

Depends on which kind of data you are interested in.

>>999848

API is ideal. Scraping tools are quicker than proxy-browser solutions like Selenium, but they can't execute JavaScript so might not always work.

>Anyone ever scrape time-series data from twitter? I'm trying to work out a solution with Selenium, but it's going to require a lot of runtime.

Kill me, Pete

Just look at their unofficial API through your browser's network activity monitor. It's pretty easy to understand, and you don't have to put up with the bullshit rate limiting that comes with the official api.

>>999782 (OP)

Just use the devtools protocol if you don't want to emulate a browser. Running chromium in headless mode is the most undetectable way to scrape data, and most likely to be successful as these websites expect an actual browser to execute javascript.

>>999848

>>999926

Need to execute the js, unfortunately. Trying to avoid going the API route, because I need a lot more historical data than is available at my price point.

>>1000022

Gonna look into this, thanks.

>>1000120

Makes sense, thanks.

kill me, pete

KILL ME NOW DO IT I BEG YOU