/tech/ - Technology★

Selenium (python), phantom/casper/nightmarejs (javashit)

>>1010943

how selenium compare to these:

Requests http://docs.python-requests.org/

lxml http://lxml.de/

>Selenium (python)

>phantom/casper/nightmarejs (javashit)

so the choice to do web automation is python and javascript? other languages don't have framework and libraries like these?

can python be compiled into binary?

>>1010943

I read about Selenium here https://www.guru99.com/selenium-python.html

If I understand it correctly, it needs a browser to be run? and it just controls the browser?

I would prefer something that will work as standalone, without web browser being run. Or can selenium just run some web browser with hidden window?

>>1010943

Adding to this, Chrome (and Chromium by extent) supports running a headless version of itself with a flag. So you can use that for automation with Selenium. As an example of integration: https:// intoli.com/blog/running-selenium-with-headless-chrome/

>>1010970

>how selenium compare to these:

Take, for example, a website full of Javashit that loads half of its contents dynamically. Requests & lxml parsing will only get you the initial page, it won't execute any JS code. This is where these methods come in, so you can dig deeper.

>other languages don't have framework and libraries like these?

This kind of thing is mostly used for test automation and QA, so the languages with the most support would have to be poo-friendly - JS, Python & Java.

>can python be compiled into binary?

You can compile a JIT for your script with the -c option, but that's about as far as it goes.

>>1010971

They all run "a browser" in a way. You need something to execute JS at least.

>Or can selenium just run some web browser with hidden window?

Headless browser, meaning no window is drawn and only under the hood actions are performed. See above.

>>1011093

>Adding to this, Chrome (and Chromium by extent) supports running a headless version of itself with a flag. So you can use that for automation with Selenium. As an example of integration: https:// intoli.com/blog/running-selenium-with-headless-chrome/

chrome and derivatives are botnet

>This kind of thing is mostly used for test automation and QA, so the languages with the most support would have to be poo-friendly - JS, Python & Java.

is Python least shit of this choice?

>You can compile a JIT for your script with the -c option, but that's about as far as it goes.

that's dangerous. what if I run same code on different python interpreter and it will fuck me up

>They all run "a browser" in a way. You need something to execute JS at least.

Actually, for my needs, executing JS might not be necessary. Or I could see in web browser what AJAX requests are done and simulate them.

>Headless browser, meaning no window is drawn and only under the hood actions are performed. See above.

But that seems dangerous. browsers are botnet and bloat, maybe better to use something without headless browser

>>1011352

>chrome and derivatives are botnet

You are already executing JS code though. That's about as botnet as it gets. And anything else will still have a web engine operating somewhere in it.

>is Python least shit of this choice?

You'll take less time writing your shit in either JS or Python. Performance barely matters since most of your execution time will likely be spent on fetching data.

>what if I run same code on different python interpreter and it will fuck me up

It's optional, you just use it to lessen the startup time of your scripts.

>Or I could see in web browser what AJAX requests are done and simulate them.

Hardly automation at this point, since you're putting more work on yourself. But by all means, if you find a good way to cross-section a website for all possible (or needed) AJAX requests, use that with any language & any library capable of making simple http/s requests.