/tech/ - Technology★

I'm seeing a lot of bot posts spammed around here. Be careful anons

Judensheim pls go

>>1057341

That could be my next project

>>1057235 (OP)

>Are you really a programmer? Show us your own 8ch downloader then.

>Here's mine.

>C:\

I'd be more impressed if you found a toilet to shit in pajeet.

You can read about my downloader if you have a decent OS installed you don't with man wget

>>1057485

UI programming is basic sh*t, but I do agree that OP is a faggot for not doing cross-platform software. (and don't give me this "Linux or GTFO" shit)

Mods REALLY need to start banning racism on here.

>>1057486

>(and don't give me this "Linux or GTFO" shit

command line or GTFO ;^)

>>1057491

>Mods REALLY need to start banning racism on here.

where does this nigger think his bot is posting?

>>1057485

Why would you want to use a terrible autist os when you could use something that actually works? :^)

gallery-dl

>>1057486

>not doing cross-platform

>java

maybe you should ask in what language was made before posting.I suggest you read more about look and feel.

>>1057785

You work at Mozilla on Firefox? Wow

curl https://8ch.net/tech/res/1057235.html | grep -o '<a href=['"'"'"][^"'"'"']*['"'"'"]' | sed -e 's/^<a href=["'"'"']//' -e 's/["'"'"']$//' | grep file_store | xargs -I{} wget {}

>>1057790

You work at Microsoft on Windows? Wow

>>1057794

>projecting

How embarrassing. I shared no evidence to suggest I use Windows, rookie.

>>1057235 (OP)

like theres anything worth downloading here. just click the save button in the browser if that ever happens or use wget but theres never so much good stuff that its worth using that.

>>1057486

whats there even to program on windows.. they have simple gui editors for that and even someone that has never programmed before can use them.

wget -P thread_media -nd -nc -r -l 1 -H -D i.4cdn.org -A png,gif,jpg,jpeg,webm --reject-regex='s.jpg' [Thread URL]

>>1057776

> Not knowing what UI libraries are

The look says nothing as it is Win10. Could be Java or C# or whatever the fuck OP wants

>>1057533

>>>/hydrus/ is even better

>>1057791

>>1057831

Not everyone is sicko mode

>>1057817

Then it will not be cross-platform.

>>1057829

Source code or GTFO

>>1057791

> curl -> grep -> sed -> grep ->wget

#read the URLs to get from the URL listed and treat it as HTML input to be parsed:
wget -F -i https://8ch.net/tech/res/1057235.html

add accept/reject lists and other options as needed

>>1057831

thanks anon I was hoping someone would post that line too.

k

https://notabug.org/darmor/meme_magic_stats/src/master/src/main.py

>>1057838

>https://notabug.org/darmor/meme_magic_stats/src/master/src/main.py

>for i, thread in enumerate(downloadBoard("https://8ch.net"+currentBoard+"catalog.html",currentBoard)):

> # get json

> json = downloadThread(thread)

Pages are served JSON format so use

https://8ch.net/tech/res/1057235.json

instead of

https://8ch.net/tech/res/1057235.html

>>1057839

https://notabug.org/darmor/meme_magic_stats/src/master/src/downloadBoard.py

>board = board.replace("html","json")

>>1057829

>22 workspaces

This fucking chad.

>>1057235 (OP)

wget -bEHkprl 'inf' -So 'wget.log' \
     --accept-regex="^https?://(media\.|softserve\.)?8ch\.net/((${B}/(res/${T}\.html|threads\.json)|main\.js)|(${B}/(thumb|src)|file_store(/thumb)?|js|static|stylesheets)/.*)$" \
     --warc-cdx --warc-file="8ch.${B}.${T}" -nH -P "8ch.${B}.${T}" -- "https://8ch.net/${B}/res/${T}.html"

>>1057829

>22

>default status bar

madman

>>1057829

>1280x1024

How comfortable is that resolution nowadays? Is there a reason to keep to it rather than going with e.g. 1920x1200?

>>1057868

I mostly use it out of habit to be honest, but it's enough for two windows with 80 chars each. Somewhat annoying for movies obviously. 4:3 master race

>>1057874

>4:3

But it's 5:4

>>1057881

Yeah, but I'd prefer a 4:3 monitor.

>>1057874

>it's enough for two windows with 80 chars each

Close but no cigar - it would work only without any vertical borders/scrollbars etc. whatsoever (unless the font has char width less than 8 pixels).

>>1057881

>>1057882

Why did VESA approve that weird 5:4 reso of 1280x1024 (rather than 1280x960 or 1366x1024) anyway?

>>1057948

>ide

hmmm

wget -O - https://8ch.net/tech/res/1057235.html |\

sed -E 's@(https://media.8ch.net/file_store/[0-9a-f]+.(jpe?g|png|gif|webm|mp4))@\n\1\n@g' |\

grep -E 'https://media.8ch.net/file_store/[0-9a-f]+.(jpe?g|png|gif|webm|mp4)' |\

uniq | wget -i -

>>1057955

any sed thing looks disgusting every time

>>1057954

>>ide

>hmmm

????????????????

>>1057948

post your code faggot

>>1057955

    public static void DownloadAllImages(string url, string xpath)
    {
        using (var client = new WebClient())
        {
            var html = client.DownloadString(url);
            var doc = new HtmlAgilityPack.HtmlDocument();

            doc.LoadHtml(html);

            var links = doc.DocumentNode.SelectNodes(xpath)
                .Select(x => x.GetAttributeValue("href", null));

            const string imageDir = "dlimages";

            IoHelpers.CreateDirectory(imageDir);

            foreach (var link in links)
            {
                var filename = link.Split(new[] { '/' }).Last();
                Console.WriteLine(filename);
                client.DownloadFile(link, Path.Combine(imageDir, filename));
            }
        }
    }

>>1057491

>>1057486

Linux or GTFO

>>1057491

Kike, pls. Next you'll demand blacklisting CIS males next I suppose.

enum class Type { Kike, Dyke, Gayaf, Nigger, Pajeet, Mudshit, XYZ, CIS_male };

bool blacklist_check(const Type applicant) {
    return (applicant == Type::CIS_male);
}

int main() {
  // Perform initial blacklist check
  auto blacklisted = blacklist_check(Type::CIS_male);  // blacklisted
  blacklisted = blacklist_check(Type::Kike);           // OK so far
  blacklisted = blacklist_check(Type::Pajeet);         // OK so far
}

>>1057955

>I've not read the wget manual

>I don't know about the -i -F -A -R -nc options

RTFM for wget

wget even has regex options, plus a config file option if the command line gets too long

See the following for ideas:

>>1057836

>>1057831

>>1057848

>>1057831

>>1058040

stop that retarded meme that is putting the - symbol where it should not be

I have an oneliner I frequently use.

>>1058055

...nice design doc?

Here is a working curl post script

curl -A "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (\

KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5" --referer "https://8ch.net" -d "thr\

ead=1031726&board=tech&email=''&body=wtfareyouevensayingtomedude&password&user_flag=&json_respons\

e=1&post='New Reply'" -X POST https://sys.8ch.net/post.php

>>1057794

>"Hello? This is Microsoft. I'm calling about the virus you have on your machine anon."

>>1058063

at least fake a recent version. no one uses those old phones anymore.

>>1057835

>>1057835

>the looks says nothing

>hurr durr i am implying that a program it's not cross-platform because it was executed on a windows machine

ok this is actually interesting.

so i could make it works (i just used braindead wget). how do i make front end of this?. i don't really want to use GUI, but i want to make it with TUI, all what i knew for other scripting language is just python,though.

>>1058088

https://notabug.org/darmor/meme_magic_stats/src/time-handling/src

>>1058063

It's not a design doc, its a usage statement. No I won't post the source, and no your curl script doesn't snipe gets lol

>>1058153

>No I won't post the source

common sign of a code baby

someone already posted a 8chan scraper here >>1057838

>>1058040

>>1057948

https://www.youtube.com/watch?v=VFbxMoYJ14E

female chest pumping aaron lawson?

>>1058155

I've shared plenty of code snippets on this board, including a nice userscript for the catalog page. I can't share the gets sniper for obvious reasons, think about it.

Also,

>scraper

>reinventing the wheel this badly

Simple browser extension does this trick, fren

>>1058055

>>1058153

>No I won't post the source,

Not even if someone makes a FOI request for the source code, since it was made with public funds glownigger?

>>1058158

let me guess your "sniper" algorithm

>get most recently bumped thread

>get last post on bumped thread

>if post number = x post reply

woooow thats crazy fam, how could you ever make something like that!!?!?!?!

let me guess your distinguishing feature is that you do concurrent requests?

>>1058162

>trying to bait me into posting the algo

2/10

>>1058164

How can I make you post something which does not exist?

>>1058015

Why are you using C++?

>>1058083

> it's not cross-platform hurr

Unless the OP shows it is cross platform, NO FAG

wget -r 8ch.net

I don't believe that there's really that much worth downloading here that I should program my own downloader for 8ch.

This is mine.

#!/bin/bash

list-boards() {
    i=1
    while :; do
        curl -s https://8ch.net/board-search.php?page="$i" \
            | jq -rM '.boards [].uri' \
            || break
        (( i += 100  ))
    done
}

list-threads() {
    board=$1
    curl -s https://8ch.net/"$board"/catalog.html \
        | sed 's/>/>\n/g'\
        | awk -v b="$board" -v FS=\" '/goto_thread_catalog/ {printf "%s,%s\n", b, $2}'
}

dl-thread() {
    board=$1
    thread=$2
    mkdir -p board"/${thread%/*}" board"/${thread%.html}"
    curl -s https://8ch.net"$thread" \
        | tee  board/"$thread" \
        | sed 's/>/>\n/g' \
        | awk -v b="$board" -v t="$thread" -v FS=\" \
            '/class="fileinfo"/ {getline; printf "%s,%s,%s\n",  b, t, $(NF-1)}'
}

dl-image() {
    board=$1
    thread=${2%.html}
    image=$3
    if cd -- board/"$thread"; then
        wget -q "$image"
    fi
}

export -f list-threads dl-thread dl-image

list-boards \
    | parallel --joblog .joblog.boards  -r -j 4 --lb -- list-threads \
    | parallel --joblog .joblog.threads -r -j 4 -C , --lb -- dl-thread {1} {2} \
    | parallel --joblog .joblog.images  -r -j 4 -C , -- dl-image {1} {2} {3}

>>1059058

Unless you collect frog photos.

function downthread () { # Download all files in a thread
    link="$1"
    site="$(cut -d '/' -f '3' <<< "$link")"
    case "$site" in
        boards.4channel.org) trunk="i.4cdn.org" ;;
        boards.4chan.org) trunk="i.4cdn.org" ;;
        8ch.net) trunk="media.8ch.net" ;;
    esac
    wget -erobots=0 -nc -nd -nv -Rhtml,s.jpg -HErD $trunk $link
}

>>1059672

Java is a great language for web automation. I mean, you could use minimalist scripting methods like people are showing off here for simple tasks like grabbing images, but Java (and also C#, Python, NodeJS... but its easiest in Java) can actually launch a real browser instance and perform interactive commands using a user-centric API, using the Selenium library.

>>1059700

i forgot to mention its long as fuck but i had to get it for csci class, but hey learning a language isnt easy

>>1059700

Thanks, dude. I checked out a few of the chapters, and it seems concise compared to the stuff on the site.

>GUI

nigger

>>1059015

but, how is it working with the constant captcha?

use it in /pol/ and quote Hitler constantly

>>1060083

Nope anon, that's not just wget *.jpeg, it downloades all kind of files not just photos. (You) can also open files by double clicking the item from the list, you can move the files from the folder by dragging them from the list. I am working on new features right now in order to expand my programming knowledge, you can stay stick to wget and never learn anything new anon.

>>1060250

but the unixfags do not want one program that does everything

>>1060258

because i'd rather just download videos and not have an entire image suite that i don't need

>every one liner ITT is directed at cuckchan's cdn

way to off yourselfs.

Heres is one that isn't cucked:

wget -nd -nc -r -l 1 -H -D media.8ch.net -R html,txt,tmp -p -A.webm,.mp4,.jpg,.jpeg,.png,.gif $1
#$1 where the url goes

>>1060259

>>>/hydrus/ or get outta 8chan

>>1060259

why would anyone want to download the cutted up low quality videos that get posted on imageboards? i hate them and i hate the people that post them without source.

>>1060263

>when you don't know how to use the tool so you just copy&paste a line that someone else made

thats the reason why the posts are like that.

>>1059685

Java is SHIT

SHIIIIIIIIIIT

https://selenium-python.readthedocs.io/

https://nightwatchjs.org/

fuck your boilerplate

>>1064413

>muh dynamic typing

I honestly just feel sorry for you. Hopefully you can learn a real language some day.

>>1064433

>real language

>Java

Java is an interpreted language just like python or jewscript.

I honestly just feel sorry for you. Hopefully you can learn a real language (C/C++) some day.

>>1064436

>whatabouting this hard and still failing to prove a valid point

Java and nodejs are both compiled just-in-time, as is Python if you use the pypy implementation. However only Java provides static typing, which allows for far more powerful IDEs and easy API discovery.

>C/C++

Why would you do web automation in a systems programming language? Or are you just LARPing as hard as you can and throwing every buzzword you've learned against the wall to see what sticks?

>>1064436

>only compiled languages are real languages

Aw shit, here we go again.

>>1064477

I wish people would stop parroting this epic Stack Overflow meme. You are not parsing HTML in general but the very specific HTML that 8chan outputs, which you can actually do with regular expressions even if it's pretty ugly. Slapping an HTML parser on that doesn't make your code any more robust because you are already married to the specific structure.

>>1064480

>Slapping an HTML parser on that doesn't make your code any more robust because you are already married to the specific structure.

Lol are you kidding, that's exactly what it does. It is more robust to change; you're not married to a specific structure at all. Eg, the addition of elements won't affect a query by attribute.

>>1064496

>the addition of elements won't affect a query by attribute

Badly chosen example because that doesn't affect a regular expression for attributes either. Point being, the structure does not change very often and an HTML parser can get fucked by internal changes (e.g. a class name change) just as easily. If you're going to do something involved with the HTML, sure, do it properly. But to replace markup tags and extract URLs from an imageboard, this suffices.

>>1064504

>Badly chosen example because that doesn't affect a regular expression for attributes either.

Depends on the regexp.

>HTML parser can get fucked by internal changes (e.g. a class name change) just as easily.

Depends on the query.

A regexp is inherently more brittle. The regexp in this thread don't even parse HTML really, they're just looking for CDN URLs, and that will passively fail if more CDNs are added later. I'd prefer to grab them semantically and not rely on URLs, but hey that's just me.

>But to replace markup tags and extract URLs from an imageboard, this suffices.

Yes you're probably right, especially since the site is not under active development. In general, web scrapers go out of date quicker than any other software you can think of and rely heavily on functional tests if you need any sort of reliability, regardless of your approach.

>>1057955

Liking this pipe action.

>>1057848

>>1057836

>>1057831

>>1057791

>>1058063

>>1060263

These are all great.

>>1059672

Try this tutorial series for Java. https://www.youtube.com/watch?v=Hl-zzrqQoSE&list=PLFE2CE09D83EE3E28

Watching videos helps when you are just starting out. Whatever the lesson your are doing you can search video sharing sites for a tutorial or even sometimes a college lecture on the subject.

>>1059997

KEK

wget ‐‐recursive ‐‐no-clobber ‐‐no-parent https://8ch.net

wget ‐‐mirror ‐‐domains=8ch.net https://8ch.net

I should take all of these commands write a recusive random get flood script. Could be cool.

>>1057491

Found the nigger