>>978477

for those who use IPFS, i wont go into how to install it, look it up, it's a decentralized filesystem, this script allows you to wget a website and upload it to your ipfs repo

https:// github.com/VictorBjelkholm/ipfscrape

>>975521

Great guide, cheers anon

>>974637

Thank you TechExperts from a Research&InfoDistributor Luddite.

>>982172

Correction: Research-er

>>978833

>>978477

>>982172

Thank you TechExperts from a Researcher&InfoDistributor Luddite.

>>974690

>>974637

>>978477

>>978833

Brilliant. This was what I was asking for.

1. It should linked to at the top of each new bread's Resources,

AND

2. on first page of qresearch index with the "ARCHIVE, ARCHIVE, ARCHIVE EVERYTHING OFFLINE" instructions for "newfags"/"normies".

>>978477

Have used wget before but always with hesitation because the options don't seem to always do the expected.

When testing your suggested wget + parms I don't see any images being downloaded…?

Can you speak to this and recommend a fix?

On Dec 17th someone sent me this wget command which scrapes all the images referenced in a single thread:

wget -P ./thread/ -nd -r -l 1 -H -D media.8ch.net -A png,gif,jpg,jpeg,webm https:// insert_thread_URL_here.html

and it works but does not fetch the fullsize images, only the thumbnails, and I haven't figured out how to modify your wget mirror -the-whole-qresearch command to also fetch images and adjust references so the local HTML pages refer to locally-mirrored images.

I just don't have the time or patience to work this out.

>>974637 (OP)

>>978477

>>978833

>>983930

A serious question: How would a clever anon go about saving a person of interest's entire twitter feed from inception, including pics, in case of deletion?

I get the feeling that these methods can narrow select/focus the file folder to be copied, yes?

>>984029

I'm not really sure about twitter specifically; I don`t use it. But when researching POTUS' tweets I came across a couple of good websites and figured that they are using twitter's api.

For people that don't program, an "api" is an "application programming interface," which is basically a set of tools for you to get what you want, designed by the maker of the app. It works in the best interest of these social media companies to develop a good api because it allows others to use and display their stuff on other websites–free advertising and spread of influence. I'll look further into it, as it relates to an app I'm working on.

As far as using HTTrack on it, I haven't tried. For the most part I've had success with websites that don't involve user accounts; there may be a way around that, but again, I'm pretty new to it myself. Just taught myself like 4 days before Q highlighted liddlekidz.

>>984291

>>974637

>>978477

>>978833

>>983930

>>877198

>>122807

>>93735

>>3138

Thanks for your reply.

I know that it is possible to copy a particular Twitter discussion thread by printing a pdf, or by using 'ThreadReader' I think it is called, but again this is only for one discussion thread, not the root or branches, and in this case only if Twitter author allows it.

So the challenge for non-Tech anons like me is to find an easy technical way to copy all roots and branch discussion threads of a particular person/institution without copying absolutely everyone on Twitter.

Maybe there is an easy solution, if so great, that is why I am asking.

Thanking all anons in advance for their patience, consideration, and time.

We all know of instances where evidence has disappeared before archiving.

Non-Tech Anons need to archive particular person's/institution's complete thread discussions for possible use as evidence etc.

>>985562

Newfag here. Long-time lurker, first post.

Wrote a beta PHP script for Twitter that works in conjunction with Youtube-DL and wget (all freeware) to archive an entire conversation piece. Vids, whether native Twitter uploads or click-thrus to Youtube, are saved in their entirety, in best res available. Pics, PDFs, same thing. Any Web URLs, their front-page HTML is saved off as an HTML file.

Let me know if interested. It is by no means a finished product but it works very well as long as the convo is PUBLIC. Oh yeah, Instagram too.

I'm speaking from a Linux operating system perspective (though aspects may apply to the equivalent windows version)

>>978477

wget manual can be downloaded here:

>>>/pdfs/8640

this board doesn't allow pdfs

I would also recommend using the -U or –user-agent options to change how the website sees the wget program. (wget can impersonate a browser when making connections). This can get around some sites that actually look for and filter wget connections.

(see p14 of attached manual)

>>983930

It looks like you were in that poster thread anon ;^)

>it works but does not fetch the fullsize images, only the thumbnails, and I haven't figured out how to modify your wget mirror -the-whole-qresearch command to also fetch images and adjust references so the local HTML pages refer to locally-mirrored images

fetch fullsize: adjust your recursion depth, if I recall correctly from -l1 to -l2 as with -l1 you would only be grabbing that page's content, not anything it links to (the full size images). I think there was a way to only get the items at depth level 2 (the full size content)

adjust references: the pages should contain relative links from the current page to the other and not absolute links (i.e. page1.html has a link to page2.html not http:// somewebsite/fulladdress/page2.html). You may wish to look at using the -m option for site mirroring. WARNING: it has infinite recursion depth and can chew disk space as it attempts to grab anything linked, and anything those links point to, etc, ad infinitum! MAKE SURE -l depth is set to stop it.

See this webpage for more:

https:// stackoverflow.com/questions/4602153/how-do-i-use-wget-to-download-all-images-into-a-single-folder-from-a-url

>>984029

>>985562

>How would a clever anon go about saving a person of interest's entire twitter feed

Off the top of my head, you would be looking at a scraper script using curl for server requests, most likely written in Python, Perl, PHP, or similar. Search "twitter scraper" for lots of hits on the sort of thing you'd be using. There are lots of scripts on gitHub or similar.

Getting a list of URLs when you have 1 per line in a file "grab-these-URLs.txt"

wget -i grab-these-URLs.txt

Downloading videos from YouTube, Twitter, basically anywhere

youtube-dl -F http-URL-goes-here

will give you a list of the available formats to download with a CODE by each (on sites like YouTube) (e.g. CODE 18 RESOLUTION 1920X1080, CODE 22 RESOLUTION 1280X720….)

youtube-dl -fCODE http-URL-goes-here

will download that version specified by the CODE

youtube-dl http-URL-goes-here

will download the best quality version of the video (= largest file size)

Creating a folder for each day of the year on Linux not sure if you'd need this, but it was on my mind for some reason

mkdir -p {01,03,05,07,08,10,12}/{01..31} 02/{01..28} {04,06,09,11}/{01..30}

This will create a folder for each month 01-12. Inside each month folder will be 30 or 31 folders for each day of the month, except 28 for February (which can be changed to 29 for a leap year)

>>987185

Welcome!

A very useful, thoughtful first post. Not everyone can say that.

Sounds great.

I got nothing for Twitter now apart from pdf-printers so hell yeah, I'd love to test it if you're up for it. Anything is better than nothing. Thank you for your reply.

>>987352

Thanks very much for your detailed and useful post with easy to read, easy to follow explanations. Really great work.

Ok, I will try to follow through with all the kind anons' advice and test on sites and Twitter.

Thank you all anons :)

>>987185

>>987352

>>984291

>>983930

>>978833

>>978477

>>974637

You guys are like geniuses.

Gotta learn some coding.

>>974732

lol

https:// duckduckgo.com/?q=twitter+scrape+python&bext=msl&atb=v70-6&ia=web

>>987606

>When you want to admit it, you'll need it to be certified by the court,

You'll may well be looking at "digital timestamping" and creating file/document hashes. You might be interacting with a time server/authentication party via OpenSSL.

The digital timestamp proves it was created after some other document, which in turn was created after a different document, and so on, right back to the beginning of the chain - anchored in time. This proves 1) the file existed at 2) that point in time.

The file or document hashes are a one way math operation to reduce a file into a signature. Every signature of a file is different, and if you change one character of the file the new signature will change dramatically. This can be used to prove a document copy is identical to the original, or a downloaded file was not corrupted/intercepted during download.

For example, a free service for timestamping documents (1st result from a quick search)

https:// www.freetsa.org/index_en.php

There are other ways it can be done with file hashes in a crypto currency blockchain.

>>987606

Good advice, thank you.

For legal evidence archiving, do both a pdf print and program site scrape for context.

Both a pdf print and a site scrape may be necessary for court admission to prevent defense lawyers throwing out good evidence.

In reality I hope that our backups won't be needed for court cases, but better safe than sorry.

Almost any personal pain is worth it so see these legions of evil people be served the justice they rightfully deserve for their disgusting crimes.

We have it easy, think of the multitudes of victims who have no voice, and rely on us and other good people to investigate and give them a voice.

Let alone our relatives and the dead from the world wars, or more recent affairs. The scars are deep.

>>987515

While I'm fluent on normie platforms (fb, twitter, insta) when it comes to what I can/can't upload, I've never attached anything on 8ch. Can I upload a ZIPped archive that contains my PHP scripts? Or is a scanner gonna see scripting in my archive and go apeshit?

I would upload the source in plain-text (not copywriting anything here…) but…it's alot.

>>987712 Thanks lol, very droll, my searches were no use

>>987798 Thanks, will try

>>987830 Thanks, good point. Will experiment.

>>987840 Thank you. This might be the most elegant solution, but without pdf-print and timestamp. I will try this too. Thanks.

>>987937

Some other anons used arc hiv e.is for the first archives, but then switched to Meg aU plo ad.

Your work sounds very useful and pretty cool tbh, so I would still love to try it if you're ok with that. I don't know anything about locking down the plain-text source so it doesn't get f-d with. I assume that once it is there on Me gaU pl oad for example, then it is safe from alteration?

Gotta start some coding.

A useful one line script to grab pdfs listed 1-per-line in the file "pdflist", and convert each into text

for i in `grep -E '^htt.*\.pdf$' pdflist`;do foo=$(basename $i .pdf);wget $i; pdftotext -r 300 $foo; done

What this does:

for i in :perform a loop

`grep -E '^htt.*\.pdf$' pdflist` :search for any lines in the file "pdflist" using a regular expression where the line starts with "htt" and ends with ".pdf" - essentially this matches any URLs listing PDFs in the file.

;do foo=$(basename $i .pdf) :call a variable "foo" the basename of $i - this strip "http….some.thing/blah/whatever.pdf" down to "whatever"

;wget $i :grab the PDF at the URL

;pdftotext -r 300 $foo :convert the grabbed PDF "htt…../blah/whatever.pdf" into text using a print level scan resolution (300ppi) and save it in "whatever.txt"

;done :loop for the next URL found in the "pdflist" file.

I've used this when downloading Hillary Clinton Emails from the Judicial Watch website.

I end up with text versions of the 1000s of emails. I can then use the grep program to search through them all and get a list of search term matches.

grep -oHC4 pizza *.txt

search all files ending in ".txt" for the term pizza, print out the matched parts, along with the filename and in context of 4 lines (e.g. 2 lines before the match & 2 lines after)

The search term I first look for isn't "pizza" but "B1" because this indicates Classified Emails.

>>987937

I'd recommend using pastebin or similar and post the URL.

If it is a file collection then you could use filedropper, mixtape.moe, MEGA NZ, or similar to upload the zip archive.

If the files are small, you could create a b64 from the zip then upload the resulting text file to pastebin, but only oldfags/nerds may understand how to decode it.

>>988648

>You actually do kind of get a timestamp, in that the creation of the files on your computer have a creation date as they are written. So long as you don't go about editing them, the date remains intact.

This would be insufficient proof, since anyone can change the clock on your computer to give any date.

The "digital timestamping" and file hashes mentioned previously are a recognized form of document authentication.

>>987830

>>988648

>1 and 2 got me his profile, 3 got me his profile in a ton of different languages and maybe a month's worth of tweets (with Chinese headings).

You may wish to see if there are filtering options, so you can ignore content that doesn't match the filter.

So a depth of 3 + filter will ignore irrelevant information like the Chinese headings.

This would be equivalent to

-r -l3 -A ext1,ext2,ext3

in wget (to recursively grab to a depth of 3 files ending in "ext1", "ext2", or "ext3"

Thanks for the link. I used to be a fucking wizard on a PC, but got into another platform and still need to figure out a lot on this side of things for this platform. Thanks again, and I will post back if I find something useful.

>>984029

ask quinn michaels, he is a programmer

find him on youtu be,he programs bots that search twitter for all items related to a specific hashtag

Unbroken link test

https%3A%2F%2F8ch.net/qresearch/catalog.html

>>991469

No one knows who you are replying to unless they are linked in to your post. Click on the Post Number "No. XXXXXXX" that comes after the posters "ID: xxxxxx"

to automatically have it inserted into your reply.

>>992419

Use the correct thread for your posts, regardless of how excited you feel about something. That is irrelevant to this thread. It is like your sports team wins something, so you interrupt a meeting of strangers to shout about it - i.e. rude. Yes it is good news, and I read your post about it in another thread. Putting it here too is poor form, and spamming. Spamming is always bad etiquette.

>>995721

This test successfully kept the URL intact, but is still broken as far as the browser/ search engine sees it.

You can paste the entire address into a browser and go without editing.

Unbroken link test 2

https:%2F%2F8ch.net/qresearch/catalog.html

Testing to see if the browser, or 8ch filters the URL based on these characters.

Unbroken link test 3

https:/%2F8ch.net/qresearch/catalog.html

Testing to see if the browser, or 8ch filters the URL based on these characters.

Unbroken link test 4

https:%2F/8ch.net/qresearch/catalog.html

Testing to see if the browser, or 8ch filters the URL based on these characters.

>>995745

Tests 2, 3, 4,all produce URLs that are not broken by 8ch's board software, but the browser can interpret correctly to open a new tab when the URL is selected and right click menu activated.

>>995761

Clever workaround. I think the board owner threw the word filter in to avoid drawing the attention of "sniffer" programs. I doubt they would register your altered versions.

>>992419

It can also be 2017 National Champions / POTUS officially became CIC in 2017….

>>988857

I think I misspoke when I said "headings." What I meant was that the entire page is in Chinese, except for the tweet itself.

What's happening is that I'm getting copies of certain .html files in every language–1000 in total for the index, login, and search .html files.

It looks like they have a typedef set up to link a language to a 4 digit hexadecimal code which is appended like so: indexffbb.html, etc. Also, the other files have different designations.

It would be trivial to write a program that found the right one, but you need to know the right one beforehand in order to set up the right filter so the whole purpose is defeated. Who knows how often they change it? That having been said, it really doesn't matter–the files are relatively small.

I think a setting of "3" is best. I pushed it to "4," and ended up getting far more than I wanted. Once your download completes, look for the folder of the person who's tweets you're collecting, and they should all be in there. It will be in "project name folder" >> twitter.com >> "person whose tweets you're getting" >> status. For me they're in English.

The difference between a setting of "3" and "4", in my case, was about a 50x increase in my download size. Twitter put me on a time-out because of it, lulz.

>>1006231

Oh, and if the helps you wget users, the file structure is as I'd mentioned: twitter.com >> (twitter handle, without the '@' in front) >> status >> *.html

>>988857

By default, wget grabs everything so it's actually better to cast a wide net when mirroring unknown servers, as you said.. you can find more hidden gems this way. If you add the flags, you'll limit your retrieval to those filetypes.

For local archive searching, I'm using Java DocFinder with decent results, curious about other standalone options for a private/local fulltext search supporting fuzzy and near, etc.

This works great for saving this board, it was posted a while back.

wget -nH -k -p -nc -np -H -m -l 2 -e robots=off -I stylesheets,static,js,file_st

ore,file_dl,qresearch,res -U "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6

) Gecko/20070802 SeaMonkey/1.1.4" –random-wait –no-check-certificate https:// 8

ch.net/qresearch

Glad to see someone else noticed this limitation with wget. I do have the patience. Searched all over the web for a solution for a couple weeks now and even tried alternatives to wget but really wanted the wget solution to work. Finally wrote my own solution. It is a bash script. Couldn't get it to work with a one-liner but got it down to two wget calls and a couple sed calls to rewrite the img URLs. It's a very early iteration but I have a feeling it will have a few more before this is all done with. Calling it 'qarkzilla.sh'. I'm releasing it further down this thread so others can start tinkering with the idea and optimize it.

>>988088

can anyone give me a FQDN, to test an archiving script? Possibly bitcoin registered outside USA (dotCOM)?

Archive.is/fo/today is censored. I've had more than a few archive links change the url they archived and the domain searching yeilds nothing. All in the past few months, so I know it's related to Q stuff.

Just curious, how big in Gig is the Qresearch site when you use Httrack? I am trying out the Linux version, and it seems to be taking forever.

>>1010886

Thanks anon!

Why is the ssh key included?

> wget does not have the ability to download both the thumbnail pics embedded in the page AND the larger size images

? wget -l 2 would grab them, like this anon >>1009947 using the -m gives a mirror copy.

I wrote a oneliner to grab only the larger images a while back, but it was probably in the early hours in the middle of an autism attack, and I can't find it in my history…Ah!! I MIGHT HAVE MADE A NOTE! HANG ON….

rm 11245726.html*; wget https:// 8ch.net/pol/res/11245726.html; grep -Po '(?<=href=")[^"]*' 11245726.html| grep -vE '*html$'|sort -u |wget -nc -A jpeg,jpg,bmp,gif,png -i -

The leading rm is to remove the previously downloaded html, because this one liner was run as an update. You can use clobber options to overwrite the old file & the -N to "get if newer than current file" and leave off the rm really.

Summary of what it does:

wget -grab thread html

grep -parse html and find any download links

grep -find any of those links that don't end in html

sort -u -sort and remove duplicates

wget -grab the from those links if they are images

>>1004570

Those who aren't technically versed in creating shortcuts would be the ones who gain the most benefit from viewing (not posting) hex coded URLs.

As you correctly implied ("in most cases" but not all), not all systems would allow shortcut configuration either, and the prescribed method works for every case.

>>987937

Here goes…and we'll take it from here.

Sorry for the wait. Several components were gonna need to be prerequisites and, considering that not everyone's a tech anon, I put some extra work into rolling up the extra components into the existing folder.

———————————–

CTTPortable beta v0.1

-Tested ONLY in Windows 10 and Server 2012, 64-bit, but any 64-bit windows should suffice.

-Will require running from a Command Prompt

-Instagram features only include downloading videos and pics

————————————–

INSTALLATION:

————————————–

1. Extract the compressed folder anywhere. (My example will be c:\temp\CTTPortable)

2. Navigate to a specific Twitter conversation or thread in a web browser, and copy the URL. (My example will be the latest [InTheMatrix] thread: https:// twitter.com/intheMatrixxx/status/984559849873780736

3. Open a command prompt window, navigating to C:\temp\CTTPortable and type

php cttURL.php https:// twitter.com/intheMatrixxx/status/984559849873780736

If all goes smoothly, this will:

1. create a subfolder roughly named after the URL

2. Place a WGETed copy of the original thread file "984559849873780736.html"

3. Download locally all youtube videos referenced within the thread

4. Download locally all pics contained within the thread

5. Download locally any external website references (html,pdf,etc)

6. Generate a SECOND copy of the Twitter Thread("984559849873780736new.html"), substituting remote links with the locally stored copies of each respective file.

I'll post a couple instructional vids later if anyone struggles with this.

>>1017385

Size too large. Sharing from this link:

https:// drive.google.com/open?id=1flFOJafHIF9hTSybhZ4JuLv5k35nk9xx

>>1010995

That wget command is from:

>>493884

It's a 20+GB download of images and HTML for all of qresearch, takes quite a while the first time. May not want to run it., or remove the file_store from list of things to get, reducing size greatly.

>>1017595

I've done this before as well. It's the pics and vids that make up the bulk.

bumping a very useful and important thread

bump

bump

The dough of every bread has "we need better tools".

This thread discusses better tools.

BO requesting sticky please

>>1206291

Thank you BO!

Ripping an Instagram profile without login

The publicly viewable content in Instagram can be archived using Instalooter - a Python program.

https://github.com/althonos/InstaLooter

The usage manual is found at https://instalooter.readthedocs.io/en/latest/usage.html

To rip an archive for a user called "foo-bar":

python -m instalooter user foo-bar -v -d

-v option rips videos as well as images

-d option rips JSON metadata containing comments found on the Instagram posts

A helpful one-liner to use with the JSON files saved - Searching the JSON metadata for keywords of interest:

grep shortcode `grep -li pizza *.json`|sed 's/^.*"shortcode":\s"\(.*\)".*/http:\/\/www\.instagram\.com\/p\/\1/' -

This will run a case-insensitive search for "pizza" and return all URLs in the Instagram profile that contain that term in the comments.

>>1010995

I never completed an enire download. I think I got to 11 GB one time and figured something was off (I was just trying to get one thread). If I were to stick to a depth of one, it probably would've worked.

>>1017595

Knowing there's an upper limit is helpful. 20 GB isn't as bad as I would've imagined.

comms are unfolding.

-you already know

>>1214842

>comms are unfolding.

May your origami be beautiful and to your liking anon.

Recommendation by anons to use clipconverter.cc for making video clips.

https://www.clipconverter.cc/

Those who are more technically minded can make use of youtube-dl as described above in this thread.

>>1251422

We need a spiritual cooking element to the whole thing. Show the entree' like one of those sick parties where someone is laying on a table.

Cocktail…blood bag?

Illuminati overlord Albert Pike explained in 1871 that the Third World War will focus on the mutual destruction of the Islamic World and the Political Zionists.

Cabalists planned to initiate this war that will bring ‘complete physical, moral spiritual and economic exhaustion’ for decades. The entire populist/Trump/Q movement has arisen at this time for the sole purpose of preventing the ultimate conflict before it starts.

The catalyst that will initiate the Third World War was to be an Iranian Nuclear strike on Israel, conducted with a nuclear weapon built in Syria, containing ‘Russian’ Uranium (U1). The Iran deal will ensure immediate US involvement and the Russian aspect would be used to foment war between the EU Countries and Russia, thus dragging the whole of Western Society into a Global Conflict.

https://www.youtube.com/watch?v=9RC1Mepk_Sw

https://www.youtube.com/watch?v=yWAFvIT-NHs&feature=youtu.be

North Korea was to be used as an agent for initiating conflict in the East. A Medium-Range ICBM was provided to the regime to engage Hawaii. The ‘false alert’ on Hawaii that occurred several months ago was a test missile, launched from NK. The missile passed over Japan, causing a simultaneous ‘false alert’. This secondary nuclear attack will drag NK and China into the World War.

Patriots have installed themselves in Jordan. They will attempt to intercept the Iranian missile that would strike Israel, thereby preventing the Third World War. However, war may still be declared on the grounds of breaking the terms of the Iran deal.

Everything has been leading up to this moment, this is a crossroad in our civilisation.

SA -> NK.

NK -> Armenia.

Armenia -> Iran

Iran ->ENDGAME

# OS X + httrack command line

Steps to install `httrack` on the OS X command line and run it. `httrack` has a lot of options to play with depending on your desired results.

# Install homebrew

https://brew.sh/

# Install httrack

`brew install httrack`

# Make a directory for your httrack 8ch files

`cd ~/Downloads`

`mkdir 8ch`

`cd 8ch`

# Run httrack

`cd ~/Downloads/8ch`

`httrack https://8ch.net/qresearch/`

# httrack update

`cd ~/Downloads/8ch`

`httrack –update`

# httrack help

`httrack –help`

`man httrack`

Requesting modification to (and reminding people of) the search script here: pastebin.com/tM53Q6AM

The original works by searching in arcdir/qresearch/res. It only works in one folder at a time. A modification is needed to search my /bread/qresearch/.zfs/snapshot/*/qresearch/res. Note the asterisk. If I run ls /bread/qresearch/.zfs/snapshot/*/qresearch/res then I can successfully get a wall of text.

Not sure how to do so myself. Doesn't look like a simple oneliner to me. I've also thought to modify the temp dir to use the folder I run the script from for temp (since the snapshots are read-only and we can't modify hundreds of folders to have a temp folder).

This is what I get when a just change the arcdir:

# Error! No HTML-files found in "/bread/qresearch/.zfs/snapshot/*/qresearch/res"

# Please check if archivePath ("arcDir=…") is set correct.

I'd like to see if I can cause any headaches revive any dead posts.

Should PROOFS/Side x Sides be posted in research threads or is there a designated thread for them?

>>1265626

>This is what I get when a just change the arcdir:

I have no idea what line you changed, since there are several places in the script you could have done that, or what you changed it to, as your question is fairly vague.

However, I set up the necessary directory structure, and tested the script, and had no problems with it searching multiple directories.

For instance, I used a glob (i.e. * ) to pick more than one directory on line 11:

arcDir="${HOME}/../../tmp/a/*/b"

Works as expected, and searched through multiple directories.

My guess is you changed line 11 from:

arcDir="${HOME}/archive/qresearch/res"

to:

arcDir="/bread/qresearch/.zfs/snapshot/*/qresearch/res"

but the archive is in your home directory, not root directory???

If so, line 11 should be:

arcDir="${HOME}/bread/qresearch/.zfs/snapshot/*/qresearch/res"

which would specify the correct path.

Otherwise check you really do have any html files in those directories.

Help. Not tech savvy. Received a message on my computer to call Apple Support "NOW" regarding security issues. The message listed my IP address and gave me a case number, which has me freaked out b/c the end of that case number is: ….-qch8nt Could it be someone from 8ch is trying to contact me?

>>1276775

>fairly vague

..and by "vague", I mean the problem is clear ( I understand what you're trying to do, and what the end result should be), BUT the specification is vague (your description isn't specific enough to know exactly what you're doing wrong, so I'm guessing where you need to fix to get the required result.)

>>1276833

Likely a scam.

Never bareback an imageboard, ie. use a VPN. at least.

Highly doubtful anyone would contact you, since this is an anonymous imageboard.

>>1276872

Thank you. Sorry, but I'm a tard and don't even know what "bareback" is or ow to use a VPN

>>1276914

Also, I remember one of Q's posts saying we would be receiving a scary message, or something to that effect, so thought maybe this tied in to it somehow.

>>1276914

>>1276929

bareback = connect from your home router without any proxy to cover what your home IP is.

VPN = virtual private network. You connect to any of the VPN companies proxy servers to get an IP in the country of your choice, so it makes it look like you are connecting from somewhere other than your home. (There's a lot more to it, but that is the basics)

Also, in certain countries like China, EU, UK the government collect what websites you visit and rank their citizens accordingly.

>>1276987

This grannyVet thanks you, not only for helping me, but for helping to get the truth out to the world.

>>984291

>>987600

>You guys are like geniuses.

Yes, thank you anon(s).

>>984029

>saving a person of interest's entire twitter feed

Just attempted. Account wasn't set on private, I wasn't a suscriber. HHTrack hummed along 15h, was up to 30Gb on last check, and it just finished with this error:

>15:31:39 Panic: Too many URLs, giving up..(>100000)

The site did mirror, but only to about 20 tweets.

So it was a fail. If you have any ideas HHTrackanon, I'd be grateful.

I know NSA has it all ultimately, but I suspect this twitter account has leads to high level tech elite pedo stuff on west coast. I don't want to publish to anons w/o mirror. Meantime, I'm just copying to word from time to time. -Ty

>>1276833

Just FYI, there are a lot of hoax websites that will pop up a scary-looking message telling you that you've been infected. Then, they'll give you some fake contact information, you'll contact them, and they'll try to convince you to do something that ultimately will allow them to screw you. They can be very, very tricky, but believe me: no legitimate business operates that way.

Here's how you deal with those kinds of things: close the tab in your web browser that the scary page is on. If you only have one tab open, open another and close the bad one. If that doesn't work, close your web browser (Opera, Firefox, Chrome, or whatever you use to get on the internet). If you open it again and you get the same message, you need to change your homepage to something normal, like startpage.com.

Don't ever, every contact those people–they're lying. If you're really, really worried, take a picture of the message and the web page address, then call Apple and ask them about it.

Hope that helps, and welcome to the internet–you'll get used to it in no time :)

>>1277388

It sounds like you just need to change one setting–see this post:

>>988648

I think a mirror depth of "2" is enough. This is the thing–you won't be able to open the folder, click on "index.html", and get all of the posts up like you would if you did an advanced search…but if you look in the folders, you'll find one that's named the same as the person you're trying to "grab." Inside that folder will be all of the tweets–you can double-click each one and take a look. It's not pretty, but it's pretty thorough.

Also, make sure that you do an advanced search before you start mirroring, with the start date as far back as you think you'll have to go to get them all.

If a "mirror depth" of 2 doesn't get you everything you want after doing both of those things, then go ahead and try "3". But don't bother going up to "4," because it gets way out of hand.

Another note: if you didn't use a vpn, odds are that twitter is going to rate-limit you after such a large download. You'll have to go through a proxy if that's the case. Good luck!

>>1277909

Thank you, as well for helping me. Again, what really spooked me was the end numbers/letters of the "case number," which was qch8nt (Q + 8chan.) You guys and gals are the BEST

>>1278055

Ty anon. lol at the large download. I should have used a VPN, but didn't. This anon has always made a clumsy operative–either sperg out on every detail or just decide fuck it and dive in headfirst.

I'll report back if I make progress.

Thanks again for your detailed help.

Much appreciation

https://8ch.net/qresearch/res/974637.html#1278055

https://www.youtube.com/watch?v=HzCiPjS7KeI

Different way of archiving but it can archive all media, links, etc of a site. You can run it locally if you have any skill or run it from the webrecorder.io site.

>>988164

>for i in `grep -E '^htt.*\.pdf$' pdflist`;do foo=$(basename $i .pdf);wget $i; pdftotext -r 300 $foo; done

No need for the basename strip as the shell can handle string parsing, so:

for i in `grep -E '^htt.*\.pdf$' pdflist`;do wget $i; pdftotext -r 300 ${i%.pdf}; done

is equivalent.

¢♄Δ⊕$

Fireshot works for easily saving high-quality PDF renders of webpages. Can't handle really long webpages, however.

>>1278294

don't worry about it–I speak from experience re: the large download. And I sperg as well. You might notice some big posts where the author calls out and corrects mistakes right after putting it up–that's me a lot of times.

>>1278274

I would've made a double-take as well.

I notice that I get a "Q" in around half of my captcha challenges. If you figure 26 capital letters, 26 lower case, and ten digits, that makes for a one-in-sixty-two chance of getting a "Q" for each character. With six characters, the odds of getting one should be something like 9.3%. Superstition or not, it gives me comfort.

It's funny how noticing patterns can either make you an idiot or a genius. It's probably the most important predictor of both mathematical ability and paranoid schizophrenia, ha ha.

>>974637

now i can finally download terabytes of my interracial lesbian cuck midget tranny porn….

>>1290990

…is that you sucking up all the bandwidth on my server?

WireShark will help you see whats going where

# Twitter: Scrape / download a user's tweets on OS X

Isn't real pretty but it worked for me

Doesn't require a twitter API key

Scrapes twitter search instead

Most steps are performed in an OS X Terminal/shell

Requires basic shell experience

### Install homebrew

https://brew.sh/

### Install jq

`brew install jq`

### Install TweetScraper

https://github.com/jonbakerfish/TweetScraper

`cd ~/Downloads/`

`git clone https://github.com/jonbakerfish/TweetScraper.git`

`cd TweetScraper/`

`pip install -r requirements.txt`

### Run TweetScraper

Here, `SaRaAshcraft`, is an example twitter user name

`scrapy crawl TweetScraper -a query="from:SaRaAshcraft"`

`cd Data/tweet/`

`find . -type f -print0 | while read -d $'\0' file; do jq 'select((.is_reply==false) and .is_retweet==false) | .text' $file ; done > ../saraashcraft-all.txt`

### Open your new text file

Use any text editor to open your new `saraashcraft-all.txt` file

`vim ../saraashcraft-all.txt`

When POTUS speaks at various events he should ensure that there are strategically placed mirrors behind him that forces media to show the crowd!

>>987600

Teleport pro

greatest tool for archiving - ever

www. tenmax.com/teleport/pro/home.htm

Once you have the data mirrored, this is awesome for smaller filesets, handles Q archive fine, with regular expressions "near" phrase, synonyms, etc. No image searching, but does filenames and content of text, html, pdf, etc.

Open Source, java, increase java memory limit if doing dozens of gigs of text/PDF.

http://docfetcher.sourceforge.net/de/index.html

>>1344533

Eyes Wide Open?

<iframe width="504" height="283" src="https://www.youtube.com/embed/tpH5L8zCtSk" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

Kek! These are some great titles, Anon… thanks for the morning laugh.

Related: I cancelled Netflix because they went full-retard with their Soros-level programming. Namely the show that about how shitty white people are. When I heard Obummer and Rice had partnered with the company, that was it for me.

Unrelated: I can't stop seeing Biden in this pic as Simple Jack.

>>1344533

>>1344502

>>1434024

>>1485667

>>1497828

>>1497913

>>1498246

>>1498284

>>1498327

>>1498375

>>1498428

>>1498641

>>1498686

>>1498919

>>1504090

WRONG THREAD. USE THE CORRECT ONE

This thread is for contributing methods to archiving evidence.

>>1505178

This is what happens when bad programmers dont know how to program for humans…and think all humans will understand your code structure.

its called updating your site beyond 2002 style.

Thank you

– The autists and The anons and other organized diggers who have moved beyond 2002.

>>1505538

>its called updating your site beyond 2002 style.

>it's not my fault I didn't read what page I was on, it's someone else's fault.

No, it's called "reading the page you're on and posting accordingly - not being lazy and blaming someone else."

>updating your site beyond 2002 style.

This site can have ANY STYLE YOU CHOOSE!

You can write your OWN style/theme and use it under the options mention. So don't blame "2002 style" for your own laziness for 1) not finding this out by reading the FAQ, and 2) not writing your own style.

sage for off-topic

>>1505995

>mention

*menu

>>1251964

The Hawaii Missile alert was at 8 am on Saturday morning. The Japan missile alert was on Tuesday. They were days apart, not simultaneous.

>>1578565 (pb)

Watch the water - sky event

Population culling

>>1504090

>When I heard Obummer and Rice had partnered with the company, that was it for me.

But Q told us that they were on a plane to GITMO.

>>1585689

Don't remember seeing that post. Which post number was that again?

What is the expiration time for images not frequently accessed on 8ch.net?

I'm trying to do an independent archive of the qresearch breads. While I can still pull down all the old html files, I find that most of the images and thumbnails for the early days 404 out. The ones that still work are mostly frequently used memes. This leads me to suspect that files which are inactive for a specific period of time are discarded to same space.

Its a shame. Historians researching America's second revolution 100 years from now might like to seen that stuff.

>>1284257

>for i in `grep -E '^htt.*\.pdf$' pdflist`;do wget $i; pdftotext -r 300 ${i%.pdf}; done

It isn't equivalent, since basename strips off the leading http URI element. ${i%.pdf} is OK for a filename but not a URL.

You could do a more complex match, but might as well use basename since it will handle any unusual cases more smoothly.

sK:dd

>>1657831

Ask the BO they might have a better idea.

At a guess, it might be a frequency of caching algorithm. The older and less requested files 404.

>>1657831

Images hosted on 8ch expire when they are bumped off the catalog in to the archives. Otherwise, there is no timed expiration that I'm aware of.

The 8chan servers must accommodate nearly 18,000 boards, with almost 2000 posts per hour, many of which will include images, PDFs, SFWs, oekaki pictures, etc.

CodeMonkey would have all the details. admin@8chan.co

>>1659588

I was going to say, that seems to be the delimiter. Threads in the archives lose images, but not all of them. I don't believe it's based on size–there are some big images left, while small ones are lost.

All the more reason to archive threads individually, or archiving the whole board periodically–say once a week, or whenever you think something epic is happening. Even if an image is deleted by the person that posted it, you'll still have a copy on your hard drive if you catch it in time. This is especially important when we get visitors :)

>>1706888 weather Q

https://www.youtube.com/watch?v=JeqHOhtlhQ0

>>1498919

Great stuff anon

>>1736185

You Clowns sure are terrible at memes….. I mean this is below the IQ of Forest Gump.

Put on some cloths, put down the peanut butter and hang your head in shame you idiot shill.

Send it to corsi, he will like it and tell you how great you are…….faggot

>>1777260

Looks like he has a lazy eye (points a little off to the left).

Wray's Face and Answer at the end of this video.

>>1807836

Q is not a LARP;

Q is a master disinformation spreader.

He has fooled everyone.

The optics are now SO bad on the administration that even I believe that Trump is Hitler for "caging" children.

Horowitz and Wray have NOT delivered. Horowitz provided the FULL report, not a REDACTED report like Q said.

This is now a massive goat fuck.

Horowitz: a deep state lifer.

Wray: a new, indoctrinated deep state lifer.

>>1815238

IT APPEARS THAT YOU ARE CORRECT.

ONLY TIME WILL TELL…BUT THIS IS HOW POLITICIANS OPERATE, THEY OBFUSCATE, DISTRACT, AND MUDDY THE WATER SO MUCH THAT NOBODY WANTS TO HEAR ABOUT IT ANYMORE…THATS HOW THEY BURY PUBLIC, INTEREST, DISSENTION, CONCERN, ETC.

SO BY REPEATING TRUCKLOADS OF INSINUATIONS, AND BLOVIATING SO MUCH BULLCRAP, THE FACTS BECOME LOST AS A NEEDLE IN A HAYSTACK, SO MUCH SO THAT NO VOTERS KNOW WHAT TO BELIEVE.

HOPEFULLY PEOPLE LIKE GOWDY WILL BE HEARD AND PEOPLE LIKE CUMMINGS WILL BE IGNORED. THESE DEMOCRATS ARE VERY GOOD AT CONFUSING OR IGNORING THE FACTS. MIXING THE SUBJECTS TO MAKE A POINT ON TV RATHER THAN FOCUSING ON WHY THE WITNESS IS THERE.

BAIT AND SWITCH. WITH AN AGENDA…..

>>1815238

Fuck off clown

Q–message from @prayingmedic

Congressional Research Service major swamp

Medic's contact former military

States CRS needs drained in a big way.

ThankQ

>>1010886

Thanks, you inspired me to upload a few scripts that I have been using to track all the places in the qresearch threads where someone or somebot has written a post with 1 or more lines that are of the form "X = Y".

>https://g ithub.co m/daizyr/qresearch-extract-equals

Instructions to run it are in the readme. The resulting file is called "training.out" because at the time I wrote it, I was actually trying to demonstrate it was used for bot training data (I still think it is, but to a lesser degree than back in the day).

Anyway, I think it's still relevant for tracking and archiving things. I hope others find it useful. It just downloads all thread HTML, no attachments or thumbnails. It is also a good basis for building tools that download all HTML, then analyze the contents of each comment. Threads are inspected in numerical order (so older threads are processed first.)

Example output for a specific thread in training.out (tho this file contains exerpts for ALL threads processed). It is suitable for posting as a comment/threadlink elsewhere in 8ch:

>[2018-06-20T03:13:27Z] Q Research General #2298: The Shills Should Masturbate Edition

>>>/qresearch/1823809

>css = cascading style sheet

>Trust the plan = also

> Occam's Razor = D = Donald

>pass = qanon:qanon

> As D5 = 45 I thought D meant Delta

>cut off soros funding bc of human rights abuses = no more antifa

> [1] Do you think the delta between posts and tweets could be who is posting at any given time? [1] Delta = POTUS ? Just a thought

I do not plan on maintaining this code, but feel free to fork it and use it. It was meant to scratch a specific itch. not to become a maintained software project.

>>1847914

full example output of training.out:

>https://pastebi n. com/7mHZLkBp

Conquer we must as conquer we shall.

Watching The Darkest Hour today. Was listening to Churchill's 1st radio broadcast to the people w/the direct address regarding matters in the world. His details might have been off, but his motive was pure. It was time to rally the people for a mighty fight. And he succeeded in doing so by using this approach. The victory was for all. So grateful for this motive being held in the hearts of my patriotic leader & his teammates.

So grateful for all of you. I stand acting in part in any way I can imagine to. Happy Independence Day!

>>1531142

Can an anon post a command line version of this using HTTrack.exe? Scriptable version = more powerful than GUI. Wget isn't getting me what I want, this might do the trick. Thank you.

>>2054108

I too have been having wget problems. It doesn't regrab the html files for me, leaving a lot of partial breads.

I found this page: https://www.archiveteam.org/index.php?title=HTTrack_options and this page.

I've been practicing on the comms board, it should be about 600 megs, httrack is grabbing too much for my preferences. Currently it's at 3.6 gigs so I canceled it. Gonna go back and read the manual and any examples I find and if I find something that works I report back.

bump

>>974637