/comms/ - COMMS

Verification

Go to https://wearethene.ws/about which has a link back to this thread now:

>If you'd like to get in touch with the maintainers of this site, you can post on the /comms/ board on 8kun here: https://8kun.top/comms/res/7366.html

>>7366

you are more than welcome here

If you look you'll find that threads can last for years here.

actually your posting here reminds me of something I'd forgotten

>>7368

Thank you Rusty! o7

>>7493

Thanks for your service baker, and welcome. Without you none of this would be possible at all!

>My question is, sometimes notables seem to get truncated, for instance if a baker posts #NNNN Baker Change, or embedds new baker within the notable bun does this cause notables to be truncated?

When people add the "baker change" lines to the notables this shouldn't cause an issue. Be careful not to add any blank lines in the middle of a notables bun though.

Splitting notables across multiple posts is not currently supported, but this is something we can probably fix pretty easily.

One thing that does cause issues sometimes is splitting a notable across multiple lines where only one line has the notable title. It's better for our scraper if long notables are all on one line, or if each line is tagged with "Notable title (pt1)" and "Notable title (pt2)". Long notables can make for a good, article-like read, so at least for wearethene.ws having a single notable with lots of posts isn't a problem.

Whatever happens, one reason we started this thread is so that bakers and anons can call out any issues, things like this are pretty easy for us to clean up but we don't always see them. So please feel free to post here if you see anything, something like "bread number 12345 is missing a few notables", and we'll take a look. Over time this will help show us whether we need to change something on the site.

>Have you thought about bit bucket, or git and opening development to the community (Language?)?

More anon contributions would be great, but exposing the site's code publicly might make us vulnerable to doxxing or other unpleasantness. That is something we will keep thinking about. Maybe things will change after MARCH MADNESS is over.

>Splitting notables across multiple posts is not currently supported

Here's what this means, in case it wasn't clear:

#12345
>>8888888 some notable
>>8888889 another notable
--- POST ENDS ---

--- NEW POST STARTS ---
#12346
>>8989898 new notable
>>8989899 super notable

With the way the site works right now, the #12346 notables will not be picked up from that bread because they were posted in a separate post. This usually isn't a problem because the same set of notables will be posted in a few breads in a row, and this is probably something we can change if bakers want to keep doing this.

We also look every few days and make sure there aren't any breads missing or numbered incorrectly.

Thanks very much for setting up this thread!

>When people add the "baker change" lines to the notables this shouldn't cause an issue. Be careful not to add any blank lines in the middle of a notables bun though.

Many, many thanks for the clarification. Baker here. "Baker change" matters to bakers bc it's important to know who is responsible for what in each bake. Somebody said, oh we can't do that any more bc is messes up the aggregators. Glad it doesn't.

I know you guise don't like buns but never heard about "baker change" problems. Sounds like there aren't any, kek.

RE buns:

Yes, i still use them sometimes. Used them a lot for Rusty's photos last spring, for example. But here is what i do now: If a bun is really needed, I make one up. But in most buns (not photos but other kinds), there are just one or two super substantive posts. So i will put the two "key" posts first, followed by the bun link:

>>33486, >>33996, >33998 Notre Dame DIGG

where

>>33486, >>33996 = super substantive posts

>>33998 = bun link

that way, wearethene.ws readers get most of the "meat"

but i can also include secondary info (details, anon discussions, etc.)

Appreciate your feedback on this, bc we currently have a baking class and we're about to talk about notables. If you have anything to share about how bakers can supports wearethene.ws in their notes–and you want bakers to see it soon–reply in the latest bakers live class thread. We are in there a lot between class, so it's a good place to dialog.

>>7568

literally the "next-door" thread at the moment.

will post >>7582 link there

(not sure i understand the example in >>7569)

>>7985

sorry, two wrong links, must need moar covfefe:

should be

–will post >>7982 there (the wrong link i posted is actually in the latest baking class thread, kek)

should be

– not sure i understand the example in >>7984

>>7985

>I know you guise don't like buns

Don't worry about this, one enhancement we've considered is making it possible to see pics/videos on posts that expand as part of a bun. The only drawback would be that wearethene.ws won't save the images so they would disappear from public view when they disappear from 8kun.

>So i will put the two "key" posts first, followed by the bun link:

This is a good strategy also.

>If you have anything to share about how bakers can supports wearethene.ws in their notes–and you want bakers to see it soon–reply in the latest bakers live class thread.

The biggest thing is get the fking bread numbers right in the titles and the notable posts!!! Kek! Mistakes happen so triple-check the numbers.

>not sure i understand the example in >>7984

Right now when the scraper looks at a new bread, it expects all notables for past breads to be in the same post. If previous notables are posted in two separate posts then only one of those will be picked up.

>>7987

just posted after you next door, ha-ha

>>7987

good luck on asking bakers to ALWAYS get the bred numbers right! Triple checking the numbers–you mean, like when you take over a fast bake @600 and then Q posts?

ha-ha-ha…..

>Right now when the scraper looks at a new bread, it expects all notables for past breads to be in the same post. If previous notables are posted in two separate posts then only one of those will be picked up.

hmmm….that applies to my baking style, because i sometimes cross-reference pb posts, to make it easier for diggers to see the context

which post gets picked up, btw? the first or second?

>>7993

>hmmm….that applies to my baking style, because i sometimes cross-reference pb posts, to make it easier for diggers to see the context

No, pb posts in notables are fine.

>which post gets picked up, btw? the first or second?

the first. I don't think I explained clearly though, I'll link to a real example next time I see this.

>Triple checking the numbers–you mean, like when you take over a fast bake @600 and then Q posts?

ha, ok. we'll keep cleaning these up when we see them, your efforts are appreciated anyway.

>>7995

thanks–RL examples are good, i'm collecting them for the baking class, so new bakers can see RL problems that really crop up. My favorite example is "how to know when a cat is baking"

>>7363

doc, there's also stuff in class #5 thread.

pretty funny to go on a digg for wearethene.ws on google and then find it literally next door.

>>8037

pulled it up, thanx

>>8245

My apology, somehow I accidentally posted this twice. I think because I had chose to archive it, but was too large according to the warning notice, so I unticked the Archive button then retried posting. Maybe after I unticked the Archive the post box, it was still going to post it anyhow since I had initiated the process already.

I'm not a member of the boards. I've browsed it once or twice but I typically read from a news curator like wearethenews.ws

Anyway, I watched this video about Space Force and I've never seen it shared before https://youtu.be/KsPLmb6gAdw

I think it may be one of the single most important videos out there. For me it explains why China may have released COVID-19 on the world and America. To slow us. To cripple us. To roadblock Trump a little.

https://youtu.be/qm4LicEB5o8

The site hasn't updated in about 17 hours. Since Q Research General #11019.

>>9233

>>9234

Thanks anons. This was fixed last night.

Thank you wearethene.ws whoever you are. Know your service is appreciated and is notable in my book.

Some Random Patriot

>>9287

We're happy to help, patriot. Remember to spread the word on social media.

#CoronaVirus #Covid-19 #5G #NaturalRemedies #HydrogenPeroxide

FYI; for your consideration

Please watch this video and I give links to natural remedies for you to consider. Do your own research. 10mins 01sec ΝYC ΙCU DR with nothing to gain shares INCREDIBLE insight; Apr 4, 2020… https://youtu.be/1EWQPgF6-UQ

Natural remedies:

https://archive.earthclinic.com/remedies/hydrogen_peroxide_inhalation.html

*

https://archive.earthclinic.com/remedies/food-grade-hydrogen-peroxide-cures.html

*

6mins 10secs Learn the Right Way To Do Hydrogen Peroxide Inhalation Method for Viruses, COPD, Congestion

•Dec 3, 2014 https://youtu.be/TAHhe7hJ0Ug

*

where to buy world wide: https://archive.earthclinic.com/remedies/hydrogen-peroxide-where-to-buy.html

*

where to buy USA, I bought locally and not easy to find. Not sure about hydrogen peroxide shipping regulations. If you find resources in your area please inform others in the comments below. https://archive.earthclinic.com/remedies/hydrogen-peroxide-where-to-buy12.html

PLEASE SHARE WIDELY

Your article on Jason Reid-Kolsevich says he graduated from West Chester University in 2010. Here is the archived link to the WCU 2010 yearbook and there is no Reid or Kolsevich.

I will also add link to the WCU archived yearbooks in case of a different year unless there is more fuckery than we know.

https://ia800301.us.archive.org/7/items/serpentine2010wcu/2010Yearbook_live_text.pdf

https://library.wcupa.edu/c.php?g=61441&p=395885

>>10867

>>10868

RSS feed is on the list of planned enhancements for wearethene.ws.

>>11587

Welcome newfag. Here are the steps for posting on the Q board correctly.

- Go to https://8kun.top/qresearch/index.html

- Find the latest "General" thread (Ctrl+F for General)

- Click the title to load that thread

- Post a reply there instead of making a new thread. (If the thread already has more than 750 posts then that means you picked an old thread and you need to go back to the index page and try again.)

Sent an email over to admin from 8kun and they responded that I needed to contact admin over here at wearethene.ws.

Please contact me at the email address provided or if you leave one here I will contact you directly.. Thank You

>>12427

Please don't post your name on the boards, and get a protonmail.com or other disposable email address.

It's also probably better to handle inquiries in public, just ask right here in this thread.

Namefagging and posting with tripcode. Note same ID (611d80) as original post >>7366 and >>7367.

Hi.

How long after a bake is completed, doe it take fo that bread to show up on the aggregators?

context of question --> >>13433, >>13440, >>13441, >>13443

5G MindControll Covert Microwave Attacks On Innocent Citizens

https://www.bitchute.com/video/wIsO32WVKri8/

https://www.brighteon.com/40a17350-dbc4-4eef-933b-13eedf6557c9

I think i got some good info. Hey newfag(very fast learner so let me know of my mistakes where i make them) here so bear with me but i think i got something juicy for veteran anons think about.We own a monopoly on oil production now right? Who's economy runs strongly on oil and refuses to care about the environment? Why did Potus fill his oil reserves and strike while iron was hot. Now that Potus is in control of the oil and China is a big importer of oil for their economy to even function. Could Potus be planning to tariff the shit out of the oil to china once we release the WHO investigation findings? Think about a modern day Anaconda Plan. Constricting them by limiting their primary energy source and forcing them to play ball or face ruin. The Chinese people are not too happy with with way things are run. If we don't declare formal war on them even though we all know China and gates + globalist cabal just caused an act of war with corona, he could diplomatically disarm the Chinese government while simultaneously giving the people of china a reason to revolt again they can't be too happy after being locked in their apartments then they loose power and ability to transport. Could this be the way things are moving? As I said i am a new fag but i have been following potus very closely watching the news daily (i don't sleep much either) and i find myself days sometimes weeks before potus does something, i'll say this is exactly what he should do. Like when the virus started i was wondering why the hell he was taking so long to ban the flights (now i know he was fighting a whole room of 20 people against him weighing the options out) But unless i'm mistaken don't patriots now control the oil supplies? don't get me wrong middle east might be a lot tougher to hold since china can reach it from their mainland and we are further, but we took iran for a reason right? and Q said Israel is next. Where does Israeli get gobs of money? from one of the strongest economies who play ball with them the globalist Chinese who use shitty business practices to rip us off. Cut off more of israel's money flow and they will be hurting even more allowing us to constrict them much like the anaconda plan. Like I said newfag but i'm very curious on this. first real post lmk where i fucked up as i'm sure i did in places, only learn through diving in though. at least i'm not doxxing myself by filling in the name line haha 5:5 frens i love and respect the work you've all been doing risking your lives many of you, i'm trying to catch up on everything i can learn

I think i got some good info. Hey newfag(very fast learner so let me know of my mistakes where i make them) here so bear with me but i think i got something juicy for veteran anons think about.We own a monopoly on oil production now right? Who's economy runs strongly on oil and refuses to care about the environment? Why did Potus fill his oil reserves and strike while iron was hot. Now that Potus is in control of the oil and China is a big importer of oil for their economy to even function. Could Potus be planning to tariff the shit out of the oil to china once we release the WHO investigation findings? Think about a modern day Anaconda Plan. Constricting them by limiting their primary energy source and forcing them to play ball or face ruin. The Chinese people are not too happy with with way things are run. If we don't declare formal war on them even though we all know China and gates + globalist cabal just caused an act of war with corona, he could diplomatically disarm the Chinese government while simultaneously giving the people of china a reason to revolt again they can't be too happy after being locked in their apartments then they loose power and ability to transport. Could this be the way things are moving? As I said i am a new fag but i have been following potus very closely watching the news daily (i don't sleep much either) and i find myself days sometimes weeks before potus does something, i'll say this is exactly what he should do. Like when the virus started i was wondering why the hell he was taking so long to ban the flights (now i know he was fighting a whole room of 20 people against him weighing the options out) But unless i'm mistaken don't patriots now control the oil supplies? don't get me wrong middle east might be a lot tougher to hold since china can reach it from their mainland and we are further, but we took iran for a reason right? and Q said Israel is next. Where does Israeli get gobs of money? from one of the strongest economies who play ball with them the globalist Chinese who use shitty business practices to rip us off. Cut off more of israel's money flow and they will be hurting even more allowing us to constrict them much like the anaconda plan. Like I said newfag but i'm very curious on this. first real post lmk where i fucked up as i'm sure i did in places, only learn through diving in though. at least i'm not doxxing myself by filling in the name line haha 5:5 frens i love and respect the work you've all been doing risking your lives many of you, i'm trying to catch up on everything i can learn

>>13445

For wearethene.ws, anywhere from about 5 to 15 minutes. As long as the bread is correctly formatted (standard title format, bread number in title is correct, bread numbers in notables are correct, etc).

If something went wrong then those notables might appear when the next valid bread is baked, or when we check the site and clean up any errors (at least once every few days).

>>13451

>>13452

>>13462

You're posting in the wrong place. See instructions here: >>11704

>>13464

Thanks for guiding me where to post forgive us newfags as we know not what we do

>>13464

TY o7

baker here

several questions

1. What notables are scraped for the site, those in dough or those in bread

2. What is necessary for scaper to pick up

#111111 at the top and bottom?

3. How can anons have their own buns scraped for we are the news rather than one large notable bun?

>>13905

Hi bakerfren

1.

notables are scraped from the dough at the top of each new bread

2.

correct bread title format

correct bread numbers in title and in dough. the bread number at the top of each bun in the dough matters, the one at the bottom is mostly ignored.

there are a few other things but the bread numbers and title are the most important.

(the scraper tries to correct for some common errors, but this isn't always possible, so we'll come back eventually and clean things up manually)

3.

not really possible at the moment. the wearethene.ws site only works because bakers have an established way of deciding what isn't notable and what isn't.

we could probably come up with a way for bakers to tell the site to expand certain buns, but it would be messy (lots of duplicates, and the baking process shouldn't really have to change just for wearethene.ws, it worked just fine before the site existed)

you can already expand buns on the site by clicking on their individual posts, so another way to handle this would be to improve those features, like letting you see images/videos in posts after you expand them

hope that helps. any other questions, we'll be here

>>13960

thanks for answering

appreciate it

anons are always biting at the chomp to get their stuff notabled, wasn't sure

I think the numbers at the bottom of notables then is to keep them separate, now I know

need a refresher on buns, they don't show all the posts on we are the news, is that correct, in this way anons could have their own buns without it being considered "notable" by all, won't go out onto the site

correct?

>>14087

>need a refresher on buns, they don't show all the posts on we are the news, is that correct

yes

>in this way anons could have their own buns without it being considered "notable" by all

not sure I understood this part? anons can post whatever they want as part of a thread, but it won't show up on wearethene.ws unless a baker puts it in notables

I think making a bun notable is not usually a great idea anyway, it works fine for the "anon bun" and "other news roundup" notables but not really for other stuff

as an example, there was a notable in #11798 that just linked to >>>/qresearch/9218593

the notables line looked like this:

> >>9218593 Do our health experts admit to colluding with WHO and China?

even for anons reading the notables on the board, it would have been much better to just list those 6 posts directly in the notable, like this:

> >>9218168, >>9218194, >>9218222, >>9218239, >>9218260, >>9218273 Do our health experts admit to colluding with WHO and China?

so we changed it on the site: https://wearethene.ws/notable/105652

Hello?

Is the maintainer of wearethene.ws on here?

I have a little question from the technical side.

I'm currently implementing that generals are automatically archived using archive.org.

The idea also was to archive full size pics + videos when a baker UID touched the posts.

Baker is identified by the very first post of a bread. I will also go backwards from 1 general to the previous to figure out baker changes and mark posts accordingly.

And now to the technical question:

how are you currently identifying general breads?

I guess it must be some kind of heuristic, like some text or a regular expression matching.

Can you please tell me what you do?

I would like to stay compatible with everything else.

If you also got some tricks working through the JSON data, please tell me as well.

>>14193

Hi,

>how are you currently identifying general breads?

the first part is a simple regex against the thread title:

/Q\s*Research\s*General\s*#\s*[\d,]+/i

this can and will fail, the most common reason is if bakers make a mistake in the bread number or the title

so once every few days we look at what the scraper has caught and clean everything up manually

the notables parsing is also somewhat self-correcting since notables for bread #1000 usually stay around in the dough until #1003 or so. that gives bakers a chance to fix mistakes in the next bread and then have the notables from previous breads picked up.

>Baker is identified by the very first post of a bread. I will also go backwards from 1 general to the previous to figure out baker changes and mark posts accordingly.

I wouldn't recommend this. there is a loose process for baker handoffs but it's for humans, not computers, so it's not precise enough to be parsed. there's also no way to link users together across different threads since IDs change with every thread. there is all kinds of other stuff that can happen like 3 different bakers/notable collectors working on the same long bread during nightshift.

>The idea also was to archive full size pics + videos when a baker UID touched the posts. Baker is identified by the very first post of a bread.

probably the best you will be able to do using this approach is to make it work for about 60-75% of breads, again due to handoffs.

the approach wearethene.ws takes instead is to validate each new bread and then archive images from the posts that were quoted in the notables section. this probably works for about 95% of breads.

our site doesn't archive videos, PDFs, or animated gifs due to space requirements so that would definitely be valuable.

there aren't really any tricks or standards in general, just "whatever works". wearethene.ws only scrapes the HTML, I seem to remember some issues with the JSON threads list especially being out of sync with the rest of the board, but I don't have any definite info on that.

if there are any API endpoints wearethene.ws can provide that would make your projects easier let us know and we'll see if it's feasible. you're also welcome to scrape the HTML of course.

>>14225

Just added:

https://wearethene.ws/rss

Let's get a bit of testing on that here, and then I'll add a link to the site.

If you have any issues then please post the name and platform of your feed reader and a screenshot. (No personally identifying info)

>>14296

anon aint saying you got skills, but this is exactly what i needed and when i needed

ur feed is running on mi site <3

http://qbrief.com/

>ssl is next on the list

-Frederick Qune

#PatriotsUnited o7

>>14307

>>14308

nice, site looks good

a couple of quick suggestions:

- add some space or maybe a horizontal line in between different news stories to make it easier to tell when one ends and the next one begins

- at the bottom maybe change "Follow Us" to "Follow Patriots" or something like that, "Follow Us" makes it sound like those are the people behind the site

anyway thanks for keeping the source for your feed articles in there, appreciate that

I just added qbrief.com to https://wearethene.ws/about

>>9329638

>>7366

Q:

how long does a notable stay in the news loop on the site?

>>15181

what do you mean by "news loop"?

Recent article - https://wearethene.ws/notable/109472

Completely read your source WRONG.

"Dr. Michael Baden said it appeared that Jeffrey Epstein’s neck injuries were more consistent with homicide than suicide."

Baden said Epstein's death was a homicide, not suicide that was the other medical examiner.

https://www.shtfplan.com/headline-news/is-it-one-gigantic-psy-op-george-floyds-coroner-was-also-jeffrey-epsteins_06012020

>>15294

Thanks anon, updated (see the bottom section of https://wearethene.ws/notable/109472)

As it says on the About page of the site, wearethene.ws is not trying to be an objective source of truth. Think of it as a different perspective on current events, and you still have to think for yourself.

Still, anyone is welcome to point out any issues with articles here, and we will take a look.

>>15261

how many times or how long are those notables left on the site before taken down?

>>15334

Nothing is taken down, the site is intended to be a permanent archive.

She is talking to us!!

Spread this far and wide!

https://twitter.com/drk2lt/status/1270025774335025157?s=21

c.-s.-s.-c.-p.-s.-g.

Can someone confirm?

RJG. Real? Fake?

Syntax is valid?

is there a way to search all the notables? I sometimes want to show a fren something and have to search manually page by page.

thanks for the site fren

>>15960

Coming Soon (tm)

@wearethene.ws site ops. Can you please consider adding comments on the notable pages to open up another area of discussion? It's often that I see a notable that is inaccurate or that I want to add detail to but the 8kun directory for that subject is already full and correcting it would get lost…it'd be much easier and more cataloged if users could post comments on wearethene.ws. It could also potential open more minds and faster. I know that has a bunch of other implications though, so just a thought. Thank you I check the website almost daily.

>>16462

Hi anon, we've been thinking about this some, but as you say, there is a lot to consider.

the easiest way for us to do this securely is to use an external provider to store the comments data.

https://utteranc.es/ is one example. would you make a github account to use this?

github is owned by microsoft and I don't know their stance on censorship, so we would take a backup of all data regularly.

also this should go without saying but as far as I'm concerned you should be using a good, privacy-focused vpn for all of these discussions including 8kun.

open to other suggestions for comment providers, but disqus in particular is pretty bad.

Hello,

Just a heads up, I just finished the code for basic auto archiving of the last ~40% of the breads in the catalog.

They will now all get archived automatically on wayback machine.

I'm not sure if there is a simple way to go to the latest archive of a given URL, but at the very least:

http://web.archive.org/web/*/https://8kun.top/qresearch/res/9755188.html

should work.

I will add archiving of videos soon. That way videos will stay available unless deleted by wayback machine.

I will try to identify notables and then archive videos that were part of notables. Well either that, or I will try to archive all videos in a bread, I'm not sure about that right now.

Please always leave a link to 8kun, because sadly wayback machine doesn't archive every single image. Must be some bug, or maybe it's 8kun's fault.

o7

>>18794

I just figured it out.

Latest archived version can be selected by going:

http://web.archive.org/web/https://8kun.top/qresearch/res/9755188.html

Also see

https://archive.org/web/web-advancedsearch.php

Advanced URL locator hints and tips

>>18794

If anyone is able to contact site owner / CodeMonkey.

Please tell them to look into that archiving issue.

I really wonder what's going on.

Maybe it's even DDoS protection.

Ask them if there is anything they can do about it.

Same issue on archive.today.

It really seems too many requests are made at the same time, and at some point everything gets blocked.

Maybe the amount of requests for these bots can be throttled by robots.txt or so.

>>18796

https://www.ctrl.blog/entry/arcane-robotstxt-directives.html

Crawl-delay

Sets a delay between each new request to the website. For example, Crawl-delay: 12 tells a crawler to wait 12 seconds between each request; limiting it to no more than five-page requests per minute.

This directive is recognized by Bing, Yandex (45 % market share in Russia, and 20 % in Ukraine), Naver (40 % market share in South Korea), and Mail.Ru (5 % market share in Russia).

Due to the distributed nature of search crawlers, you may see more requests than expected as its unclear whether the limit applies to the entire pool of crawlers or applies to each individual crawler. Bing specifies that the limit is applied to their entire crawler pool, but none of the other search engines provide any documentation on this.

Request-rate

A variation of Crawl-delay that sets the request rate rather than a delay between each request. For example, Request-rate: 5/1m isn’t equivalent to Crawl-delay: 12 as all five requests could be filled in the first few seconds within one minute. (Request-rate: 1/12s would be equivalent.)

>>18796

I'm now also archiving the .JSON files as well.

Archiving is also done for 80% of the catalog now. That still means that most of the time threads take 2 days until they are getting archived, so hopefully potentially illegal content got removed until then.

Sometimes I'm getting internal server errors (500) from wayback machine. I will simply try to archive them again some hours later.

>>18843

Today, I also added archiving of HTML + JSON for archive.today.

Next step: archiving MP4s.

I looked through a few threads, and not that many MP4s are getting posted. I think I will implement archiving for all MP4s.

>>18906

Hi AAA, this is good work.

Have you looked at sending archive requests to archive.today also? It is probably more reliable with the DDoS protection issues.

I think the only way to get a reliable, full archive is by 100% custom code that parses the threads looking for new posts, but that is a good bit more work.

>>19470

I'm archiving using archive.today as well for a few days now.

It's archiving JSON + threads on there as well as on the regular wayback machine.

Today I figured that creating another archive after an archive already got created doesn't properly work right now for archive.today, but I figured out the changes that are needed to fix this already.

It seems archive.today only does this when a correct referer is given, the user agent needs to be a web browser + there has to be a submitId.

I'm already looking for new posts, that's possible by using the catalog.json timestamps.

When I detect that, I'm creating another archive.

For the MP4 archiving I will have to look through the breads of course, but that's not that complicated either. I already prepared structures for that job, but I want to fix archive.today first.

When I'm done with fixing archive.today, I will try to contact qresear.ch-anon, it should be possible then to link to both archives for the threads since 7/1/2020.

>>19514

Should be fixed now.

Archive.Today has to have some heuristic because it didn't work for me from the Linux/R3 machine for 10 minutes and suddenly it worked.

@ResignationAnon / @QResear.chAnon

If you read this, I am able to create a text file containing URLs to full size graphics per thread.

I can't archive these separately because it would take way too much time. Wayback machine accepts at best 6 URLs per minute.

But this could be used to download all graphics and then upload them somewhere.

I don't know if its worth the effort, but it would be possible. My internet access is way too slow to do that, especially the upload speed.

@WeAreTheNe.wsAnon

You keep the notables graphics on your server, right?

If there would be a way to access the notable post numbers from your end, I could implement official archiving of the graphics that were included with these notables.

My main priority is archiving MP4s, but this would be possible on top of that. Is RSS looking good? That would be a way to transmit the post numbers.

MP4-Archiving is a go!

9821675

https://web.archive.org/web/20200713164519/https://media.8kun.top/file_store/4897ff54eddf0b1f497b88113d7e17bcc2b37c470b47efa8c65f20e2c885fe54.mp4

https://web.archive.org/web/20200713164556/https://media.8kun.top/file_store/0b225c2b9bb21f3b13c4ffa1fbd7278c8832d274e7b4d800fe0b35b572d215f9.mp4

9822413

https://web.archive.org/web/20200713164644/https://media.8kun.top/file_store/1753508ba5325ab39322071084cc634fff9bfc32a96f84a0eeb0e8dd6c9cb489.mp4

9823235

https://web.archive.org/web/20200713164843/https://media.8kun.top/file_store/a8d834b7a5cf49d686c52c4b8b2e12755edb3d52030b823503308dd745567a5c.mp4

etc.

This will take quite a bit of time, and I'm getting 520/500 errors from wayback machine every now and then.

Atm 1385 MP4s to archive, 24 done.

Thank you!!!! 🙏 Keep up the amazing work!

WeAreTheNews-Anon, can you please tell me if you ignore threads that are missing JSON data?

I'm currently rewriting the archive-code, and I will try to be up-to-date and archive videos as soon as possible, because then you could link to archive.org as well as 8kun.

Right now there are a few threads that are missing JSON data and it isn't fixed. I would also like to put in code that tries to fetch the headers of the JSON file and I will only archive the thread if that data is found.

Do you do the same, or do you even parse the HTML instead? I would like to use the JSON data.

Okay, I hopefully finally fixed an issue with ArchiveToday, enforced archiving after the first archive was done. It's so weird, it seems they have some protections in place that you can only archive a website once, unless you use a web browser to do it.

I'm now archiving at 7:00 UTC, 11:00 UTC, 14:00 UTC, 17:00 UTC, 20:00 UTC, 1:00 UTC as well as 4:00 UTC.

WeAreTheNews-anon, how often do you contact 8kun for fresh data? Once every hour?

I could try to set my jobs close to yours.

On every job run, I'm now getting all JSON data for updated threads, going through these I'm archiving WEBMs + MP4s as fast as possible.

Plus I'm archiving threads when they reach 751 replies or the thread is in the lower 75% of the catalog.

I went through automatic uploading to anonfiles.com, but that's a problem because let's say a thread has 5 mp4s, and each is 15 megs, that's unlikely but whatever. The problem is now that my upload speed is really really limited, 100 megs takes ages and I looked up my ISP, only the very expensive contracts offer a higher upload speed. I guess that's intentionally so that people are unable to share movies and games.

Anyway so in the worst case my whole internet connection could fill up, and there would be nothing to do about it. I think I could at least still upload JSON data, but I'm not sure if that makes sense. JSON data is now archived on 2 archive websites.

>>20419

>Do you do the same, or do you even parse the HTML instead? I would like to use the JSON data.

wearethene.ws just parses the thread HTML. the JSON seemed to be pretty unreliable when we were making the scraper.

>>20563

>how often do you contact 8kun for fresh data? Once every hour?

basically constantly looking at all open threads for fresh posts.

>>20563

>The problem is now that my upload speed is really really limited

yes at some point you need outside hosting and a server for things like this.

>>20683

JSON data is unreliable?

In what regard?

Posts missing?

The only issue I currently see is that after this problem on clearnet happened (downtime) some days ago, several threads simply do not have JSON data, but that's it.

Did you try to contact anyone about these issues?

If so, no reply?

>>20686

JSON data missing posts and getting stuck, like what happens with just about every other page on the site occasionally (catalog, individual threads)

Ron doesn't seem to know how to fix that.

>>20691

If you append a cache busting string to the end of the URL then that issue gets better. That might be enough to make the JSON data pretty reliable, but if it is entirely missing for a thread then there's not much you can do.

>>20844

I think it's pretty weird when the JSON data is fully missing, but the HTML is there.

Doesn't make sense to me.

And last time it wasn't 1 thread, it was around 6 or 7 threads.

Why can't CodeMonkey fix this?

Did anyone contact him over this?

>>20691

That's really weird. I'm checking the catalog that I saved in an SQL table every day or so, and the Generals are all at 751 posts (based on JSON data), so this looks fine.

I have now even implemented code that internally marks Q posts as archived so that they don't get deleted when they fall off the catalog, all attachments of Q posts are also marked for archiving (this archives even pngs or jpegs) + I also mark posts that Q made but without tripcode in a special way and also mark them as archived. Works fine for now.

Still I got another issue that I stumbled over yesterday. Some videos, and just some are not actually archived on wayback machine. There is an URL when you ask wayback machine, but when you go there you are told that it wasn't archived. First I thought media.8kun was blacklisted, but today the latest archived videos work again. Wayback Machine seems to be quite broken.

I will now have to implement code that verifies that an archive actually exists.

>>21039

>Why can't CodeMonkey fix this?

>Did anyone contact him over this?

admin@isitwetyet.com

hope you have better luck than I did

kek

>>21264

>rss

I figured it out. One must copy the link and add it directly to the RSS feed by providing the link in the provided field when manually adding to one's feed.

Automatic discovery of the RSS link is not possible with the RSS feed addon I use in Firefox, FeedBro.

>>21265

>Automatic discovery of the RSS link is not possible with the RSS feed addon I use in Firefox, FeedBro.

thanks. will look into this.

>>21287

just added the RSS discovery tag a few minutes ago. you must have been seeing different behavior based on how you accessed the feed or something.

I am checking to see if an announcement was made since it seems no one is maintaining the wearethene.ws site. It's been 24 hours since it's been updated. Hope everything is okay and all is well?

Hello wearethene.ws Admin

Currently the breads on wearethene.ws site are stuck on #13040 as of aproximatly 24 hours ago actual QR bread is now at #13055

Hopefully you can fix it.

regards a (Dutch) anon

Same here. WeAreTheNe.ws stuck on old posts from 24hrs ago. Thanks Admin.

New notables have not come through for about a day. Thank you to all who put this site together. I like getting yanked out on the q boards but this has saved me a lot of time to get the main info in the threads. Thanks again!

Use the site every day, it's valuable on a great level. Unfortunately I don't have the skills to contribute, but I can show my appreciation at least!

SwedAnon

>>22349

If this situation isn't fixed, I can try something as well. I hope everything is well for wearethene.ws.

I'm already archiving all breads including JSON data including higher resolution pics.

Figuring notables should be doable as well, but it isn't easy at least from inside a bread.

But maybe I could also get notables from the follow-up bread (the first few posts in each new bread).

I can't host my own site though.

Maybe I can also try to contact Regination anon and we can put notables on his site.

Its been 3 days since the last update. I miss my fave news source. What's up?

>>22361

> Various issues arise–bread shiting, bad formatting in topmatter dough post itself, bad fresh bread numbering, bakers using different conventions in their in-bread notables

None of these are really a problem.

I can simply take all the links of the first few posts in a thread that were posted by the same UserID as the first post and use these links to whereever they point and mark these posts as notables. This way numbering doesn't matter. Of course this way we would get all notables from all threads, so there should be a way to distinguish the general, which can be done by a simple text search.

The only problem would be of course someone who creates a fake bread to mark all sorts of posts as notables, but oh well…

I can do all that, but as I said I can't host my own site. I could for example upload data to a FTP server for further processing.

>>22364

What I tested yesterday, one can img src link to media.8kun.top as well as archive.org (when you have the image link) from any HTML file. Nothing like that is blocked.

This means that the website that does all this does not have to host images or videos, these can all be taken from 8kun-server and later from archive.org.

One could take at least the images, that's what wearethene.ws does.

>>22365

What would also great about this:

We would then get all notables from all threads!!!

Right now wearethene.ws was only about the General, not all the country specific breads.

The only problem I see is if notables aren't taken properly. If for example the baker links only one post, but the original poster made 3 posts, then only that one post would show up. Of course that would be a baker issue.

>>22366

Anons still asking in every bread, "Where is wearethene.ws?"

Thx for your diligence AutoArchiveAnon–great idea to get notables from all threads if it can be done. I can't follow most of your logic trail, a bit too technical. But if you succeed in coming up with an additional option to wearethene.ws that would have that feature, would be great. Give us both an alternative and a bu.

qanon.news admin here.

Does this help at all?

https://qanon.news/archives/x/20200808

[20200808] is the date code. [YYYYMMDD]

The system aggregates all the notables from the General threads into a single thread for every day. I can point you towards JSON or you could possibly scrape as a backup from there. I've got 14,806 breads archived, with full breads starting back at #365 ~2-8-2018.

I talked to qresear.ch anon a while ago about adding a link in the search results to my archives to get around the 8kun archives that are gone, but he wouldn't do it.

I'm happy to help any way I can. Should I set up a comms thread for qanon.news?

>>22377

I'm archiving all threads since around a month onto wayback machine. That way graphics are archived and preserved too. raw JSON data is also archived.

I can understand that he doesn't want to 3rd party sites, and wayback machine can also be modified/censored, but the point of it is being a backup. 8kun archives threads too, but all the graphics + videos are gone at that point.

qresear.ch also has a notable section, but this one is just a listing of the notables without any links nor graphics, which makes it quite unusable. It's just the baker notable posts.

I would prefer something like wearethene.ws with all actual notable posts including videos + graphics, plus a search for that, plus links the original threads, plus links to wayback machine archived threads. I think that would be really useful.

>>22379

Plus links to archive.is archives as well.

I archive to both of these sites (JSON data included)

>>22377

What you could do is to check if a thread got archived on wayback machine and then offer a link to there, and also check archive.is too.

The wayback machine is kinda buggy, but I figured that this API here:

https://archive.org/wayback/available?url=https://media.8kun.top/file_store/7690504a4047614878dcd2377a5ac53c9f22772c48f46da051d7049ee7cf0a32.mp4

works fine 99% of the time.

You will get JSON data that points to the archive. You can do that with videos, graphics and of course threads themselves.

I'm typically archiving every few hours. Threads are either archived when they are in the lower 80% of the catalog, or when they are at 751 posts.

I only save graphics that exceed 1777x1777 resolution atm, I was playing with the idea of archiving everything above 1000x1000, but I can't do that at once.

All Q posts attachments are saved, all attachments of posts in reply to Q posts are archived too.

All videos are archived as well (mp4, webm).

And I saw that sometimes some threads were already archived before, but this wasn't done all the time. You can check the timestamp returned by wayback machine as well, so that you know if it's up-to-date. I'm doing exactly that before archiving threads.

>>22379

>I can understand that he doesn't want to 3rd party sites

Meh. It's his site he can do whatever he wants. I just have better archives online that allow the user to see the context in the posts. Linking to my site would only confirm that his results and my archived post is the same. Whatever, if he doesn't want to do it, it's fine by me. I know how to use his search and convert his results to use my archive.

System currently archives full size images from every bread that Q posts in. Don't archive video due to space limitations. Current archive is ~ 115GB. Scrape runs every 10 mins. Breads update till hitting 750.

New dev for me on this site is on autopilot.

Maybe I'll do up a wearethenews alternative.

>>22384

>wearethenews alternative

To clarify, just as a backup to wearethenews. Anon does great work as it is and many use it.

>>22386

Yes, alternatives are always good.

If one goes down, the others may still be there.

wearethene.ws is great, could use a search though.

>>22384

You could link to the archived videos on wayback machine. That way we don't lose the videos.

is there an issue with the WEARETHENEWS site??? hasn't changed in a couple days

Fixing the notables updates on the site today

Also happy to add links to other archives for posts/threads/videos/whatever, just need some info about how they are formatted

>>22396

You need to figure out the archived links.

Most of the time simply going to:

https://web.archive.org/web/https://media.8kun.top/file_store/4c797bc1ec665621f30b59237bcbc5b24fb74949b0345f435fa157ad1518ab17.mp4

should work and will look up the most recent archive, but I had situations where this returned that the URL has not been archived despite it was.

What works for me all the time is this API:

https://archive.org/wayback/available?url=https://media.8kun.top/file_store/7690504a4047614878dcd2377a5ac53c9f22772c48f46da051d7049ee7cf0a32.mp4

This will return JSON data with the most recent archived URL.

Videos will play when calling such URLs.

If you want the raw data URL, you have to look how wayback machine calls the images/videos themselves.

For example most recent archive for the previous URL is:

https://web.archive.org/web/20200715221046/https://media.8kun.top/file_store/4c797bc1ec665621f30b59237bcbc5b24fb74949b0345f435fa157ad1518ab17.mp4

This will load their mp4-player though and add wayback machine information on top.

Actual video is here:

https://web.archive.org/web/20200715221046if_/https://media.8kun.top/file_store/4c797bc1ec665621f30b59237bcbc5b24fb74949b0345f435fa157ad1518ab17.mp4

Archive.IS has a similar way of going to the archived data.

You can also implement the wayback machine URL lookup code in JavaScript, so that the client side checks for the archived data.

Or you could do it on demand, like when a user clicks on a video the first time, you look up the URL. If you get an URL, you save the URL in your local database.

If you do that, you need to support multiple archives for threads + json data / check that threads + json data are really up-to-date (compare time stamp of 8kun thread with the time stamp of the archive on wayback machine, if I remember correctly both are UTC).

Images + Video URLs never change, and 8kun is also not using different URLs for the same video. When the same video is uploaded several times, the same URL will serve the content.

>>22399

To explain further:

You know the URL of each thread.

For example

https://8kun.top/qresearch/res/10201968.html

To figure out the wayback machine archive you call

https://archive.org/wayback/available?url=https://8kun.top/qresearch/res/10182731.html

From there you get

http://web.archive.org/web/20200805010421/https://8kun.top/qresearch/res/10182731.html

I just notice that the last 1-2 days I received an archived URL from wayback machine, but when I try to get the URL right now I don't get any, plus the previously returned archived URL doesn't work either.

It started with this thread:

http://web.archive.org/web/20200805040335/https://8kun.top/qresearch/res/10183484.html

the one before that works fine

http://web.archive.org/web/20200805010421/https://8kun.top/qresearch/res/10182731.html

But the archive.IS link works fine.

http://archive.is/hGdzm

Fuckery afoot.

I know it's been said but I'm realllly missing the updates….

Hope all is well, and up soon

Not to pile on, but I miss the updates as well! Appreciate all you do! o7

any info on wearethene.ws not updating noteables???

Would it already help if I created a General Notable thread and updated it daily with all notables?

So like the sticky is meant to work, but it getting updated all the time?

I think I fixed the archiving issues of the last few days, and I'm currently working on detecting notables and marking them as such.

When I got them marked, I can easily export them.

>>22466

>Duplicate bread number

Assuming this requires manual intervention since it like confuses the script.

Woohoo!! Thank you bakers!

Awesome to see it populate! Thank you all who work on this site. It saves me so much time. God Bless!!!

>>22468

any chance of providing a FORK of your noteable harvesting script/source, this anon would like to create a FRIENDLY alternative 'news team' and I would kindly appreciate the (non-commercial) KICKSTART … wearethene.ws IS AWESOME!

o7

>>22468

On it.

I noticed yesterday that the deserializer that I'm using for the JSON file doesn't decode \uXXXX Unicode characters properly. Need to fix that first, so that the text isn't garbled up. Wouldn't matter that much in Generals, but for country specific threads it's a problem.

I should be able to make it so it identifies and extracts the notables from each bread (always from the follow-up thread), and then creates a nice list for me to copy+paste anywhere.

I'm ignoring links to threads, only taking in links to specific posts, that seemed to work quite nicely and even if a baker fucks up the links in one thread, it would get the links from the follow-up thread.

>>22477

When I'm done with my collection code, I can either put the exact logic I'm using on here, or even put the actual code on here. I'm using SAP / ABAP, so maybe the code won't be that useful, you need a working SAP system up and running to use it. There is a demo system free to download called NPL, but then you have to have some SAP knowledge to use it.

>>22477

What I heard is that wearethene.ws takes the HTML and interprets that. I'm using the JSON data instead. JSON works fine except that sometimes the JSON data is gone. Happened right now again with one thread, but I archived it before the JSON was gone. It's really weird.

>>22396

o7 Thx for your work anon! I've seen may posts over the past few days in the generals wondering what was happening on your site.

qanon.news archive links are nearly the same as 8kun if you want to link there.

https://qanon.news/Archives/x/10249855#10250524

[10249855] == threadId

[10250524] == postId

>>7366

Thank you, anons, for putting together this excellent resource, wearethenews, for others to use.

Your hard work and dedication do not go unnoticed.

Trust me when I say, Heaven is watching.

You'll see…

>>22468

Great Job! O7'

A lot of anons (ww) are using your awsome site.

Thnx for getting it running again.

>>22399

>What works for me all the time is this API:

>You can also implement the wayback machine URL lookup code in JavaScript, so that the client side checks for the archived data.

>Or you could do it on demand, like when a user clicks on a video the first time, you look up the URL. If you get an URL, you save the URL in your local database.

We will look into this, I like the idea of the site trying media.8kun.top first and then other archives if that's not available any more.

>>22477

>any chance of providing a FORK of your noteable harvesting script/source

We will likely be releasing the code at some point soon. Lot to do first, basically need to take a good look through all of it and make sure there is no sensitive info that shouldn't be there.

>>22498

>When I'm done with my collection code, I can either put the exact logic I'm using on here, or even put the actual code on here.

Excellent, the more the better. Notables and in general better sources of news are too important for us to be relying on a single site or service.

>>22505

Likewise anon. It looks like these archives are only for /qresearch/, is that correct? wearethene.ws also has bread archives but they are not linked from anywhere at the moment: https://wearethene.ws/qresearch/res/10249855.html#10250524

>>22507

>Heaven is watching.

Heaven is watching all anons, bakers and others who help to fight for a better world. None of us can do this alone.

Thanks everyone for the support, we really appreciate it.

Thank you so much - your website is a huge service to help spread the word - it really helps people that would otherwise have a hard time following the 8kun boards.

>>22530

for parsing JSON it's best to use a library dedicated to that task because there are other escape characters too, and "\uXXXX" "\\uXXXX" "\\\uXXXX" etc all need to be handled differently which has to be done using a parser. no idea if SAP has a full JSON library, all our stuff is based on linux.

there are two reasons wearethene.ws doesn't store videos and PDFs. first one is disk space, and second is safety, those are more complicated file formats that can contain all kinds of nasty stuff (especially PDF).

and I don't trust wayback machine either, they definitely delete stuff. I do have a local archive of most media posted on the boards that could be uploaded somewhere but that would take up a lot of space.

>>22532

I'm using an existing library for the parsing, but the library doesn't have \uXXXX support. They fixed that later, but I can't access that patch using my NPL-System and at work we don't have a 7.5x+ system yet, and that patch is for 7.5x+ systems only. The NPL system is 7.52, but it counts as a demo, and it even has some weird crap in the SQL database, so that you can't grow it forever. There is a trick though and you can remove that restriction (which I did).

The good thing about SAP is that almost everything is basically open source, so you can check what they are doing (and in normal systems even modify it when there is no other way).

The NPL system is also running under Linux (SAP system is available for Windows as well as other OS).

My code seems to be working properly, I even checked what 8kun is doing when JSON related characters are in a post. Those are not replaced with \uXXXX, but with for example \", \\ etc. If these characters came in using \uXXXX, I would still replace them with escaped characters, so that the library doesn't have a problem with it.

If my bandwidth was big enough, I could download all videos and upload them to somewhere else. An idea would also be to upload it to anonfiles.com. As far as I can remember they don't delete files unless it really violates the law.

Agreed about PDFs. I really don't understand why 8kun accepts PDFs, but at the same time plain text files are rejected. Plain text is not dangerous at all.

PDFs are archived on wayback machine as well by myself. Sadly archive.IS also doesn't support PDF archiving, otherwise I would do that too.

All threads are archived there though, so if you offer links to the original threads, you can offer links to the archived threads on wayback machine as well as archive.IS. Typically archive.IS threads are better, I haven't seen it once that pictures were missing. On wayback machine that happens sometimes.

>>22521

>archives are only for /qresearch/

Mainly qr. I get projectdcomms, and some from QRB and COMMS. I looked into getting some stuff from endchan, but the json was totally different and was going to cause a lot more work so I bailed on it. Do you get more than just QR? Need to know so I can work out a link to your archives.

>>22555

wearethene.ws archives /qresearch/ and /qrb/. links are the same as 8kun just replace the domain.

examples:

https://wearethene.ws/qresearch/res/10286057.html

https://wearethene.ws/qrb/res/42619.html

Hi, could you guys forward me the link to the official site and page of the Anthony Huber criminal record sheet? I can't seem to find it. Thank you. –Eric

>>23539

Hi fren, you are in the wrong thread. see instructions here: >>11704

also don't post your name and email here.

Apparently Wearethene.ws doesn't give a fuck anymore. Is another website in the works or we just going to say fuck it?

>>23957

Problem?

>>23957

Works fine. I don't see a problem.

shitbags trying to make a plan.

https://www.reddit.com/r/conspiracy/comments/im4lln/antifas_plan_to_disguise_themselves_as_trump/?utm_medium=android_app&utm_source=share

You have a shill baker pos. This hurts our goal/message of UNITY.

https://wearethene.ws/notable/138671

Muh jew stuff like this scares the shit out of normies. Please fix.

xyz

>>23546

Mother Earth Angry! Give baaaad people big storms, much fire!! Vote for BIden or Mother Earth will kill you!!!!

I am new here.

Okay, but seriously, if we are 12 steps ahead, who is in control of the weather warfare apparatus? From the ground it looks like the black hats and they continue to assault the US.

It is not hard to stop this, probably as much effort as stopping the Kenosha riots. I'll help. Are anons hip to this?

Can you please change video URLs to stop going to invidio.us, because that instance is down, and you get a selection which includes jewtube.com, which is kinda ridiculous.

You can use

https://invidiou.site/watch?v=XXXX

Hello, not sure if someone is already looking into this.

Notables since last night haven't been feeding to wearethene.ws.

Just wanted to let you know, thanks!

notables arent updating on wearethene.ws

Since RBG died…

Yep RGB died and WeareTheNews stopped updating.

We lost the new guys already?

Why? 😭 what happened?

>>26538

>>26547

>>26602

>>26570

>>26508

Patriots, check out the Notables tab on qresear [ dot ] ch. It will be a bit manual to open the threads there but thankfully it's the workaround we have at the moment.

Hoping the site updates soon too!

Withdrawals are becoming unbarable. Need a fix to get my fix. Your time is appreciated. Everything happens for reasons!

>>26800

Anon, check out >>26672. There's a workaround we have but it's a bit more manual.

>>22456

I think that would be convenient anon.

>>27035

I'm still on it. It got complicated because of JSON decoding, which is required for properly extracting notables, plus there is also a problem right now with archive.today archiving. That site activated Captcha since end of last month, and I'm still looking for a way around it. Strangely it works fine when I use the same code on Windows, but I get the captcha on Linux. Maybe has to do w/ slightly different build of curl.

>>27145

I'm trying to create it in a way that I can simply copy+paste notables into that thread every day.

Test

>>25667

I think I got notable detection down. It's prone to baker errors though, but I guess better than nothing. I just need to save that data now and make it easily accessible for myself for easy pasting.

Maybe wearethene.ws is even sending a message and wanting us to have further options just in case anything goes boom.

>>27156

qresear [ dot ] ch [ fwd slash ] notables

…is the other option for now.

>>27168

Sadly that's without links.

On 8kun there would be links + hover-ability.

Test

Q Research General #13698: Over Your Dead Bodies Can Be Arranged Rez Edition

>>>/qresearch/10702976 The reactions to this Karen’s response to his tweet is comedic gold.

>>>/qresearch/10703009 BIG NEXT WEEK anon decode

>>>/qresearch/10703028 Senate Ukraine Report On Biden, Burisma Expected ‘in days’, Says Top GOP Senator

>>>/qresearch/10703050 Observation: A lot of mainstream, public figures and intellectuals are calling for arson attacks and violent riots. They sound indistinguishable from antifa.

>>>/qresearch/10703095, >>>/qresearch/10703286, >>>/qresearch/10703308 Moar Sedition & Salt

>>>/qresearch/10703187 Anons watching the water? - TGD

>>>/qresearch/10703238 Sen. Amy Klobuchar (D-MN) stated that Supreme Court Justice Ruth Bader Ginsburg should “not be replaced until a new president is installed

>>>/qresearch/10703309 Presidents are elected, NOT INSTALLED

>>>/qresearch/10703345 SPREADSHEET & OFF-SITE GALLERIES ARE UPDATED

>>>/qresearch/10703389 We’re now walking to Mitch McConnell’s house to protest.

>>>/qresearch/10703432 PUNISHED TRUMP. - FLASHBACK FOR AN AMAZING READ

>>>/qresearch/10703469 Statement from the President on the Passing of Supreme Court Associate Justice Ruth Bader Ginsburg

>>>/qresearch/10703479, >>>/qresearch/10703511 PF

>>>/qresearch/10703493 Joe Biden struggles to read prepared remarks after passing of Ruth Bader Ginsburg.

>>>/qresearch/10703525 Low IQ Maxine Waters Howls at the Moon! Orders Senate Democrats in Minority Not to Allow SCOTUS Appointment Before Election

>>>/qresearch/10703619 Notable (just after peaceful protest) Potus Response to first hearing about passing of Justice RBG (vid)

>>>/qresearch/10703624 To Honor Ginsburg, Democrats Have One Choice: Go Nuclear - David Corn

>>>/qresearch/10703672 #13698, #13699, #13700

>>27171

they are even sorted according to postid, me likey

>>27170

10703676:Q Research General #13699: The Color Revolution Just Died With RBG Edition

>>>/qresearch/10703757 #Belarus is mobilizing it’s forces towards #Poland! Watch how #Russian border will be cleared from resistance!

>>>/qresearch/10703767 Megan Kelly Bleeding on twitter

>>>/qresearch/10703812 If reports of her "last wish" are true, then she had a fundamental misunderstanding of her role.

>>>/qresearch/10703827, >>>/qresearch/10703877 10 - 2 J - B

>>>/qresearch/10703913 Rep Omar Just Destroyed by ICE

>>>/qresearch/10703915 Q PROMISED BIG HABBENINGS THIS WEEK AND HE DELIVERED COMFY AS A MUTHA

>>>/qresearch/10703982 The wicked witch of the SC is ded.

>>>/qresearch/10704163 Melbourne Stazi doing their thing

>>>/qresearch/10704316 BEFORE, AFTER

>>>/qresearch/10704352 Warroom/Endchan displaying what projection is

>>>/qresearch/10704418 #13699

I posted a few inside the notables thread on /qresearch/

If anyone could verify them, that would be nice.

I will continue pasting tomorrow.

I need some sleep.

>>25519

… This is an interesting question actually but why the actual fuck would you post it here? Put this in the Main Bread.

I really found WATN super helpful and a time saver?

It dead?

Sad.

>>27975

I bet it's just broken and they are busy with something else.

It will work again after a while. I hope they will also link the official notable thread on there, so that people can go there if it's stuck once again.

My own parser works pretty good though, so maybe I will tell them how I parse the notables. Had no problems yet, and my parser is even error-prone. It will get notables retroactively as well.

What happened to wearethene.ws? I am lost now. That is how I get all the news and post it on my Twitter.

>>27986

Don't enter your email, you are doxxing yourself.

Until wearethene.ws is updating again, just go to the official notables thread on /qresearch/

I'm updating that one at least once every day.

>>>/qresearch/7003045

You can mouse-over any link and see the post, or you can click on it to go to that thread.

Really missing the https://wearethene.ws

I rely on the RSS feed to stay up to date.

Hope the maintainer(s) are able to get the site updating again, it's an invaluable resource and your work is much appreciated

What do you need to get the site working again?

Are you under attack?

Are you threatened?

Your site wearethene.ws is MOST VALUABLE, we need it back on line Anon!

Just tell us what help you need.

>>28134

great news newsb

what a pain in the ass huh

so, as long as the structure of the dough doesn't change again, as in headers, we can change any smaller additions, as in things under the headers

do they need to have *** or something bakers should know about so we can do it or not, whatever works best kinda thing

still wearin the hell out of qbin and luvin it

tyvm

>>28134

Glad everything works again.

I'm now even archiving all notable files, even when it's small pictures. My archiver is trying to catch up, but has tons of things to do.

btw. it seems qresear.ch anon got doxxed, and only after a day or so the thread was removed.

I'm also not sure about qmap.pub. That site had misleading titles for Qposts all over. That always looked fishy to me.

Can't you parse threads in a way that threads that are marked for manual review are skipped at first, so that at least other breads are shown?

Thank you sir for sponsoring the site, and refreshing it. Know you are sincerely appreciated Anon of wearethene.ws!!

>>28134

Thank you for updating the news. Wishing you a lovely Sunday! :)

>>28152

glad you are enjoying doc.

bakers should be able to change the headers in the dough as normal. I relaxed those checks a bit and I will also be looking in on that more often to make sure it works right.

>>28159

>Can't you parse threads in a way that threads that are marked for manual review are skipped at first, so that at least other breads are shown?

I'm sure that can be improved further, but one problem with this is it's very hard for automated code to tell the difference between these two situations:

- #15001 is valid, #15002 through #15010 look different, #15111 is valid again

- #15001 is valid, baker makes a mistake and bread number skips to #15011

I think the best solution for now is to add some more monitoring for what is happening with new breads.

>>28193

Is it such a problem if there is a duplicate thread, or a thread is missing?

My current code is ignoring bread numbers. It simply marks posts independently as notables (but remembers which thread marked them as such) and when I get a bun later via GUI, the code is actually selecting the marked posts instead of getting the notable list.

I mean let's assume a shill creates a fake bread, and we capture notables from there, this may still happen even when you try to verify everything, because there is no way to verify that a real baker created the bread (because tripcodes are disabled for everyone except Q and maybe BO).

And I mean just in the last few days we had a few "Q Research #"-breads, instead of calling them "Q Research General #" and that was the fault of real bakers. I think it's more likely that an actual baker simply made an error.

and I assume that fake breads are removed within a few minutes or hours.

>>28243

a way to get around such fake breads would be to delay parsing threads until the notable list is at least let's say 4 hours old. That way if something invalid goes through, it's the fault of BO / BVs.

Over the last months I have never really seen that happening (of course maybe I just missed it).

>>28244

another method would be using a small delay like 30-60 minutes, at the same time checking for XX hours that this thread stays up, and if it's deleted removing those notables.

That way it would run automatically and through the actions of BO/BVs it could even fix itself.

>>28243

>Is it such a problem if there is a duplicate thread, or a thread is missing?

Yes, one of the goals of wearethene.ws is for the record of notables and bread numbers to be as good as possible, so that wearethene.ws serves as a high-quality data source for people researching these events in the future. Different aggregators having different goals is also fine.

The problem with waiting for BO/BV actions is the delay it would introduce, first of all, and second that BO/BVs usually don't fix all the issues with bread numbering.

>>28244

>Over the last months I have never really seen that happening (of course maybe I just missed it).

Maybe, but what does happen a lot is stuff like bakers accidentally putting the wrong bread number in the thread title or the notable buns. When the bread number is baked as #12800 but the correct bread number should be really #13800, notables from a thousand breads ago could easily get overwritten.

This used to happen earlier in the site's evolution. It was a huge pain to clean up when it happens accidentally, and it would also be easily exploitable by bad actors to hide specific information.

This is why the parsing for wearethene.ws is pretty strict now. Usually this works out fine, since part of the baking process is fixing those mistakes when they're noticed, but in any case the parsing will continue to evolve over time.

>>28246

I would (and I do) use the thread number, not the bread # for saving notables.

I'm using the bread title, but only for viewing.

btw. just this bread right now has a wrong # once again.

>>28267

>The problem with waiting for BO/BV actions is the delay it would introduce

As I said, you wouldn't have to.

You could simply recheck if the bread you took it from gets deleted or not and use it straight away. That would be a good idea regardless, because who knows what could happen.

Oh and I actually got a reply from isitwetyet, weeks after I sent the email. They replied they would take care of the JSON data issues, and for the last 1-2 weeks I haven't seen anything like that happening again.

>>25535

>Can you please change video URLs to stop going to invidio.us, because that instance is down, and you get a selection which includes jewtube.com, which is kinda ridiculous.

if the instance list stays up at invidio.us, this is better, because then we aren't sending all our traffic to one instance.

and I just see yewtu.be which is fine. anyway if you don't wanna see the words jew, joo, muhjoo, etc, then chans are not the right place for joo, kek.

>>28267

>use the thread number, not the bread # for saving notables.

wearethene.ws uses both. bread numbers are for humans, thread/post IDs are for computers. it's more user-friendly to have sequential bread numbers but it does make parsing harder.

>recheck if the bread you took it from gets deleted or not and use it straight away

most duplicate breads never get deleted. they usually get filled or locked.

>>28269

>isitwetyet … replied they would take care of the JSON data issues

excellent, you're a lucky anon ;)

>>28301

>Recently discovered wearethene.ws, an inspired piece of work; dark mode awesome! o7

glad you are enjoying anon, o7

>Is there a mode so links open in a new tab?

I guess this would be for links to other sites? at the moment this isn't an option but we can probably add it. for now just remember to right-click or long-press on links.

>>22349

>codefag reporting for duty if needed.

you or other codeanons still here? leave a protonmail address in a comment here, and we'll get in touch.

>>25518

>looks like wearethenews last grabbed qrb #59.

thanks for the reminder fren, I'll get the qrb notables up to date now.

Coming here to say the same. Wearethene.ws says its account has been suspended.

>>28838

about page goes to "account suspended" for me, maybe your browser had it cached?

>>28838

Says suspended for me as well.

I'm not really surprised by this.

qanonbin.com got suspended as well.

Same provider?

I hope everything is fine and you can take another provider, which will probably do the same shit. Maybe try a Russian provider for a change? kek

>>28764

>and I just see yewtu.be which is fine

I get offered yewtu.be

Stuff like that isn't normie friendly and I see that as a problem.

>it's more user-friendly to have sequential bread numbers

Why do you even mention bread number?

Shouldn't that be irrelevant to most people anyway?

>excellent, you're a lucky anon ;)

I actually didn't expect to receive any reply, and it took weeks for it to show up. Even today no problems, despite the Biden vs POTUS debate.

It seems they really fixed it for good.

>>28764

>you or other codeanons still here?

I'm a codeanon as well, but I'm not really into HTML and such, but database, C, C++, ABAP and others. I got ages of experience though.

>>28844

Checked.

I wonder if it would make sense to create another board on this place for notables, and use a format similar to wearethene.ws?

Of course posting will create CAPTCHAs, so at least some manual work will be required. At least that should be safe.

I have been updating the notables-thread on /qresearch/ for the last few days and fitting that into my regular schedule. Works out quite well for now and I doubt I will go on holiday anytime soon kek

I wonder what the official reason is:

Spreading misinformation? (then CNN should have been banned a long time ago)

Being part of the Quuuu conspiracy theory? (so what's the actual fault?)

Illegal (uncontrolled) news aggregator?

There is no way that this is legal.

Jumping straight to FUCKERY any time something happens is not good research methodology. Credits ran out with the hosting provider for wearethene.ws and qanonbin.com, paid and sites are back up now. They are charging a bit more than necessary but have been perfectly trustworthy other than that.

>>28852

Thank you newsbaker! o7!

>>28764

>thanks for the reminder fren, I'll get the qrb notables up to date now.

Thanks newsbaker, just baked #62

Today in Catania was held the trial in which Matteo Salvini, former Minister of the Interior, was accused of defending the Italian borders from the assault of NGOs in July 2019.

The Audience is postponed to next November 20: the Gup, after two hours in the council chamber, has decided to hear also Giuseppe Conte and three other ministers. For Salvini a first important victory

In particular, the Gup of the Etnean Court decided to hear in court also the President of the Council Giuseppe Conte, as well as the current Minister of Foreign Affairs Luigi Di Maio and the then Minister of Infrastructure Danilo Toninelli. In the courtroom will also be called the current owner of the Viminale, Luciana Lamorgese.

Italian Patriots are with Matteo Salvini

#ItaliansFirst

#OrgoglioItaliano

Make Italy Great Again

Wearethene.ws-anon, I'm surprised that there are no problems on your en

d.

The kitchen is a mess right now and it's chaos.

Baker fights. Duplicate bread numbers. Panic.

Possibly even multiple (and different) notables posts.

Just a heads up on that.

God bless.

I conducted an investigation on Rafael Nieto, a candidate for the presidency of Colombia 2022, he has worked with Mclarty for 17 years.

My research found the links between Mcalarty and Soros, his brother, the US treaty with Iran, relations with Cuba, the Nicaraguan canal.

I have a lot of information

The article was published on the Frater America portal, here the link.

In Colombia, those who say they are correct are some Nazi communists, starting with former President Alvaro Uribe, a drug trafficker at the service of the New World Order.

https://www.frateramerica.com/l/rafael-nieto-loaiza-otro-caballo-de-troya-de-paso-internacional-al-ritmo-del-nuevo-orden-mundial2/

>>30454

>Possibly even multiple (and different) notables posts.

There are a couple of threads to fill in but not too bad. Anyway this is why notables parsing is "strict". The site will basically just pick one of the posts that's formatted correctly, but I will look at the notables thread later and see what should be combined. I don't have time to be doing that constantly though.

>>30579

I've read a bit about Uribe, seems like he is a piece of shit who needs to hang. Good luck fren. https://en.wikipedia.org/wiki/%22False_positives%22_scandal

Next time you should post in the main thread, see >>11704 for instructions on how to get there. Also don't post your name or email on this site.

>>31822

>Decide on some checks and balances bullshit and get everything squared away please.

Get on the general board and help "square it away" anon. I don't have time to properly deal with the massive amounts of faggotry on the board this week.

Please post what you can

The whole baker war is stupid control

Makes me think that operators of wearethene.ws are part of a side of that battle versus doing what’s best for all anons and normies

Best effort and some notables are better than no notables

Important moment in time. We need not normie accessible links to quickly pull from.

>>32620

It seems wearethene.ws is comped as much as the bakers on the main board. Notables have looked mostly fine for days, yet no updates. A shame, as it was a great resource.

A shame. Was a good source of info

>>32620

>Makes me think that operators of wearethene.ws are part of a side of that battle versus doing what’s best for all anons and normies

No, we just don't have unlimited amounts of time to deal with this childish bullshit, and fixing it properly requires going through the notables for every single bread over the past week or so.

In the past, anons and bakers have mostly been able to curate notables pretty well, but since this started that hasn't been true.

>>32649

>It seems wearethene.ws is comped as much as the bakers on the main board.

See above, and then kindly fuck off.

>>32909

Please ignore the idiots, wearethene.ws is still greatly valued.

Looking forward to the return of the notables in my feeds when you get it working again

Want to send a HUGE thanks to AutoArchiveAnon for compiling the notables over in the /qresearch/ notables thread. That is going to make it much easier to get wearethene.ws back up to date under the current situation. Now that is what you call using the whole buffalo!

Is wearethene.ws in the process of updating? I know many that are shy of the boards and rely on it. Thanks for all of your work on this valuable resource!

Wearethene.ws has never been more important than now, just weeks before the election. Please resolve whatever is going on, and get it up to to-date. Your efforts are heroic and very appreciated by this qanon.

>>33483

>Please resolve whatever is going on,

How are we supposed to resolve massive and unceasing faggotry on the main board exactly?

Anons needed, get in there and get the bakers to stop shilling against each other.

Please put wearethene back online. Anons aren't fighting anymore.

>>33500

Stay (essentially) shut down.

You're too pussy to be a part of this movement.

Let the stale notes currently displayed stand as a reminder of what faggotry looks like.

I have no idea what goes on over at the boards, but wearethene.ws is where I come to for what the qanons are searching, please Site operator can you please at least get wearethene.ws back online until the election.

this is the most important time in this sites life.

This is not a good time for all this. And we know nothing about the board or the fuckin shills. All we want is to put your info out across the Internet. Isn’t that why you are here? Q? So tell someone to get wearethene.ws back online!

It should be obvious by now that the dismantling of wearethenews was part of the agenda in the bakers' feud. Somebody wanted wearethenews taken down and it worked. Wearethenews and its feed have become so widespread, popular and most importantly, valuable, that the dark forces were compelled to take it down. The feud facilitated it. Hell, it might be the reason for the feud in the first place-to take down wearethenews.

To all the anons who make wearethenews possible, your work is very greatly appreciated and it should be known that the importance of it goes beyond what most people might think. The notables aren't just an interesting compilation of intel sourced from the internet, the notables are a critically valuable resource for awakened humanity and they have a wide readership. The dark forces know this. That's why wearethenews has been attacked more than once. Obvious? To some, yeah, but to others, maybe not.

In war, enemies aren't going to attack targets of lesser importance. They'll go after critical, important infrastructure. Like we're seeing.

Please start posting again… I think the Patriots fight board is settled

We need wearethene.ws back up! Please update!

>>33568

>>33734

>>33744

>>33887

>>33901

Working on it. Thank you all for your support.

>>33734

>I have no idea what goes on over at the boards

"What goes on over at the boards" can be thought of as a training exercise for real-life government.

If you don't like what you are seeing, you need to get involved and help fix it.

>>33842

I concur with your analysis anon. "Attacks will only intensify."

>>33700

Go fuck yourself.

>>34264

Once again, thank you, anon, for all that you're doing, for the many hours spent helping to get valuable intel to the awakened community.

Trust me when I tell you, Heaven is watching and recognizes your efforts. It has not gone unnoticed. Soon, you will understand.

Vaya con dios, amigo.

>>34295

Gracias amigo. I will continue doing what I can.

Can we get wearethene.ws back up?

Update please?

hello wearethene.ws folks

i'ma bakin' over on /midnightrider/ board right now.

bun is getting kind of long - one of the 'perils' of having no shills.

we're already at about one bread per day - which is fast enough to be pretty current.

and wil get faster if /qr/ continues to uh….deteriorate. (hope not, but there it is….)

Re splitting the bun:

I've done it on qrb before but had no way to check to see whether that approach works for aggregators like you's.

Let's say bred number is 3549 and we wanna split it up.

Assume we're collecting going from BOTTOM to TOP (like Q drops).

Split bun would look like this:

#3549b

>>xxxx Notable 2

>>xxxx Notable 1

>>xxxx #3549b

#3549a

>>xxxx Notable 2

>>xxxx Notable 1

>>xxxx #3549a

Does that work ok for wearethene.ws?

Will ck back in a bit….hope you're around.

o7

https://newz.com/cnn-broadcast-of-presidents-covid-19-address-may-contain-insider-trading-information/

>>40154

As i understand it, sometimes there is no update if the baker leaves out a colon (for example) in the bread title. Don't know about how dup bread numbers affects anything but that probably creates problems as well.

Would be good to know get a LIST of what formatting problems will throw off wearethene.ws

>>>40155

sorry - /midnightriders/ board

typo

Update please?

We repost everything on wearethene.ws, and we need someone to update the page.

Many rely on this site. It's critical at 7 days out from the election. Please update! Thank you!

The site will be up when we have a few hours to deal with this again, it is somewhere between a half day and a full day of work to properly clean up after a faggotry outbreak on the boards. It's a bit like disinfecting everything for covid but online.

If you want wearethene.ws to keep updating immediately 100% of the time then the best thing you can do is get on the boards and make sure BO, BVs, bakers are doing what they are supposed to be doing.

Other, new aggregator sites that do not rely on wearethene.ws are also very welcome. "Diversity is strength" as the globohomos say but in this case it's actually true.

https://newz.com/jeffrey-epstein-is-a-billionaires-troll/

thank you for all that you do. we appreciate this site - its my main source for news these days.

Yes thank you. It's the only legit news source.

It is really sad that we get this close and dropped the 🏈 ball!!!

>>40488

Please, please try to find the time to fix the site. We are so close to the end…

Please try to leave it "all on the field" as General Flynn says.

We need the site up through the election, please.

Wearethene.ws you win the award of the worst business model on the web. You should just put a note at the top of your website that your panties are in a twist because someone made you mad….Fucken pussy

>>40366

>wearethene.ws administrator has the authority to tell the QResearch board how to function and determine what it's BO and BVs are supposed to do.

Either you are an arrogant self-aggrandizing self-important person thinking he knows what is best for the world better than everybody else. Or you are a paid asset trying to control the information flow on and from the QResearch board by weak attempts of extortion using the fact that masses of people have become used to finding that information on a site you control.

Either way. AFAIAC your site has no legitimacy anymoar after you clearly only aggregate when the information suits you.

Past the time to either shit or get off the pot.

I have never seen such a big ball drop from the Q community. We need crowd-sourced news more than ever, and this site is having a pissing match and not updating. Days before an election.

If any anon can link to a better news crowd-sourcing site, we can all just go there. Fuck wearethe.news.

>>40155

>i'ma bakin' over on /midnightrider/ board right now.

Thanks for creating a refuge. I'm curious why you guys chose to create a new board instead of using /qrb/?

Anyway, this will take some work to add the new board and get the split notables parsed correctly. We will do that but getting qresearch stable again is the highest priority (it's catching up now).

>>40157

>Would be good to know get a LIST of what formatting problems will throw off wearethene.ws

This specific information would immediately be exploited by shills. General rules of thumb - just follow the same standards as past bakers, only change the dough calmly with anon consensus, and try not to fuck up the bread numbers or split the breads too much.

---

Thanks to some of the other commenters here for your patience, you know who you are.

---

Then we have some ungrateful assholes who are just coming here to complain about something that is being provided 100% for free.

It's not a pissing match either, I literally just have too much shit to do to be babysitting notables, and the garbage that has been coming through the boards lately is so bad that the site stops, as it is designed to do.

Here's a couple of better options for next time that happens:

- Go help fix the problem at the source. Learn how the main board works and help kick the shitheads out. Q is trying to teach us that decent government is only possible if we get involved directly...... and apparently some of you are slow learners.

- Go over to >>>/qresearch/7003045 and read the notables there.

- Learn to code and make your own aggregator site. We will happily add it to wearethene.ws/about unless it's a total piece of shit, like some of the comments here.

Thanks for the updates on wearethene.ws! Y’all are hero’s in this fight. We appreciate what you do!

>>40891

Hey newsb, tx for responding

Happy election day!

Might have split notables in the future but we are baking @200 instead of 300 now - this will help keep them manageable. Lotsa diggs with little shitposting adds up to fat buns fast.

Do hope you'll pick up mnr board material, it's solid.

>>40891

Thank you, anon, for all the hard work in making wearethenews operate. Truly. From the heart and soul.

Ignore the bastards. Heaven is watching. Your efforts to help others is noticed and appreciated.

>>28134

Hey man, this https://wearethene.ws/notable/163247 is fake and gay, could you please take a look? It's a meme "boogaloo" website. It'd be nice taking it down and issuing an apology.

Cheers.

>>40891

Hello newsbaker, werethenews is stuck again at #14665 since the yesterday.

Can you please get it going (once) again?

It would help anons (ww) a lot cause we just can't rely on the fakes in this exciting moment in time.

>>41244

I added "unconfirmed" to the title, and added a post that quotes from the CISA website, thanks for the tip.

>issuing an apology

CNN is over that way. Give them our regards.

>>41436

Yep, working on the latest updates now.

>>41461

works for me, thanks buddy

>>41461

Thanks for fixing it, you made a lot of anons very happy including me!

Hello, it appears wearethene.ws has not been pulling notables since bread 14736. Can you fix? It would be greatly appreciated. Your site is a huge help for anons. Thanks in advance!

with code monkey resigning as 8kun admin, is anyone taking over? will wearethene.ws be picked up again or is it toast?

I know that the problem with the breads sucks, but this site is a very important tool for spreading real news. Please make the effort to get this scraper going again. Let us know what we can do to help.