Do people still respect robots.txt? Or does everyone ignore it now? Was it ever something that was respected in the first place?

Email
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
* = required field	[▶ Show post options & limits] Confused? See the FAQ.

Comment *

File

Select/drop/paste files here

Password

(Randomized for file and post deletion; you may also set your own.)

* = required field

[▶ Show post options & limits]
Confused? See the FAQ.

Flag
Oekaki	Show oekaki applet (replaces files and can be used instead)
Options	Do not bump (you can also write sage in the email field) Spoiler images (this replaces the thumbnails of your images with question marks)
Allowed file types:jpg, jpeg, gif, png, webm, mp4, pdf Max filesize is 16 MB. Max image dimensions are 15000 x 15000. You may upload 3 per post.

Flag

Oekaki

Show oekaki applet
(replaces files and can be used instead)

Options

Do not bump
(you can also write sage in the email field)

Spoiler images
(this replaces the thumbnails of your images with question marks)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 3 per post.

File (hide): e9f433a3f6c6d82⋯.png (12.09 KB, 480x344, 60:43, robots.png) (h) (u)

[–]

▶Anonymous 12/12/18 (Wed) 20:21:58 No.1007812>>1007813 [Watch Thread][Show All Posts]

Do people still respect robots.txt? Or does everyone ignore it now? Was it ever something that was respected in the first place?

▶Anonymous 12/12/18 (Wed) 20:43:51 No.1007813>>1007814

>>1007812 (OP)

>people

>robots.txt

🤔

▶Anonymous 12/12/18 (Wed) 20:53:39 No.1007814>>1007816

>>1007813

do people create robots that respect the robots.txt if you want to be pedantic.

▶Anonymous 12/12/18 (Wed) 21:07:04 No.1007816>>1007819 >>1007868

>>1007814

Pretty much all crawlers respect them. Google, Internet Archive, you name 'em. In fact, Wayback Machine is such a fucking nigger about it because it respects them ex post facto, i.e. if a site that didn't have robots.txt before and then suddenly uploads it, all previous archives of this site get nuked.

▶Anonymous 12/12/18 (Wed) 21:07:56 No.1007818

>robots.txt

>hacker.map

▶Anonymous 12/12/18 (Wed) 21:08:20 No.1007819>>1007830 >>1007855

>>1007816

so, domain snipers can delete history?

is there a statue of limitations on this ability?

▶Anonymous 12/12/18 (Wed) 21:11:17 No.1007820

Hey guyz, let's archive internet archive! XDDDDDDDDDDD

▶Anonymous 12/12/18 (Wed) 21:49:32 No.1007830

>>1007819

>so, domain snipers can delete history?

Happened quite a few times, just from personal experience.

>is there a statue of limitations on this ability?

No clue, honestly. You probably can contact them and explain the situation but I'm doubtful.

▶Anonymous 12/12/18 (Wed) 23:03:31 No.1007855

>>1007819

It's not deleted, so much as it is censored. The other guy is right about archive.org being niggers about it though, so no limitations are in place. I've encountered the same thing.

▶Anonymous 12/12/18 (Wed) 23:44:10 No.1007868

>>1007816

>Internet Archive

You are a liar

/tech/ - Technology★

General

WebM

Theme

User JS

Do not paste code here unless you absolutely trust the source or have read it yourself!

Favorites

Customize Formatting

Filters