/tech/ - Technology★

>>883963 (OP)

this is why I hate buzzwords

Something like this might be a legitimate use of a blockchain.

>>883963 (OP)

Extremely useful, get to work! I’ll pay you when you’re done if you post your wallet address or a patreon or something. I want this done now. This project is vital to free speech. If you do a good job I’ll seriously send you good money like 1-2Gs here.

>>883966

i'm not into the blockchain meme, that seems like a lot of bloat to simply match hashes between a client and server

>>883967

that's good to know, I'm sure I would take donations because I don't hate money, but the whole thing would be GPL, even the server side. that's the only way to avoid the censorship, even if (and i have to do more research on this) the hashes are not considered copyrighted content, that doesn't mean it couldn't get sued offline. GPL client and server side would make that irrelevant.

Thinking about this further, a browser side archiver would cause nothing but problems due to website's changing their output based on user-agent and other things. The client side would likely be a separate client.

-archiver core

-create the archive and the hashes, CLI front-end

-client side

-GUI for archiver core

-server side

-website that use's archiver core, delete's the archive and only use's the hash afterwards, store's in the database, search functions, etc.

does the EFF answer questions like this regarding the copyright status of a hash?

>>883963 (OP)

>>883972

What about ads? Every person that tries archiving website X will be shown a different set of ads thus creating non matching hashes. Obviously you could block ads as default but then you wouldn't be making a perfect copy of the website. The other archivers make it possible to check who was advertising on a given website at a random time period for example

Following up on what >>883979 said

You would need to apply some heuristics on what to cull in your hash data.

You have things like ads / tracking ids like was suggested before, but you have much more benign things as well.

- Dates

- Customization for specific user agents (not common in modern times)

- Paywalls

- Membershipwalls

- username of a logged in user

One solution would be to take the DOM tree and for all of the nodes below height X with children and compute the hash for them.

>>883979

that's a good point, i didn't think about that. 3rd party domains could be blocked from the archive, most ads come off a different domain, but that might also block out other stuff.

The only way I can think to deal with that is to hash all of the individual files in the archive, and then provide all of the hashes. Page location layout might be approaching copyrighted territory, but I don't see a way around it. The client could then have an option to remove any files that don't match the server's hash (or save another copy), which would leave only a verified page, whatever was deleted would just fail to load.

Another possibility is to hash only the source. Or extract all of the text content and hash only that. Hashing only the source still might cause problems with dynamic pages and ad content, you could remove all the javascript, but then half the internet isn't going to load at all.

Another consideration I saw checking out archive.is, is GeoIP targeting. archive.is sends the user's IP address to the site in a header (didn't know that), to match the user's location instead of the servers. I don't know any way to deal with that except to replicate the functionality on the server, delete the user's IP address afterward and allow for multiple correct hash's on the same website for the same exact time.

I think all of these options might need to be implemented to deal with these problems.

-Text Only

-Block 3rd Party

-No Javascript

-Individual File Hashes

That's becoming a lot of combinations.

>>883984

if there's a paywall or membership wall or any kind of required login the disclaimer could be it's flat out not going to work and not expected to. there's no way a server is going to be able to replicate something behind a login wall. the page has to be publicly accessible in the first place.

Hashing every child in the DOM tree is interesting, but at that point you might as well provide the entire source yourself, your now providing the entire DOM layout, albeit hashed. I think that get's extremely copyright risky. It's a good idea though.

>>883986

I was just being comprehensive when listing stuff like a paywall.

>every child in the DOM tree

That is not what I said. Please reread it. I just noticed I accidentally said height instead of depth.

To reiterate my idea. You take all the nodes with the depth below a threshold and then create a tree that mirrors the DOM tree but only includes the nodes you have hashes for. In this new tree you store the hashes you calculated. In order to cut down on mostly pointless hashes, I suggest to cull nodes which do not have any children.

One problem with this approach is if a website's DOM tree is bloated. In this case you might not be able to get as good of results for this. A solution to this problem is instead of doing this for the DOM tree, you have an algorithm to determine the proper root of the website and then measure depth from that node.

Storing hashes like this allows you to compare with the server to see what sections you have match the server and which don't. Hopefully if the dynamic stuff ends up in a different subtree than the static content, you'll be able to verify specific content as being correct. Often times, when you want to verify an archived page, you could care less what the footer of the website says so if your footer doesn't match; whatever.

>>883994

I don't think picking and choosing which part's the dom tree to hash is going to work. The entire DOM tree is going to have to be hashed item by item, children and parents. The way I see this DOM tree walking going. website's are going to have different depth.

1. Client hashes every element on the DOM tree.

2. Client sends the server it's DOM tree hash list.

3. Server hashes every element on the DOM tree.

4. Server removes conflicts (this will get fun, parents can't be removed because children don't match)

5. Server sends the client back the new filtered DOM tree hash list.

6. Client filter's it's DOM tree to match the server.

7. Client and Server now have identical DOM tree's, download page requirements, images, etc.

8. The server will still have to provide hash's for every file in the archive, because the page requirements may be different, ad images, etc, this type of back and forth could happen again to verify/delete page requirements that don't match.

9. Eventually with this back and forth, the client and server will have a matching hash/hashes, depending on if it's a single file now being hashed with page requirements removed that mismatch, or multiple file hashes (but the source html/DOM tree should always match).

Walking the DOM could turn into a serious mess, and I'm not sure how to go about it without back and forth between the client and server with hashes, But it could be done.

A simpler alternative:

1. Server is given a url by the client (no client in this case it's the browser).

2. Server create's the archive and the browser download's it.

3. Server records the hash and delete's the archive.

4. Archive and hash now obviously match.

5. Archive can still be verified by the server later, but there is no download that can be DMCA'd.

This would operate more like a caching proxy. You don't hear about proxy servers being hit with DMCA's.

>>884002

I'm not sure which is better here. DOM walking seems more takedown resistant, and less centralized, but it really isn't. If your sending the whole DOM tree hashes back and forth to agree on a common tree, you might as well send the whole archive, and as far as centralization, the DOM walking option is still totally centralized.

The only question here is if the one time archive download is more DMCA'able than the hash back and forth between client and server, that and weather or not DOM walking is going to produce a useful archive for heavily dynamic sites.

>>884002

If you wanted to be lazy you would not use a tree, but just a list of all the text nodes and store a hash of each one. You could also add support for images easily.

>>884006

Can you even DMCA a hash. Technically you wouldn't even need to distribute these hashes to others. You could just have the server send back the bad hashes.

>>884002

actually the simpler alternative, while it be the same DMCA risk as DOM walking, has an additional risk. it's going to have the same problem proxy servers have, only more so, someone tries to archive a site with illegal shit on it, it's not just a proxy it's packaging those files and offering a download. i could see that being a major problem.

>>884011

>Can you even DMCA a hash.

i'm not sure, this is the only real source i have on an answer to that

>As long as a fingerprint is only a fingerprint that helps uniquely identify an item, but it conveys no other information about the item, it can be considered unencumbered by copyright. An examples of such a fingerprint would be the MD5 hash of a complete file. We cannot derive any information of the contents of the file from its MD5 hash. But music fingerprints are different.

https://lennartb.home.xs4all.nl/fingerprint.html

which is not exactly reputable.

sending back the bad hashes would work also. I think ideally what would happen is the client-server interaction would end up agreeing on everything, and removing anything that disagrees, if the images are dynamic, they would be removed. this way the server isn't really giving out anything, even the url structure of the content, out to DMCA, its only the hash.

>>884011

>If you wanted to be lazy you would not use a tree, but just a list of all the text nodes and store a hash of each one.

the text only mode could be even lazier, just strip all the html and hash the text. client and server just match methods of stripping html to match hashes.

the problem with that is these javascript heavy site's now insert text and build the entire dom on the fly (ie this cancer https://reactjs.org/)

>>884013

It's not illegal to have image hashes of CP. Many large websites use them to block image uploads of common CP automatically.

>>884019

that supports the dom walking, but what if the site packages the entire archive and offered it for download, even if it was a one time download. I think that would be a major problem, which kills the easymode alternative.

>>883963 (OP)

>nobody trust's you

>store's the hash

>delete's the content

Please tell me you're not a native English speaker.

>>884023

trusting the site is an issue, but there's nothing that can be done about that. people trust archive.is to not fuck with the content. the user already has the content in this case, all the site does is provide a 3rd party to verify that it came from the site at the time. the only thing that can be done as far as trust goes is make all the code GPL, but that's not a lot server side.

why do people think reddit spacing is the norm?

it's fucking unreadable

This is never going to work with javascript.

Every single test page I've ran, just with wget, the html hash has been fucked due to javascript bloat. Unique id's, timestamps, CSRFTokens, half of the internet now creates the DOM dynamically or atleast heavily tweaks it with javascript and the javascript is generated dynamically. It's not just the javascript either, some of these pages are entering unique id's into the html in custom tags, which i'm sure is used later for tracking and javascript.

maybe the only way is text only, atleast for the html. on a side note reddit pulls 140 files for 1 thread.

>>884049

Sounds like a personal problem to me.

>>884019

But having those hashes is proof that the site possessed CP. They're not getting only the hashes from the government like major sites do. Although it doesn't prove intent to get CP and the site would delete the content immediately after hashing (two things that would make courts lenient), it would probably be enough for a CIAnigger to shut down the site.

>>884023

>2018

>being unironically autistic

>>884191

Implementing the hash check client-side would keep it from ever touching the site's servers, although a court could expand the client-side code as being part of the domain of the site owner.

>>884195

That's true. Although OP said they wanted some centralized way to verify the hashes. I guess it's either have a central server prone to takedown or have decentralized verification prone to poisoning. It's poz all the way down.

>>884195

a client side hash check while keeping any form of verification would require the client be checked that it wasn't altered, which sounds like DRM. The client could act as a simple proxy for the server when it grabs the page, but that doesn't sound appealing at all.

Just drop it already, it is simply not working with non-static (shit) sites. Any workaround you can come up with will be too complex.

>>884072

> some of these pages are entering unique id's into the html in custom tags

Maybe I'll just start saving pages to .TXT files with "lynx -dump" (possibly also with -nolist, if links aren't needed).

>>883963 (OP)

content based addressing is an old idea and already sovles this. IETF are making some official goy friendly version of this over the last decades and soon it will replace the web

>>883966

no, blockchain isn't even applicable

>>883979

>>883984

>>884072

>>884248

This is an interesting side discussion, but it doesn't strictly invalidate the utility of OP's clever idea, even if you realize you'd only be able to hash the same archive once.

For instance:

>User A: Wants to archive a page (or something much bigger, like an image, video, or ZIP file) associated with an URL and time.

>Trusted Server: Accesses URL on User A's behalf, archives it, hashes it, gives User A access to archived file and hash for limited time, deletes file, permanently retains hash and metadata.

>User A retains archive, may possess the only extant copy at some point.

>User A gives copy of archive to User B

>User B wants to verify archive actually represents what it says it is, asks Trusted Server

>Trusted Server verifies it as authentic or forgery

This would allow something like archive.is to exist, too, by keeping the hash Db separate from a Db of full archives, which (depending on takedowns) could consist of anything from a normal website like archive.is to a cloud of torrents or a blockchain.

>>884454

I think I am going to write some proof of concept code for the simpler alternative method, none of the code is revolutionary here, probably literally use wget to pull the site archive.

I came up with a few extra options today:

-Instead of providing the archive, provide diff files

----Client hashes all binary files, sends file listing and hashes to server (this is going to require a custom client).

----Client sends full text/html files to server.

----Server checks binary hashes for mismatches, missing files, extra files, etc

----Server run's diff on text/html files, generates diff and patch files

----Server sends back binary hash mismatch data, along with text/html diff and patch files

----Client creates a single tar archive with client's original archive + mismatch file + diff/patch files, hashes it

----Server does the same, hashes should match, it has the same data as the client now.

----Server store's this hash for verification.

Advantages:

-The client can either patch the text/html files to generate a fucked copy or not, the archive should be verified either way.

-The server doesn't provide illegal content to the client in case the client is an asshole, which is going to happen.

-The server isn't providing the website file's directly, seems like should be less DMCA-able.

Disadvantages:

-Even if the hashes aren't, the diff files on the text/html source are probably considered a derivative work, it only provides them during the exchange, there's nothing to DMCA but the hashes, but during the exchange they might be able to claim there's copyright infringement going on.

-Instead of providing hashes, sign the archive with GPG

Advantages:

-Hashes can't be DMCA'd, they don't exist, the verification isn't with the site itself, it's with a 3rd party keyserver.

-If the site goes down, the archives can still be verified.

Disadvantages:

-Server must provide the full archive, signed. If the diff method above is used, it would have to return the full archive back to the client instead of just the diffs/mismatch file.

>>884461

How could copyright violation be said to occur during the window in which a server-side archiver would provide data? That would also be during the time when whatever's being archived is still publicly accessible (otherwise the server obviously couldn't see it). Has, for instance, a proxy server been prosecuted for some reason?

Also, I'm not seeing how a diff would be useful for the primary purpose of verifying that a shady random archive file is an authentic copy of something that a certain URL served at some point in time.

>>883963 (OP)

tangential, but mozilla is going to delet all non-quantum addons soon; maybe we can fix that somehow.

>>884479

>How could copyright violation be said to occur during the window in which a server-side archiver would provide data

I'm not sure, I'm assuming the kike lawyers could come up with practically anything, and think of all the sites that have a ToS 30 pages long. The archive sites and google have lawyers that can smash bs immediately, whoever hosts this system likely will not. This isn't even taking fair use into account, or the lawyer speak google and the archive sites use: "direct agent of a human user"

The goal of this system should be that even if forced to comply with a DMCA, it can still be verified. Complying could mean blocking future access to whatever, hopefully not removing the hash, but gpg would solve that problem. That and minimizing as much as possible, what might be considered copyrighted to begin with, diff's and hash mismatches are less than a full archive of the site coming from the server.

I'm really not confident though that the extra back and forth between client and server to generate these diffs/binary hash mismatches would even be worth it. An archive a server sends out is clearly copyrighted content, but the diff/binary hash could definitely be considered a derivative work, which is the exact same problem if anyone gets pissy. The big benefit here I think is illegal content. It seems like it would be easier to defend the system if a user uploaded the illegal content themselves as opposed to the site reaching out and grabbing it on behalf of the user, but at the same time, it would be doing it anyway to generate the diffs, it just wouldn't be giving it's copy back to the user, it would be sending the user's copy back, maybe this distinction is pointless, I don't know.

>Also, I'm not seeing how a diff would be useful

The diff by itself wouldn't be. The diff + the user's original archive, combined into a single file and then hashed or signed with gpg would. The purpose of the diff + binary mismatch information would be to avoid the site having to provide the archive at all, the users themselves would submit it. The server would still generate it's own copy, but the user would only get back diff's and mismatches, and with gpg signing, a copy back of the archive they sent to the server.

The html/txt diff's and/or patch files combined with the original client's html/txt, would produce the servers version. Binary files, images/videos, would simply not be provided, it would just be indicated that these files could not be verified due to a hash mismatch.

>>884492

ya i think this is beyond using another add-on. it's either going to work with no client side at all, in the case of the server providing the full signed/hashed archive, or require a client that's going to have to be dedicated, it'll require more permissions than webext allows. I don't think webext can modify files. You might be able to do it with native messaging, but at that point you still have to install a binary on the client side's system, with the additional requirement of the webext being certified kosher by mozilla/google. It seems stupid at that point, the client side shouldn't require a browser webext front-end to operate in that case, it should just operate independently.

Again, this is should be legally identical to a proxy, and I'm not aware of proxies ever having been held liable for the actions of their users.

>the diff/binary hash could definitely be considered a derivative work

I don't think so. This isn't even like a magnet URI, where it can be used to find a file copy, its only possible use is to verify a file's provenance.

>maybe this distinction is pointless, I don't know.

Eh, it might be useful for some or other reason, like server bandwidth savings.

>The diff by itself wouldn't be. The diff + the user's original archive, combined into a single file and then hashed or signed with gpg would.

Ah, got it.

>>884510

>the diff/binary hash could definitely be considered a derivative work

i worded that wrong. the diff would not be a hash. the diff would be an actual diff of the text/html, or a patch file, this is what i think would be a derivative work. the binaries would just be hashed, i think there's strong support that that is not considered a derivative work, because no traces of the original remain, but the diff/patch file would have snippets of the original site.

> this is should be legally identical to a proxy

hopefully that's a strong defense right there. i think this is where the archive site's and google's use of the lingo "direct agent of a human user" comes from.

Nothing should be archived

>>890152

why not?

>>890152

fuck you

https://web.archive.org/web/20180328221121/https://8ch.net/tech/res/883963.html#q890152

I get it, you ideafag.

>add on that archives web pages

>user installs said browser add on

>user saves snapshot(s) of websites

>website 404'd

>user2 wants to access 404'd website - and had the same add on installed

>user2 pulls from user1 through the add on

Problems that arise

>MitM from dns, isp, or native malware toolbars

>MitM from cuckflare

content might not be the same as the true original

how do you verify it?

are you imagining a crawler with torrent?

who verifies the content is it native or a certain server (like torrent trackers)

>>890338

this whole thing is reliant on central trusted server.

>website 404'd

>user2 wants to access 404'd website - and had the same add on installed

>user2 pulls from user1 through the add on

i don't think it would be happening with add-ons, this would be either a dedicated client (method 2) or straight website (method 1)

-user1 downloads gpg signed archive (method1) from trusted server or user.

-user1 generates archive themselves, uploads to trusted server/user, receives gpg signed archive back which includes their original archive + diffs from what the server received (method 2)

-user1 shares their gpg signed archive with user2

-user2 verifies authenticity with gpg.

it's up to user1 to share the archive however they want, the trusted server/user only provides it once, the whole point of this is to be resilient to takedowns.

>>883963 (OP)

have you checked https://github.com/webrecorder/webrecorder

>>890420

that's interesting. can pretty much run that and sign it with gpg and there's method1. method 1 isn't that complicated to begin with though, this just does it differently. it's more bloat than I had planned but it's good to look at, haven't seen that before.

my plan for this is to make it completely independent of the website.

url -> archiver -> archive file (method 1)

url + client archive -> archiver -> archive file (method 2)

how it gets the input and how it deals with the output should be outside of the scope and a separate project