/hydrus/ - Hydrus Network

>I also want to dedicate some time to improving Linux stability, which is still not there yet.

Yay! Please do.

hydev, you gave me some code for an Inkbunny downloader the other day, but i have no idea how to add it to hydrus.

is there a tutorial or should i just wait for it to be added to the base program?

also i just found out a bug on the e621 downloader. any images downloaded that have a parent/child post relationship, either directly via URL or by gallery parser, will only download the parent image and not the child.

i suppose it has something to do with that URL recognition thing you added before, but im not sure, thats my guess from the parser log im showing here in the pic.

here's an example pls no bulli for the pictures contents :^) in this example 1537096 is the only picture to download, the posts 1537100 and 1537099 attempt to redownload 1537096 instead of their respective pictures.

>>8922

It's the sources linking to the same tumblr page that is causing the problem. See >>8856 and devs reply >>8869 for how to fix.

If you know how to SQL I have a query that will allow you to find all the posts affected.

SELECT * 
FROM urls 
WHERE hash_id 
IN(
	SELECT hash_id AS t
	FROM urls 
	GROUP BY hash_id 
	HAVING (
			SELECT count() 
			FROM urls 
			WHERE hash_id = t 
			AND url LIKE '%://e621.net%' 
			AND url NOT LIKE '%://e621.net/post/show/%/%'
			) >= 2
	AND (
		SELECT count() 
		FROM urls 
		WHERE hash_id = t 
		AND url LIKE '%tumblr.com%'
		) >= 1
	)
AND url LIKE '%://e621.net%'
ORDER BY hash_id DESC;

And to delete all the affected urls

DELETE  
FROM urls 
WHERE hash_id 
IN(
	SELECT hash_id AS t
	FROM urls 
	GROUP BY hash_id 
	HAVING (
			SELECT count() 
			FROM urls 
			WHERE hash_id = t 
			AND url LIKE '%://e621.net%' 
			AND url NOT LIKE '%://e621.net/post/show/%/%'
			) >= 2
	AND (
		SELECT count() 
		FROM urls 
		WHERE hash_id = t 
		AND url LIKE '%tumblr.com%'
		) >= 1
	);

Now I ask dev is there a way to disable the check for source urls as it's causing a lot of problems when multiple posts are linking to the same sources or like in pic. The e621 post id is a good unique identifier whereas the sources are wonky user defined.

i see the "manage file URLs" option but it has no option to delete

and here's the problem just like you said, the known URL's are getting mixed up which leads to one picture being downloaded several times in place of other pictures related to it.

>The e621 post id is a good unique identifier whereas the sources are wonky user defined.

i agree, at least as far as e621 is concerned, using the post UID seems to be the best solution to stop this from happening.

>>8923

if there really is no other choice whats a quick way i can run this script you've given me? do i have to install SQL?

oh and, sorry for the multireply, don't mean to spam the board.

>>8931

well, shit.

can you make a script do re-download all of these properly? or will the glitch happen again until this is fixed?

>>8908

>>8939

Thanks–that info is useful. I am sorry for this shit.

>>8913

I've set it now so if you DnD onto a multi-watcher page (i.e. that multi-watcher is the current page in the client), it'll be added to that page rather than the leftmost. This should make it doable to have multiple multi-watchers (eventually with different checker options and so on, so you can have 'fast multi-watcher' vs 'slow multi-watcher').

>>8916

Thanks m8. Things are a bit stressful on my end right now but it is nothing existential. It should get easier in a bit. I'm hoping to have a half-week for E3, so hopefully I'll be able to bosh some of the more pressing issues by v310. I am feeling good about current progress.

>>8918

Thanks, I will check this out. Since the mp4 links are in the post body, I think this will ultimately be the 'files and attachments and inlines' parser (or just a watcher, which does this), but that also isn't picking up the mp4 links correctly, so I will look into it.

>>8921

The download ui is all half-finished and still under construction, so best bet is to just wait for now. I'd like to roll in a bunch more support for all kinds of shit over the coming months.

If you want to brave it, check out the network under-construction sections. url classes, parsers, url class links should be what you need for now. Don't click apply on any of that if you can't figure out basically what is going on in each place.

>>8922

>>8923

>>8924

>>8925

>>8926

>>8931

>>8934

I've fixed this issue a bit more for v309. Now, if a url that is supposed to have one file actually has many–like your incorrectly mapped e621 urls here–that info will be discarded and the file will be checked anyway. Hitting 'try again' on these entries in v309 should get the right files (you'll just be left with some dupe e621 file-url mappings from the error previously) but I will also add overriding 'get file even if url known' and '…if hash known' checkboxes to force a fix if we encounter more logic holes like this.

The manage urls ui is still 1.0, which I threw together. Double-click to delete for now. I will give it another pass at some point.

If you aren't completely comfortable with the SQL (make a backup beforehand if you want to try anyway), please wait a couple of days and do 'try again' in the new client and let me know if it works for you.

If you do want to do this URL stuff, do it in v308! v309 has a different url db structure!

>>8932

Transparency is working a bit better these days–can you post/link/email me some images that render bad for you, along with a screenshot of the situations in the client where it messes up? I may be able to fix it natively just by jiggling some palette/colourspace stuff around.

>>8937

>>8941

You probably saw exactly this section, but if not, these are my complete thoughts on sticking to hashes and a non-human-friendly file store:

http://hydrusnetwork.github.io/hydrus/help/faq.html#filenames

Hydrus has no way to save tags to images or image filenames. It always stores tags separate from files inside its internal database in a complicated and highly indexed way so searching works fast. Tags have to be indexed in a database as soon as you get above 10k or so files in order to deliver results in a reasonable time. The larger the collection, the less useful it is to store tags in explorer-friendly ways.

There are also no folder structures, although if you decide you want to go with hydrus, I recommend you tag by folder as you import your files. Adding local tags like 'reaction images' or 'pepes' or whatever based on folder names is a great way to get some initial tags going so you can find things to post real fast.

Just as an aside: saving tags to images, in their metadata, is a common cause of headaches when trying to manage large collections, due to the file's byte content changing and hence changing the file's hash. The tagging systems are also often OS-specific and a pain to extract with third-party programs en masse. If you expect to get into any kind of big file collection system or ever want to pull tags for your files from boorus with any program/script, I recommend refraining from setting NTFS or jpeg/png metadata tags.

For exporting images from hydrus to other programs, the absolute easiest way is to just make a thumbnail selection and drag-and-drop it out of the client. You generally should be able to drag and drop it to a website's 'drop files here' upload box in your browser. (I just did it for this post)

If you want to DnD to Discord, that can be more tricky–try the BUGFIX option under options->gui.

Unfortunately, DnD exporting like this keeps the ugly hexadecimal SHA256 hash filename (you'll notice that it is generally the same as the 8ch filename–they use the same scheme). I'd like to extend the file export system to auto-rename files based on some tag info, so if your file had 'filename:hack master' tag, it would auto-name to 'hack master.gif' on all kinds of quick export.

If you ever want to export with better filenames or sidecar .txt files, check out thumbnail right-click->share->export files. You can fairly quickly export a hundred tagged images and give them names like 'evangelion doujin whatever - chapter 2 - page 12.jpg' or 'character:rei ayanami plugsuit smile.jpg' if that is the sort of thing you are looking for. I expect to do something similar for the auto-rename system when I get to it. If you give it a go, let me know if you run into any trouble with it.

>>8940

iirc, copy_file sends a 'file' content type to the clipboard, which has multiple ways of being caught depending on what the destination program tells Windows it can take.

So, if you copy_file and then paste into an Explorer window, it should actually copy the file, but if you paste into notepad, Windows will collapse it into the text path again. copy_path just sends the text type. My guess is that ShareX can't take the file content type but can take text, so you are only getting the path.

I am afraid I don't know anything about ShareX though, so I can't speak with any authority about it. What's the destination ui look like here? How it is 'catching' the paste events?

In general though, I don't recommend using copy_image to share files, as it actually copies a bmp (which is then converted to png usually by the browser/whatever destination). It is useful for pasting pixel data into image editing programs.

Can you drag-and-drop a hydrus thumbnail selection to ShareX? If not, can you DnD to a common folder like your desktop and then just point ShareX at that?

>>8942

It looks like they are rolling out a new dynamic layout (which the parser can't handle) to users randomly on login. I am working on this problem. You can try clearing the login under network->DEBUG menu once or twice to see if they reassign you back to the old layout, but I don't know if this actually works. Your feedback on whether this works would be useful.

>>8953

>If you do want to do this URL stuff, do it in v308! v309 has a different url db structure!

nah, its fine, im in no hurry. i'll wait on ya.

i did however follow in anon's script to find all the 1200 affected pictures and that's as far as i dare to go, deleting the url's with the second script would be no good for me because then i wouldn't remember which potential 1200 urls i have to download again

>>8954

>It looks like they are rolling out a new dynamic layout (which the parser can't handle) to users randomly on login.

jesus fuck Pixiv changes UI more often than jewtube, japs bls stop

>You can try clearing the login under network->DEBUG menu once or twice to see if they reassign you back to the old layout, but I don't know if this actually works.

done, i tried it

>Your feedback on whether this works would be useful.

and here's my feedback: no

hope that helps ya hydev

>>8953

>If you do want to do this URL stuff, do it in v308! v309 has a different url db structure!

Does this mean it will be possible to (have the option to) transfer URLs from inferior to superior when doing duplicates?

This is at the top of my wishlist.

>>8955

I hadn't thought of this sort of need before, but sure, I will add a copy_file_bytes or something. If you get use out of it, let me know how it works for you.

>>8962

FYI: If you can munge a list of 'bad URLs' into a newline-separated list in your clipboard, you can then easily queue them up again in a new e621 booru downloader page by right-clicking the file import status button and going import->from clipboard. Tomorrow's logic should be fixed for your problem.

It may be easier to fix down the way, though, in the client. Maybe some combination of a future system:num_urls and a 'select->copy all urls' or something. Let me know how you get on and how I might automate a fix here.

Thanks for this info about pixiv. I don't know what the nips are up to then–maybe opting-in randomly on login and never rolling back. I know another user had luck by creating a new, throwaway account that then didn't get the new layout. Another user who tried this did get the new layout.

Current plan is to extend the parser to just work with both layouts.

>>8963

You should be able to do this now! I added it a couple of weeks ago, I think. Check the merge options in the duplicate system–there should now be a checkbox for 'merge urls from worst to best' or something. I think I re-laid out a bunch of that panel. Let me know if you can't find it.

>>8964

Oh shit, it's true, thanks! I had to look for it, but it's there. Time to redownload everything and transfer the URLs. I'll be spending the next week legally dead for organization reasons.

Hello Hydrus Dev. I haven't posted here in such a long time, and my Hydrus Network was very outdated.

So I downloaded this newer version here, and from what I remember all I had to do was extract everything, then replace the new files with my old ones. But now it just won't start. It gives me an error telling me it is missing master boot files or something. It has something to do with my db, because it works, until I move my db folder over. Don't tell me I'm going to lose all my tags ..