/hydrus/ - Hydrus Network

>>10578

That is not going to be convenient unless we can make it simple (especially when someone has TBs of data on a RAID-enabled NAS)

>>6718

Any comments on these?

https://github.com/restic/restic (Golang, not too slow)

https://github.com/gilbertchen/duplicacy (Golang, fastest)

https://github.com/duplicati/duplicati (C#, fast)

https://github.com/bup/bup (Python, slow)

https://github.com/jborg/attic (Python, slow)

See https://github.com/gilbertchen/benchmarking for info

>>11118

Borgbackup is your ticket to make it simple.

>>11123

> https://github.com/bup/bup (Python, slow)

Actually this one is a fine CLI tool too, but unlike borg it's not considered stable by the developers, and you likely do want some borg features like the ability to prune older database backups and staggered retention for the other files.

Slow is extremely relative. Yea, python is not the fastest language around even if you use pypy (which however definitely makes most software that crunches some data quite a bit faster over standard python). But even that may not matter much if you're mainly waiting for IO coming from your HDD, which is usually what happens if you try to get data about your 40TB of image files (or whatever you might have on your array).

> See https://github.com/gilbertchen/benchmarking for info

A very limited test without the various configurables and apparently without any attempts to isolate how much time was spent waiting for IO or whatever, done only on the weirdly scheduled OS that is OSX?

And I don't even see what interpreters they used for python (was the "python" symlink pypy or mainline python and not a link?) and such. Never mind there are actually a LOT of configurables in these tools, and for some weird reason they didn't even try to pick the same compression algorithms even when it was possible (why not run the same settings on the same zlib or lz4 or zstd compression on all the tools so they become more comparable?).

Anyhow, use Duplicacy or restic over borg if you prefer, but I wouldn't pay too much attention to that benchmark.

>>11123

I'm using duplicati myself, it's still slow because OS chokes on the amount of hard drive seeks, takes around 30 minutes to back up a 100k set for me, probably going to be even slower when you're backing up from an encrypted volume. It doesn't even matter if there were any changes to the collection, just the act of checking 100k files on a hard drive is a very slow process.

>>11123

I haven't used any of those, so I can't comment cleverly. I expect they do the job well. I use FreeFileSync once a week on my laptop's IRL 1.2M-file hydrus db, backing up to a WD passport. It takes several minutes to compare all the millions of files and thumbs and then a few more to sync the backup. Unless I have a gonzo week, I'd estimate it is usually less than 20min total. Since I have it going in the background as I put the hydrus build together on my dev pc, it is no trouble at all.

>>11110

we have IDs here cowshitter

>>10944

Finally this happened again. Hydrus had to run a pretty big maintenance job at exit, and once it was done it's just stuck there in the task manager as client.exe. I noticed about two and a half hours after I exited the program. There are no .db-shm and .db-wal siblings in the db folder, and it's not using CPU/HDD.

However, once I started writing this, it actually did shut down. Since I only caught this just before it shut down my notes about the .db file siblings and CPU/HDD usage might not be accurate. Here's the log, notice the 2 hour gap…

2018/12/26 09:15:06: hydrus client started
2018/12/26 09:15:06: booting controller…
2018/12/26 09:15:06: booting db…
2018/12/26 09:15:06: preparing disk cache
2018/12/26 09:15:10: preparing db caches
2018/12/26 09:15:10: booting db…
2018/12/26 09:15:10: initialising managers
2018/12/26 09:15:17: booting gui…
2018/12/26 09:15:21: Import folder cosplay imported 6 files.
2018/12/26 09:18:51: shutting down gui…
2018/12/26 09:18:51: waiting for daemons to exit
2018/12/26 09:18:53: vacuuming main
2018/12/26 09:18:53: Vacuumed Y:\db\client.db in 176 milliseconds
2018/12/26 09:18:53: Could not vacuum Y:\db\client.mappings.db (probably due to limited disk space on db or system drive).
2018/12/26 09:18:53: vacuuming external_master
2018/12/26 09:20:04: Vacuumed Y:\db\client.master.db in 1 minute 10 seconds
2018/12/26 09:20:04: vacuuming external_caches
2018/12/26 09:20:06: Vacuumed Y:\db\client.caches.db in 2.2 seconds
2018/12/26 09:20:06: database maintenance - analyzing

done!
2018/12/26 09:20:06: public tag repository sync: processing updates
2018/12/26 09:20:06: analyzing specific_deleted_mappings_cache_1_3
2018/12/26 09:20:06: analyzing specific_deleted_mappings_cache_9_3
2018/12/26 09:20:06: analyzing deleted_mappings_3
2018/12/26 09:20:06: fattening service info
2018/12/26 09:21:07: processed 1172848 definitions at 21831 rows/s
2018/12/26 09:30:02: processed 7445542 content rows at 13909 

>>11113

Thank you for this follow-up. Since the db does seem fully shut down and disconnected, my best guess here is that one of the daemon threads (which do maintenance stuff in the background) isn't waking up to receive the 'program shutdown' signal properly. In this case, the process will hang on, with that one thread asleep, until it wakes according to its natural check period (which are typically on the order of hours).

I will check this code. It may also magically fix in py3 due to the different way some thread signalling works as well, so please let me know if this improves/worsens after v335.

In this case, as the db is completely closed, there is no danger in just killing the process in task manager when this happens again.

>>11045

>but first i wanna ask about what storage hardware is best for hydrus?

I don't think it matters, but you may want to

keep your media files on a HDD and the rest on a SSD. Just stay away from Seagate, their HDDs are known for having a lot of issues. I'd recommend HGST or WD.

https://hydrusnetwork.github.io/hydrus/help/database_migration.html

>>10985

For bandwidth? Go network->data->review bandwidth usage. For your specific problem, I think the easiest and best solution is:

edit default bandwidth rules to change your client-wide rules for all subcriptions. Just delete any rules in there.

Then scroll the list on the main window for 'web domain:tumblr.com' and double-click and set specific rules and similarly set no rules.

This will make all your subs work as much as they want and also let any tumblr jobs work as much as they want.

>>10987

Thanks. I don't see anything obviously wrong here. If hydrus is waiting for the login, then everything should be working ok. Could HF maybe not apply your account's filters right after logging in? I am really not sure here.

My test HF login on my dev machine lasts 30 days, and I think it keeps upping that back to 30 days every time I request, so I feel like a regular client that hit HF every few days would only ever log in once, or a higher hard limit like once a year–do you see your client logging in HF a lot? What does it say under your network->downloaders->manage logins? Does it say 'yeah, logged in for 29 days' there, or something else? I wonder if perhaps your credentials are incorrect or the login script is failing to log in right, so you are accidentally being given a filtered guest login at times.

>>10992

here, this is a known link that doesn't pull images up, I tried going incognito to find ones, but apparently it finds the images that way too. for the artist, so I stuck with a known one that wouldn't work.

https://mega.nz/#!KooGTIBC!aRlDdrI1XRA-iMDmkzye7J6Q-C3bTWnb3_74-8-0eDI

Honestly, I think full stopping the download, program wide, till a log in finishes would be a good work around for this. or with the downloaders that these download from, a double tap when logging in, the first page or two of searches could get redone twice, because by then everything should be logged in, and if something didn't catch it would get them in the first two pages.

>>10991

I set everything to super high file and data limits there already though (like 100k files and 100s of TBs worth of bandwidth), and it's still "waiting on bandwidth" half the time. That change affected both my global and the specific tumblr rules.

Also tumblr.com specifically (not its subdomains) has "yes" in the "blocked?" column. What does this mean?

>>11000

Oh, never mind, I had to edit the web domain rules as well as the subscription rules, seems like. All working now.

>>10876

>>10878

>>10880

>>10901

I'm still laughing/in disbelief about tumblr's decision. I thought after the uproar they would swiftly reverse the decision, but I guess they don't want to subsidize free porn bandwidth any more. The only conclusion I can make is Verizon just don't have a real business plan for it. No idea why they took it on in the Yahoo purchase. What a way to kill a brand.

Is their a way to search through Notes or see which images have Notes written on them?

>>10859

Thanks, 0.8 works better, it scrolls a full page per tick now. I have my scroll length set to 5. One page is about 5½ thumbs high.

I think this might also be related to the fact that I have Windows 10 set to 125% scale, Hydrus doesn't seem to handle that quite correctly (for example when resizing the width of the tagging panel to the left of the thumbnail view).

>>10917

Not yet, but I'd like to present that information better in several ways. Adding search would also be useful.

>>10933

I hope to check out high-dpi stuff in the new year. The new wx is supposed to be able to handle it better, so I'll have a look and see if I can figure out what flag I need to set or whatever. I have a 4k monitor now, so I can test this better.

>>10892

I can't wait.

It's gonna be three-three-three!

Yay!

>>10896

1/3 of the millenium… time goes by fast

>>10897

I don't really get what you're saying… it was simply a play on "three three three" with the image, since NS Germany is referred to as the "Third" Germany as well.

>>10899

333 is one third of 1000, i.e. the Millennium Reich

>>10864

Please read this https://github.com/jaseg/python-mpv to see how easy that is

Might as well plug this in if people want it https://github.com/rg3/youtube-dl for downloading from YouTube, Soundcloud and all the alt-tube sites

Also regarding the API, according to most mobile devs, they would prefer Danbooru/Moebooru API over everything else, so I drafted an API spec for Hydrus https://ghostbin.com/paste/hghv2 (Gelbooru could come later)

And Nori dev (tjg1) says the downloader script GUI is overengineered, whatever.

Sorry lads, but how can I bulk-download booru-imageboard image files in bulk?

>>10869

>>10868

Sorry for not putting this all into one post - but I tried to install "bionus" "Imgbrd grabber" by the way. Is something like this possible with hydrus!? I never tried…

>>10870

Imgbrd is a separate project that does similar things, this board is not responsible for it, best ask in https://github.com/Bionus/imgbrd-grabber/issues

Bionus & pals has promised to make things easier for us to import data to Hydrus, but it is not yet well integrated.

https://github.com/Bionus/imgbrd-grabber/issues/1001 and https://github.com/Bionus/imgbrd-grabber/issues/588 has some notes

Hey, thank you for this report.

I think you have an unlucky older version of the client here. My recommendation is to update to a more recent version.

More specifically:

lz4 started having trouble importing recently in a variety of situations, I am not sure why. I put some time into it but couldn't figure it out and will revisit it in the new year when I am on python3. It is not required.

pafy and youtube-dl shouldn't be a big deal either, but I guess the new version complains if it is missing. I think you can fix this on the python end by installing 'pip install youtube-dl', which I guess adds some python bindings to youtube-dl and is actually what pafy is looking for here. Pafy (which does some borked youtube downloading) is only needed for some experimental code, so I will change the code to no longer dump out if import fails.

The ordered_dict issue turned up a few weeks ago, when requests updated and could no longer unpickle (load from a serialised object) its sessions. The code in newer versions should recover from this error and regenerate empty sessions for you this one time.

Please let me know how you get on.

>>10843

I was afraid of updating as my hydrus db has surpassed my backup drive in size. I updated to 332 (from 317) anyway, everything went fine phew.

Running from source now yields this error:

Could not import lz4--nbd.
Traceback (most recent call last):
  File "H:\Hydrus Network\client.py", line 20, in <module>
    from include import ClientController
  File "H:\Hydrus Network\include\ClientController.py", line 14, in <module>
    import ClientCaches
  File "H:\Hydrus Network\include\ClientCaches.py", line 1, in <module>
    import ClientDefaults
  File "H:\Hydrus Network\include\ClientDefaults.py", line 2, in <module>
    import ClientData
  File "H:\Hydrus Network\include\ClientData.py", line 3, in <module>
    import ClientDownloading
  File "H:\Hydrus Network\include\ClientDownloading.py", line 2, in <module>
    import ClientNetworkingDomain
  File "H:\Hydrus Network\include\ClientNetworkingDomain.py", line 3, in <module>
    import ClientParsing
  File "H:\Hydrus Network\include\ClientParsing.py", line 4, in <module>
    import ClientNetworkingJobs
  File "H:\Hydrus Network\include\ClientNetworkingJobs.py", line 9, in <module>
    import HydrusNetworking
  File "H:\Hydrus Network\include\HydrusNetworking.py", line 9, in <module>
    import ssl
  File "C:\Python27\lib\ssl.py", line 126, in <module>
    from _ssl import HAS_SNI, HAS_ECDH, HAS_NPN, HAS_ALPN, HAS_TLSv1_3
ImportError: cannot import name HAS_TLSv1_3

>>10875

Great, I am glad you figured this out. Let me know if you encounter any more problems.

>>10808

For me, I always take a url like

https://pbs.twimg.com/media/DrkOeeqU0AUdsbI.jpg:large

and turn it into

https://pbs.twimg.com/media/DrkOeeqU0AUdsbI.jpg:orig

I get the first url from just "view image"'ing them in the browser.

>>10747

I said this a while back, you may as well stop with exhentai/e-hentai

I found a script that will get download links, it will stager the downloads, and even then, at 1 dl a minute, they still catch it, its just not worth the effort form them.

>>10812

What were you trying to download?

I sometimes open various(10-20) doujinshis at once and use a full page script to leave them all opening at once to then save them, but it's a short rapid burst.

If you're downloading a lot of shit over an extended uninterrupted period of time that could be why.

>>10822

the last one I downloaded was an 8 page comic with everything spaced out 1 minute, I did all my testing on very small downloads and regardless of what I did, 1-1 week long lockouts.

It seems they want you to pay them for the ability to download things, as 20$ gets you a weekly 5gb pass on downloads, something I would gladly pay, but I need to link bank accounts and do money transfers for cryptocurrencies, and i believe my bank was one that was one that was punishing people for even trying.

>>10811

>>10808

Thanks. I hadn't heard of the jpg-orig trick. Hydrus currently does the :orig conversion, which works well as long as the artist themselves didn't originally upload a 350x400 80% quality jpg. :^)

I know that twitter can provide direct mp4 links for videos, but I haven't figured out a way to parse this info. This service provides it:

https://twitter.com/this_vid

But takes a couple of minutes to produce an answer, so my guess is they manually request it from twitter using a legacy "I'm an old phone" http header or something and then forward the link. Normal twitter video works with some DASH streaming or something that we can't handle (yet).

If you discover a way to get simple mp4 links for twitter vids, please let me know!

Is that why hydrus_dev spent the last year working on the downloader engine? I guess he wasn't actually working on Hydrus then according to you.

Fuck off with your attitude

>10830

Imagine actually believing this is a serious post

>>10829

< Not valuing API because archiving is a waste of time

When everything is deleted off the internet constantly you will regret that.

WARNING:root:pafy: youtube-dl not found; falling back to internal backend. This is not as well maintained as the youtube-dl backend. To hide this message, set the environmental variable PAFY_BACKEND to "internal".

Traceback (most recent call last):

File "/opt/hydrus/client.pyw", line 20, in <module>

from include import ClientController

File "/opt/hydrus/include/ClientController.py", line 14, in <module>

import ClientCaches

File "/opt/hydrus/include/ClientCaches.py", line 5, in <module>

import ClientRendering

File "/opt/hydrus/include/ClientRendering.py", line 2, in <module>

import ClientImageHandling

File "/opt/hydrus/include/ClientImageHandling.py", line 3, in <module>

import cv2

ImportError: No module named cv2 looks like

opencv update on Arch makes hydrus unable to boot, 3.4.3-5 -> 3.4.4-1

>>10729

>Seems fucking retarded, what's wrong with just having blue boards?

He has done the same thing to 2ch.

His explanation was like this.

If something wrong occurs in an adult board like someone uploading CP and law enforcement will shuts down 2ch entirely.

But as far as 2ch and adult boards called pink server separated and we treat these two as complete different sites.

At least 2ch is kept safe.

>>10732

Thanks. I found this after a bit of searching:

https://bugs.archlinux.org/task/60855

https://github.com/opencv/opencv/issues/13202

So I guess those are some places to watch? If the python bindings for cv got messed up in the new version, I should think reverting is the simplest solution for now.

OpenCV is critical to Hydrus's drawing, so I'm afraid there isn't much I can easily do on my end.

>>10728

Thank you for reminding me of this. I made a new url class, rolling out tomorrow, that seems to add complete support. (we'll see if after the official split if sfw content/api is still stored on 4cdn domain, which the current watcher goes to)

>>10730

>>10729

learned a new thing since posting that.

4chan lost stripe, you cant buy passes with credit cards, this is the reason for the split.