[ / / / / / / / / / / / / / ] [ dir / animu / htg / islam / leftpol / lewd / mde / vg / vichan ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Experienced user with a bit of cash who wants to help out? ---> Patreon

Current to-do list has: 1,473 items

Current big job: none, voting on next job


YouTube embed. Click thumbnail to play.

db6452  No.6032

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v258/Hydrus.Network.258.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v258/Hydrus.Network.258.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v258/Hydrus.Network.258.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v258/Hydrus.Network.258.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v258/Hydrus.Network.258.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v258.tar.gz

I had a great week. I have finished the duplicate system.

duplicates clear

Thumbnails will now tell you about their duplicates when you right-click on them. You can see how many of which sort of duplicates they have and then see those groups in a new page. You can also mass-set 'these are all alternates' and so on. This is a powerful tool, so be careful! I may end up hiding this and some other things behind an 'advanced mode' in some way, but let me know how it works for you.

This final feature means that–other than some help page polishing and ui grammar improvements and so on–I am done with the duplicates system. From now on, I will fold any requests or bug fixes into my normal time that I spend on all the other systems in the program. I am really pleased with how this has gone and the positive feedback it has received. It took a lot of planning and work to go from the bare mathematics of it to something humans can actually use, but thankfully none of it proved impossible.

If you have been waiting for the duplicate system to be finished before you jump into it, now is a good time. The starter help is here:

https://hydrusnetwork.github.io/hydrus/help/duplicates.html

more webms supported

I have added basic support for video with variable framerate. They won't render completely correctly in the client–they'll go at an average frame rate–but they will have the correct duration and number of frames. From the broken files people have helpfully submitted, this seems to mostly mean webms. The ones I have seen look fine at constant framerate, but let me know if you discover any interesting exceptions.

Anyway, if you have a folder of broken webms that wouldn't previously import, give them another go in v258. It worked for most of mine!

some setbacks

There are several ongoing problems as well. I am also fairly IRL busy and a bit stressed, although I expect that to clear up in a couple of weeks.

When I updated to the new service system, I accidentally broke the 'advanced content update' panel (launched from manage tags or review services as 'advanced service-wide update'. It was unable to discriminate by namespace, so it seems it would sometimes do nothing and sometimes include all namespaces, including unnamespaced tags. If you were affected by this, I apologise. I will write some more unit tests to make sure it doesn't happen again.

And while I got around to having a closer look at a neat new IPFS option, 'nocopy', as I had been meaning to for a while, it is unfortunately not yet compatible with the IPFS http API, which requires the file encoded with a POST FORM and, as far as I can figure out, will not otherwise accept a simple file path. In order to support this experimental feature, I will have to expand the existing hydrus-IPFS interface to support command-line interaction. I intend to do this, but the parts of it will have to be spread out over a longer time.

You also may have noticed my server has been giving ssl errors on-and-off in the past couple of weeks. This has been frustrating, requiring a reboot on my end to fix, but I now know the cause of the problem (not ssl, as it turns out), and have put some time into trying to fix it. I may have cracked it this week, but we will have to see as I do not completely understand the exact timings that cause the server to lock up. If you run a server, you probably want to update this week, and if I haven't quite fixed it yet, perhaps in the near future again as well.

full list

- added a duplicate entry to the thumbnail right-click menu

- the new duplicate menu will now attempt to fetch known duplicate counts for the focused file for the current file domain (if the db is locked, it will say so rather than block)

- the new duplicate menu's counts in the thumbnail menu can be clicked to show those files in a new page

- the new duplicate menu allows you to set the four main dupe statuses (with default merge options), or a customised verson of the same, or set unknown/potential, or delete dupe status completely, for all the possible pairs in the current selection

- added tentative support for variable framerate files, manually counting up their frames and displaying them with an averaged constant framerate in the client renderer. this now permits a number of webms that were previously 'mime unsupported'. let me know how it works for you!

- cleaned some server locking code, hopefully fixing the ssl handshake issue (which is actually a deadlocking issue)

- export file dialogs will remember the last value of 'export tags to .txt files?', and clicking that checkbox will always launch the dialog for editing (rather than clearing if there are some set)

- fixed the advanced content update (as used by service-wide update), which was not able to differentiate namespaces since the recent service changes. I apologise to anyone affected by this–I will add some unit tests to make sure it doesn't happen again

- reworked and cleaned some canvas event handling code

- the canvas will now accept mouse wheel events even when it does not have focus

- the canvas frame will no longer accept keyboard events (and hence activate fullscreen_switch) if the mouse is over a flash window

- hydrus servers now respond to /robots.txt with a 'disallow all'

- servers will now provide mappings petitions to admins in groups of similar size, rather than mixing petitions of weight 5000 with a hundred of weight 1

- fixed the back/skip buttons in the archive/delete filter's top hover window

- the file path tagging dialog's namespace and regular regexes now support parenthesis groups, and in that case will take each of those submatches instead of the wider match

- hence regexes with groups will now compile in that dialog

- you can now 'new page here' on the final (rightmost) page tab

- fixed an issue where media-viewer-launched manage tags frames would yield focus to the main gui if they had opened a sub-dialog while they were open

- the tag censorship edit panel now has a 'help here →' label in blue text

- file imports will fail properly when the final temp_path->client_files file copy operation fails (due, for instance, to the destination being on a removable medium that was recently disconnected)

- fixed serverside upnp maintenance, which was silently failing early, and also updated it to the new server object code

- wrote a new checklistbox dialog that handles the underlying data explicitly rather than messing around with external text->data maps

- replaced all the old string checklistbox dialog instances with the new one

- subscriptions will now correctly clear their 'recent error' status on a successful sync

- refactored and cleaned how basic mediaresult objects store and consult simple file information

- locationsmanager objects now handle inbox status and the related content update events that affect it

- fixed a bug in HydrusTagArchive when attempting to figure out hash type from an existing hash

- removed the obsolete servertoclientcontentupdatepackage object

- misc duplicate code cleanup

- misc serverside service code improvements

- misc small fixes

- misc cleanup

next week

Now the duplicate system is done, I will be moving on to the downloader engine overhaul. This will completely update all the behind-the-scenes of every downloader in the program, making it all more capable, flexible, reliable, shareable, diagnosable and maintainable. The first thing will be adding a proper sense of 'state' to the downloaders so they are cookie-aware and generally more polite to the servers they talk to. With luck (cross your fingers, as this may in the end be out of our control!), I'll have sankaku and hentai foundry working again in a few weeks.

b54ab5  No.6035

>I am also fairly IRL busy and a bit stressed, although I expect that to clear up in a couple of weeks.

Take a break as you need it my dude. You work on Hydrus practically every week so it's not like you don't deserve a vacation. Besides, I doubt an extra week of waiting to update will kill anyone.


9a1be4  No.6036

>downloader engine overhaul

does this include fixes to parsing script issues?


b42db1  No.6037

File: b75199670fd2ed7⋯.webm (615.18 KB, 480x270, 16:9, hug.webm)

Wouldn't it be nice to pamper hydrus_dev once and a while? Just platonically of course.


e2a73a  No.6038

Thank you so much for doing

>>5645

>I will be adding a way to manually edit dupe relationships from the thumbnail menu. Everything you can do in the dupe filter will be possible through it.

I am very grateful!

Please make sure you take some rest as needed. Better to take a short break when you can still go on than to take a long break because you can no longer go on.


14fea2  No.6039

File: 588007623c77e0b⋯.jpg (278.4 KB, 1414x1000, 707:500, 588007623c77e0b6776f5c43dd….jpg)

Excellent job as always. I toyed around with the right-click menu since I had over 800 siblings in a namespace waiting for this function to be created. So far, so good.

One quality of life improvement I'd like to see is that when filtering files marked as alternates/better-worse/whatever that either clumps them into a "collection" or that they appear next to each other. Right now it shows everything all over the place.

Anyway, hope you can relax for a bit and keep up the good work.


22adcf  No.6040

>>6036

Yes.

>>6037

Imagine forcing him to wear a diaper and feeding him grapes. Wouldn't it be funny? Just a joke haha


03a4e6  No.6041

File: 48758c696d4f9b4⋯.png (7.39 KB, 1347x40, 1347:40, client_2017-06-01_15-33-13.png)

Ok found a rather annoying aspect for the downloader that I think would be best addressed first.

here are two links, I have no idea what the files are from, all I know is the links came from /d/ backups that are not working

This one results in a 404

https://desu-usergeneratedcontent.xyz/d/image/1487/23/1487230801559.jpg

This one results in a 0 file size

https://desu-usergeneratedcontent.xyz/d/image/1383/98/1383986888606.png

Now up till now I took 0 file size as the image is dead and not working, I can easily deal with that, it's not an issue, files dead and there is no bringing it back. However the other one is a 404, I know the program can see 503, but not 404?

This is the string that it kicks back in the image, this implies that, at least to me, that there was a file there that the program is just not able to read rather than a 404.

Also, here is the text from copying the error.

~~~

MimeException: Filetype is not permitted!… (Copy note to see full error)

Traceback (most recent call last):

File "include\ClientImporting.py", line 1439, in _WorkOnFiles

( status, hash ) = client_files_manager.ImportFile( temp_path, import_file_options = self._import_file_options )

File "include\ClientCaches.py", line 979, in ImportFile

return self._controller.WriteSynchronous( 'import_file', *args, **kwargs )

File "include\HydrusController.py", line 391, in WriteSynchronous

return self._Write( action, HC.LOW_PRIORITY, True, *args, **kwargs )

File "include\HydrusController.py", line 117, in _Write

result = self._db.Write( action, priority, synchronous, *args, **kwargs )

File "include\HydrusDB.py", line 859, in Write

if synchronous: return job.GetResult()

File "include\HydrusData.py", line 1681, in GetResult

raise e

DBException: MimeException: Filetype is not permitted!

Database Traceback (most recent call last):

File "include\HydrusDB.py", line 534, in _ProcessJob

result = self._Write( action, *args, **kwargs )

File "include\ClientDB.py", line 9801, in _Write

elif action == 'import_file': result = self._ImportFile( *args, **kwargs )

File "include\ClientDB.py", line 6041, in _ImportFile

( size, mime, width, height, duration, num_frames, num_words ) = HydrusFileHandling.GetFileInfo( temp_path )

File "include\HydrusFileHandling.py", line 168, in GetFileInfo

raise HydrusExceptions.MimeException( 'Filetype is not permitted!' )

MimeException: Filetype is not permitted!

~~~~

now, again, this one is from /d/ so content is a bit iffy.

This page has a specific file, an mp4 to be exact

https://desuarchive.org/d/thread/7076232/

the file being

https://track3.mixtape.moe/nnsnfr.mp4

I have no idea why it embedded itself into the thread like it was an image/post but it was there. however in the download it wont do anything. Now, I had thought I set the program to only handle gif, png, and jpeg files however the program was importing webm's without issue, so I downloaded the video and tried importing it from file, it worked.

I'm not sure if there is something that is very specific at play here not allowing the mp4 to download, but it's a bit of an issue.

I honestly think that reporting why an image fails to download takes president over why a file wasn't part of the download to begin with but that's just me.


ad5743  No.6042

This fixed a whole bunch of my failed webms. Cheers, Hydrus_dev, for fixing an issue that probably wasn't affecting anyone else but me.


d23ade  No.6063

feature request:

please add a "trash both" option to duplicate sorting. oftentimes, i come across a pair of images where neither of them are images i want to keep. being able to discard both with one click would make things way, way nicer.


47b3ec  No.6064

Awesome job! I only have one question about the downloader. I am trying to download off of Rule34Hentai and I'm wondering if there is a way to add an account login like you can with pixiv? Some of the images don't show up unless you're logged in, and they aren't getting downloaded. Is it somehow possible to do this?


13993f  No.6065

>>6063

Not to get in the way of your dreams, but making this an icon is risky. People will click the wrong icon and get unexpected total information loss results.

Alt-del or something would probably be safer already.


e2a73a  No.6066

>>6063

You can already click 'delete selected'? No need to make two buttons for the exact same action I'd say.


9a1be4  No.6069

>>6066

That only deletes one and immediately moves on to the next pair so you don't get the chance to delete the other one


9a1be4  No.6070

>>6065

Could be hold shift while clicking the button


22adcf  No.6071

>>6063

>>6065

>>6069

>>6070

You can already do that with the custom option button, by selecting any option from it and then "delete both files".


db6452  No.6078

File: cc4dc931e864b5a⋯.mp4 (3.4 MB, 640x480, 4:3, cc4dc931e864b5a96ae838c946….mp4)

>>6035

>>6037

>>6038

>>6039

>>6040

Thanks lads. I'm actually doing great overall, and I'm under much less risk of catastrophic burnout now I am older. I specifically keep my current stress and exhaustion in mind so I can better regulate it. In any case, I am mostly over the hump now.

As well as the rest of my life easing up, I'm planning to take a bit of extra free time next week so I can shitpost over the E3 streams.

>>6036

Yes, there will be many fixes and expansions to the parsing script engine, which truthfully remains in a prototype state. All the new gallery/thread/page parsers will in some way go through a future version of that dialog.

>>6039

Thumbnails aren't themselves yet aware of their duplicate relationships, and the underlying data here is a network of pairs that is not always easy to sort. I plan to create dedicated gui to better display dupe relationship (and particularly to manage and order the different 'groups' and family relationships of alternates) in a second phase of the duplicates system. For now, it will have to be a bit ugly and awkward on the gui side.

>>6041

Thank you for this excellent report. I will look into all that is going on here for next week.

>>6042

Great!

>>6063

>>6065

>>6066

>>6069

>>6070

>>6071

Several people have now asked about this. Rather than create a special new dupe context that doesn't actually do and dupe stuff and just holds some dupe merge options for 'delete both', I will alter the standard delete dialog for the dupe filter to ask if you want to delete just the file in view or both. Let me know how that works for you.

>>6064

The downloader engine overhaul that I am just starting will include a new 'login engine' that will allow this for almost all domains. It won't be the first thing I work on, but it is coming. Please let me know how it works out for you as I roll it out.


6fe4a1  No.6081

Slideshow functionality seems to be broken on Windows 10.


03a4e6  No.6083

So the update one maybe 2 weeks ago had a comment that said they've noticed an increased load speed when they opened up the browser.

Normally for me it takes may be 4 to 5 minutes for the program to load up, I use it in a very stupid way, and to top it off I am very lazy with going through the images so this leaves may be 80 to 90 tabs open, possibly more, about 10 of which are over 3000 files big and one is for over 50,000. However after the person mentioned that and the program crashed twice, I noticed that took under a minute to load up. Now going to one of the download tabs in the program at the time and trying to sort the errors would crash it 9 times out of 10 if not every time I stopped after it started crash constantly so I had to restart the program multiple times and I watched it one of the times, the program loaded up the images that may be 5-600 images a second at some points. I'm not sure what changed to make that happen, I don't know if it's a setting that I could make happen all the time, but having the program crash while going through the settings menu just now and loading it back up and it reverting back to being really slow makes me really miss how fast that was.

Something is in there that makes loading the program potentially fast, but I don't know what it is. I don't know how useful this would be to everyone else, but I'd look into the reason why it's faster and see if you can make it happen every time.

If somebody knows how to make this happen every time I'm all years and would be glad to learn.


e2a73a  No.6084

File: 384b97c2409eefa⋯.gif (222.51 KB, 180x180, 1:1, shiggy.gif)

>>6083

>80 to 90 tabs

>mfw

>5-600 images a second

Is this fast? I have about 6000 images and it loads it in about three seconds when I load them just after launching the client. I mean the time it takes while it says "Loading … x of y".


03a4e6  No.6085

>>6084

loading off a 5200 rpm archive drive till nvme 2tb gets under 300$ and when the program normally loads, it will go 120 images, freeze, then 250 images 5 or so seconds later.

500 may be low considering how fast it loads some times, but even 500 is significantly faster compared to what I am use to.


03a4e6  No.6086

Ok going through the back catalogue of shit to download/go through made me think.

Thread watcher only works for 4chan/8chan at the moment, and effectively it does the same job as the download page of images, I don't know what raw url and gallery do at the moment so I cant comment on them.

However, would there be a way to contextually filter what you want the program to do through dragging and dropping urls?

An example, drop a 4chan or 8chan link to the program pops up thread watcher with all you needing to do is press the start button, the 4chan archive hosts would default to page of image downloads along with a 4chan page if its seen to have 404'ed, honestly dont use 8chan enough to know how bumped off threads work here to know if that's an option or possible.

It also seems like sites like hentaifoundry could be smartly scrapped, I'm not 100% sure on that as help is a bit lacking/ui for it is a bit lacking, however lets assume that it works perfectly, something like

https://www.hentai-foundry.com/search/index?query=anal

could bring up a partially filled out section to either subscribe to it, or scrape it, one or the other, possibly a scrape with a subscribe at the same time.

Or from the same site

dropping a link like

https://www.hentai-foundry.com/user/Rampage0118/profile

would tell it to scrape + subscribe

Or, like in the help

https://gelbooru.com/index.php?page=post&s=list&tags=anal+pokemon+hetero

could see that it want gel, that it wants anal, pokemon, and hetero, will scrape it then subscribe.

With 4chan like sites with thread watcher function, possibly board specific rules, while /3/ will be fine if you checked it just every other week, a thread on /b/ needs to be checked every 5 minutes or it could 404. Per board settings with a global setting for how often it checks would be very helpful, along with a till 404 setting instead of finite check numbers. For some reason I got déjà vu writing this.


03a4e6  No.6087

have another thing for you to look at when it comes to downloading links.

https://archived.moe/h/thread/4343362/

This is from the /h/ archive, apparently every I didnt let this one go all the way as its 200+ links but the other page was fully failed because of this.

loading the page up in a different browser without scripts, I can clearly see the issue,

for example, this image here

https://archiveofsins.com/h/full_image/1464049607696.png

shows up as

https://archived.moe/h/redirect/1464049607696.png

With the script.

A potential solution would be keeping up with the archives and making workarounds for said archives as they change up their method of displaying images, or, allow the program to go down the rabbit hole by a user definable level.

lets say you have a website that has 1 link on it, that would be the surface, and what it currently does is go down one hole to find it, but what if that hole leads to a redirect page to the image, letting the user tell it to make 2 jumps would solve the problem,

for the sake of keeping up to date with archives and how they work, a user definable/importable script would be for the best long term, but for the immediate letting the program go further then one jump would solve the issue, Thankfully for me i only have 3 pages of links and I can go through them one at a time and not care too much, everything else is on desu archives and that's been pretty friendly so long as the images aren't lost.


04ddda  No.6088

One thing I'm really missing from the system is a way to force hydrus to forget a file's dupe status. Say if I reimport after months a file that I previously marked as dupe or whatever, or even a batch of them. There's no way to force hydrus to make me go through them again.

An option in the right click menu to tell hydrus to "reset this file's potential dupes" or something would save me.


03a4e6  No.6090

So i'm going through the boards on 4chan like normal for me, 3am ish looking for images and threads the pique my interest, and I realise something, if I say imported each thread that I wanted the images posted in it to hydrus, I would have well over 50 thread watching tabs

If there is a way to watch threads all in one thread watcher tab, like there is for download page of images that would be very helpful

something like this

[site] [board] [thread] [check time] [check amount] [time till check]

in a list it would be like this

[4] (as there are only two sites currently working thats all thats needed) [b]

[2183904712] [7m] [5m45s] [td] (till dead)

[8] [pone] [289896] [4h] [1h34m52s] [td]

[8] [monster] [191634] [4h] 3h21m42s] [td]

so on so forth, with text being black thread isn't done, red for it's done, and bold red for something is happening and program doesn't know. the bold one would be something like the website is down, or something unexpected happened and it doesn't know what to make of it.

if the watched thread is selected/highlighted, you should be able to edit parameters, such as the re check time, amount of times, a global pause, a stunted pause, and a priority (priority would be something like /b/ where the thread could 404 if not checked right away, and priority could be handled through check times) and when right clicked you can get the option to visit the thread it came from. pair this with a smart drag and drop function and it could be very powerful/useful.


03a4e6  No.6093

File: 1207b3a796bed6a⋯.png (659.69 KB, 698x840, 349:420, 1207b3a796bed6a910a5ea1514….png)

Another issue, I'm not 100$ sure there is anything you can do. One of the uses that I have for this is organizing images so I can later upload them or share them with friends, however some programs like discord can not take the image from the program itself.

However other programs/websites on the internet will take a drag and drop like its no problem, an example, I dragged and dropped from the program to upload this image.

However in the event I have to work around, I have to open the image externally, I use the win7 image viewer for everything it will handle, and everything it doesn't irfanview.

Now to get to the file explorer so I can either drag and drop a file take

open externally

show in folder

and then drag and drop from there

with irfanview or a webm, i'm not sure how do do that.

So an idea to work with those images/files is simple

2 context menu options,

for one many places online will allow you to copy a file location and work with that, for one.

for two, open file location, this would allow you to use the external file explorer to deal with it cutting out the open external middle man.

I say these two things because I have no idea if there is a way to make this program work with all programs, if this is just discord will only work with file explorer drag and drops, or if i'm a minority and everyone else can do this and i'm the odd man out.


04ddda  No.6094

>>6093

CTRL+C on a thumbnail will already copy the file path to the clipboard.

Discord's upload issue is not hydrus' fault, it's discord's.


03a4e6  No.6103

>>6094

Figured it as much, however one program not working with it means others will too,

Did not know about the copy paste was an option to get the path, with that said, with that said, looking into the context menu a bit more, most of the options beside show in folder are under the share menu, a menu that I have never touched in other programs due to how useless it is.

I can understand the reason they are under share in this one, but it did take till you told me the function is in the program to even look.


a09983  No.6104

>>6103

> however one program not working with it means others will too,

Discord is a special case due to their security/privilege protocols. In any case the dev can do nothing about it.


db6452  No.6125

File: a782401e3479358⋯.jpg (420.59 KB, 1147x1476, 1147:1476, a782401e3479358439ca13e5b4….jpg)

>>6081

Thanks–I think this is fixed in v259!

>>6083

>>6084

>>6085

Thank you for this information. I am currently thinking about these issues as well. There seems to be two issues:

Disk cache isn't currently tuned correctly.

Sometimes active memory usage explodes to 5GB or more, sometimes with CPU.

I believe the variable load times are due to the former and the crashes the latter.

I improved how media is loaded for 259. You may have noticed things loading faster. I will be revisiting the disk cache to make sure it loads the most important things first.

I am also going to look at the internal inter-gui messaging system, which I think may be the source of the memory explosions. I think this is aggravated by clients with many gui objects open, such as your many tags.

Please let me know if you discover anything new!

>>6086

>>6087

I think this is a great idea. The new downloader engine will be able to provide the correct tag parser for any given url, so I don't think it will be difficult to extend that to the url drag-and-drop event.

The new parser engine will be much better at following redirects and other links. This issue with the archive will be completely fixable, I think!

>>6088

I will be adding more verbs to the thumbnai menu dupe stuff to achieve this in the next few weeks! Please let me know how it works for you.




[Return][Go to top][Catalog][Nerve Center][Cancer][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / animu / htg / islam / leftpol / lewd / mde / vg / vichan ]