/hydrus/ - Hydrus Network

>>4726

Yes

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v236/Hydrus.Network.236.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v236/Hydrus.Network.236.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v236/Hydrus.Network.236.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v236/Hydrus.Network.236.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v236/Hydrus.Network.236.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v236.tar.gz

I had a tough week, but I got a lot done. I did a bunch of bug fixes and other background work, and I overhauled the manage subscriptions dialog.

https network upgrade

tl;dr: Hydrus network is getting more secure, you don't have to do anything except update today or in the next few weeks.

A user and I were talking about the network when it came up that hydrus is still running on http, and hence not yet encrypted. Moving to a fully encrypted protocol has always been an important long-term goal, but absent a complete overhaul of the network, there aren't any really excellent solutions. The current network code is a mess of overlapping iterations, so adding robust encryption got a lost in the mix of everything else I wanted to clean up first.

The user mentioned it was nonetheless very important for them to be on some sort of encrypted communication, and after discussing it more, I agreed to jump on this and get something reasonable out rather than waiting for a 'perfect' solution.

So, I will be moving the current http system to self-signed https. This is not very beautiful, but it is decentralised and easy and free to implement and will completely obscure the contents of all hydrus network traffic to any typical outside observers. I can also improve on it in future.

Unfortunately, because of the way hydrus works, I can't simply run http and https in parallel, like a website might, so upgrading is slightly more complicated. I have a working prototype, but I do not want to flip the switch today, without any warning, because it will break anyone who does not update immediately. The change is significant enough that an old client cannot even detect what is wrong–so it won't give a 'you need to update to talk to this server' error, it'll give a deeper and uglier network-level exception.

So, this week I have updated only the client. When talking to a hydrus server, it will attempt http and then upgrade to https if required. In three weeks, on v239, I will update the server code to be https only and update my PTR at the same time, and then three weeks after that, on v242, I will make the client only ever try https. This means that anyone who updates their client within the next three weeks will see no service disruption from any server that updates in the subsequent three weeks.

I hope this update window ends up working out. I can appreciate how important encryption is to some people, even for the relatively stale information going across the hydrus network–I apologise, I should have put work into this sooner.

manage subscriptions overhaul

More immediately, I have overhauled the manage subscriptions dialog as I have discussed with many users. I thought this would be too large a job to fit into my normal weekly cycle, but it came up it my list again and I thought it through and figured it might not be so difficult with the other recent window updates. I scheduled a bunch of extra hours to work on it and just got it done.

So, manage subs now works on the new sizing system, and it looks more like the new manage scripts dialog. All your subscriptions will be summarised in a big list, and you edit their fine details on a separate window. You can also do some mass-editing–selecting multiple subscriptions and telling them all to pause or resume, or retry all their failed files (which are listed in the summary), for instance.

You can now also import and export subscriptions! You can put them on your clipboard or dump them to png files, just like scripts.

Furthermore, you can now export or import multiple subscriptions or scripts at once (i.e. to/from the same single clipboard or png)! If you want to backup all your subs, or transfer them to a different client, it is now easy and takes no time at all.

If you have a lot of subs, try exporting them all to a single png and then looking at the several hundred KB of encoded mess below the descriptive text–it's pretty cool!

middle-click to open new page

This is only a small thing, but it is neat: Middle-clicking on the greyspace beside your main gui's notebook tabs will now pop up the page chooser dialog.

future dupe plan

I had a long think about a workflow for 'hey hydrus client, show me whatever dupes I have so I can figure out what to do with them', and I am feeling good about getting started on it. I have to do some db stuff to find dupe pairs (much easier, now we have the faster search!), and create some gui stuff to actually display them and let you decide what to do, and then I need to create some new dupe metadata.

For this first version, I will present duplicate pairs for judgment let you filter them into three broad classes:

- they are not a dupe

- they are the same image (but different resolution, quality, or one has a watermark, say)

- they have a family relationship (alternate colours/versions/costumes, clean/messy, and so on)

I will write further gui to deal with 'they are the same image' pairs, to let you delete one if you want and merge tags and ratings across either way.

Family relationship is a much more complicated problem, however, and deserves to be its own thing on the next 'big thing to work on next' poll. Family relationships often form complicated groups with sort order and hierarchies, and dealing with that is more work than I can tuck into the 'faster dupe search' job. I can however record the family relationship pairs you discover and store them in the db until this next round comes up.

full list

- in prep for a network https upgrade, the client can now detect and escalate to https when making connections to hydrus services

- import/export to png and clipboard now supports multiple objects at once!

- rewrote the manage subscriptions dialog to work on the new panel system

- the new manage subscriptions dialog has a listctrl and a sub edit dialog

- the new manage subscriptions dialog has the same add/export/import/dupe/edit/delete buttons as the manage scripts dialog

- subscriptions are now importable/exportable, including en masse with the new multiple object import/export support!

- the new manage subscriptions dialog has retry failed/pause-resume/check now/reset buttons for easy mass subs management

- the edit subscription panel has a bit of a layout makeover

- the edit subscription panel now updates itself as its buttons are hit

- the edit subscription panel disables buttons that are not applicable

- subscriptions can now be renamed!

- cleaned some misc subscription code

- relabelled initial and periodic file limit in the subscription edit panel

- middle-clicking on the main gui's greyspace (e.g. to the right of the notebook tabs) will spawn the new page chooser!

- created a simple HydrusRatingArchive class–will do more with it in future

- added ffmpeg, python, and sqlite versions to the help->about window

- harmonised daemon code

- added a new class of daemon that will not fire while a session load is occuring

- subscriptions, import and export folders, and file repo downloads now use this new daemon

- cleaned the way background daemons check for idle

- expand/collapse panels now notify the new kind of toplevelwindow that a resize may be needed when they switch state

- time deltas (like on subs edit panel or a thread watcher) now render more concisely ('7 days' instead of '7 days 0 hours')

- serialisable object png export panel now has a width parameter

- fixed a bug where tags that begin with unicode digits were accidentally identifying as numbers for the purposes of sorting and throwing errors on convert fail

- the media viewer can handle some more unusual content update combinations–for instance, if it cannot figure out which media to show next, it will revert back to the first image rather than displaying an undefined null mess

- updated and cleaned a bunch of my old misc encryption code

- misc cleanup

next week

I'll be starting this dupe search stuff. I also started a HydrusRatingArchive this week for future easy rating migration, so I might do a bit more on that.

Getting a weird issue of images having tags but not showing up as tagged in the grid as having them, just happened on a batch download i made after upgrading

>>4733

>HydrusRatingArchive this week for future easy rating migration

Can you explain more what this is or how it will work? It sounds like something that will replace the current rating system.

Noticing that the performance of similar image search hasn't improved at my end over the last two versions. Seeing about 15 seconds to search (db locked / Loading.. in the status bar) with 373k files in the database.

I have a problem with rule34@booru.org. The error appears after a while, after I try to download something. It does not depend on the object that has been downloaded, so if I start to download again, everything is downloaded, but stuck on a another picture.

UnicodeDecodeError
'ascii' codec can't decode byte 0xcf in position 2343: ordinal not in range(128)
Traceback (most recent call last):
  File "include\ClientImporting.py", line 424, in _THREADWork
    self._WorkOnFiles( page_key )
  File "include\ClientImporting.py", line 255, in _WorkOnFiles
    self._seed_cache.UpdateSeedStatus( url, status, exception = e )
  File "include\ClientImporting.py", line 2009, in UpdateSeedStatus
    note += traceback.format_exc()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 2343: ordinal not in range(128)

>>4731

Sounds good. Will there be a dupe PTR of sorts so that this dupe info can be shared?

Also could Hydrus mark what it thinks is the "best" version based on resolution, quality, size etc. Would probably speed up the process in most cases.

>>4730

>This is only a small thing, but it is neat: Middle-clicking on the greyspace beside your main gui's notebook tabs will now pop up the page chooser dialog.

Nice.

>>4754

>re: a ptr for dupes

Hydrus automatically deleting images based on what others think is bad quality is not good, I agree. But could this not be used as a suggestion instead? Although perhaps it would be more trouble to implement than it's worth, compared to a simple suggestion by Hydrus based on file properties.

Sharing family relationships via the PTR would be very useful though.

Hey dev, do you know of or have any plans to include support for Fuskator?

"Page of images" detects the pictures on a photoset page, but then fails with:

MimeException: Filetype is not permitted!… (Copy note to see full error)
Traceback (most recent call last):
  File "include\ClientImporting.py", line 1351, in _WorkOnFiles
    ( status, hash ) = client_files_manager.ImportFile( temp_path, import_file_options = self._import_file_options, url = file_url )
  File "include\ClientCaches.py", line 878, in ImportFile
    return self._controller.WriteSynchronous( 'import_file', *args, **kwargs )
  File "include\HydrusController.py", line 348, in WriteSynchronous
    return self._Write( action, HC.LOW_PRIORITY, True, *args, **kwargs )
  File "include\HydrusController.py", line 113, in _Write
    result = self._db.Write( action, priority, synchronous, *args, **kwargs )
  File "include\HydrusDB.py", line 696, in Write
    if synchronous: return job.GetResult()
  File "include\HydrusData.py", line 1922, in GetResult
    raise e
DBException: MimeException: Filetype is not permitted!
Database Traceback (most recent call last):
  File "include\HydrusDB.py", line 469, in _ProcessJob
    elif job_type in ( 'write' ): result = self._Write( action, *args, **kwargs )
  File "include\ClientDB.py", line 8697, in _Write
    elif action == 'import_file': result = self._ImportFile( *args, **kwargs )
  File "include\ClientDB.py", line 5327, in _ImportFile
    ( size, mime, width, height, duration, num_frames, num_words ) = HydrusFileHandling.GetFileInfo( temp_path )
  File "include\HydrusFileHandling.py", line 166, in GetFileInfo
    if mime not in HC.ALLOWED_MIMES: raise HydrusExceptions.MimeException( 'Filetype is not permitted!' )
MimeException: Filetype is not permitted!

But they seem to be normal JPGs.

The HTML structure is like this:

<body>
  <form id="aspnetForm" method="post" action="/full/blah/blah-blah.html"> /* the path from website main directory to current page */
    <div id"main">
      <div class="hreview-aggregate">
        <div>
          <div class="imagelinks">
            <a href="#2"> /* always the bookmark numbered one more than the current image - clicking an image jumps to next image */
              <img id="i1" alt="blah-blah-1" ondragstart="return false" onselectstart="return false" onload="autoSizeImage('i1', 752, 1155);" src="//i6.fuskator.com/large/blah/blah-blah-1.jpg" width="752" height="1155">

I'm thinking maybe the bookmark links are messing it up or something? Or Hydrus doesn't recognize the type of jpg encoding they use?

>>4771

Sweet. I figured the links were messing it up because it was looking for a full image. Will eagerly anticipate the engine scripting bit.

Once you get that finished, you should consider stickying a thread for people to share all of the various things you can import and export as images in.

Also, if possible, a way to bake delays between each file download straight into each script would be handy for avoiding bans. On that same note, some sort of scripting socket to a basic rand generator could allow us to avoid downloading at even milliseconds and getting autobanned for that.

I have a request for the parsing scripts. It would be neat if you could make it so link nodes only trigger if a certain string is found in the link address. That way you could have multiple nodes for different websites on the same parsing script, and you could for example do an iqbd search on all websites available on that site at once instead of a specific one at a time, which would save a lot of bandwidth.