Version 306

Name
Email
Subject
Comment *
File	Select/drop/paste files here
Password	(Randomized for file and post deletion; you may also set your own.)
Archive	Archive [500 char limit]
* = required field	[▶ Show post options & limits] Confused? See the FAQ.

Embed	(replaces files and can be used instead)
Voice recorder	Show voice recorder (the Stop button will be clickable 5 seconds after you press Record)
Options	Do not bump (you can also write sage in the email field)
Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf Max filesize is 16 MB. Max image dimensions are 15000 x 15000. You may upload 5 per post.

HookTube embed. Click on thumbnail to play.

Version 306 hydrus_dev ## Board Owner 05/09/18 (Wed) 21:29:49 ae28c2 No.8808

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v306/Hydrus.Network.306.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v306/Hydrus.Network.306.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v306/Hydrus.Network.306.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v306/Hydrus.Network.306.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v306/Hydrus.Network.306.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v306.tar.gz

I had a great but full week. There are many fixes and improvements to different systems.

import folder file presentation fixed

Last week's importer unification introduced a bug in how import folders publish their files to pages and popup buttons. The files imported ok, but the pages and buttons got a single invalid entry that would then stop the session saving to db correctly.

This problem is fixed. Import folders present to buttons and pages correctly again. I have also updated my session saving code to recover from bad pages more gracefully in future.

If you have any 'poisoned' sessions still open in v305 that refuse to save, please try closing any import-folder-related pages and try again. I apologise for the inconvenience here.

subscriptions can now publish to page

Like import folders, subscriptions can now publish their files to a page. If the page does not exist, it will be created (named after the sub), and if a so-named page already exists, the new files will be appended to it. You can also turn off the file popup buttons, and even turn off all publishing.

Furthermore, you can now tell subscriptions to publish each individual query as separate buttons or pages (named 'subscription_name: query_text').

This is all in the usual edit subscription dialog, as three new checkboxes.

file import status improvements

The file import status window (launched by clicking the icon button on any file download context) has a bunch of little improvements–its menus are a little neater, and some of the behind-the-scenes code is cleaner–and it can also now 'show files in a new page' on any new imports! Just select some 'successful/already in db/deleted' results that are newish–actioned maybe since v303 or so–and you'll get a new menu entry that lets you open the resulting files in a new page, and it even works from the manage subscriptions dialogs!

This is neat if you want to revisit some files you missed or had set not to publish to ui earlier. It'll happily show trashed/deleted files (the latter with blank thumbnails) as well, which is a little odd but still maybe useful so I haven't taken it out.

Some parser work

I have added a pixiv parser that pulls the raw japanese (i.e. non-romaji) tags in lieu of the available translations. It is not the default for pixiv, but if you would like to try it out, update the parser 'link' for 'pixiv file page' URLs to 'pixiv single file page parser - japanese tags' in the network->manage url class links dialog.

The 4chan thread parser has fixed 'if there is no subject, use the first line of the comments for page name' parsing.

Newgrounds have changed how their galleries work (it is all dynamic javascript stuff now), which has broken the old parser. Thankfully, I happen to be working on a new gallery parsing engine, so I hope to have this back in the coming weeks as the other gallery stuff rolls out. But for now, Newgrounds is removed from the downloader selection ui, and I recommend you pause and Newgrounds subscriptions.

I've also done some advanced stuff to improve how pixiv and twitter post urls are checked in the db for 'already in db/deleted' status, which is another step towards manga page (and other multi-file Post URL) support. The client's "have I seen this URL before" logic is becoming significantly more sophisticated, and it can now be less of a big deal if two files have the same URL.

I also discovered that pixiv are rolling out a new ui design for some (likely older) usernames. If your pixiv downloader suddenly stops working at all, it may be that you have been selected to beta this new design. The solution for now is to make a newer throwaway pixiv account and use that instead. I hope to have a proper fix in here (and for tag searches) as I roll out the gallery downloader overhaul work.

duplicate system can now sync urls

The duplicate filter merge options now have a dropdown choice to set if 'known urls' should be copied from 'worse to better' and 'both ways'! If you are into the duplicate system, please set these how you prefer–I recommend 'worse to better' for 'this file is better' and 'both ways' for 'both files are the same'.

I've also done some layout/presentation work on that options panel. In an attempt to make it less confusing, obscure merge choices are now disabled for common operations, but please let me know if your normal workflow is now affected. Weird stuff like 'delete both files' is still on for 'custom' actions.

open in web browser

All users will now see share->open->in web browser off the thumbnail right-click menu, which can be useful if you want to use a browser plug-in to do some IQDB-type source-lookup. Unfortunately, the command can be a little buggy unless you set a specific browser path at the new option under options->files and trash.

This new browser path option will be used for all link-launching in the client as well, so if you have had trouble with that, please set this up and let me know if it works better for you.

misc

All file queries that involve multiple tags or that involve at least one tag and simple system predicates now run much faster!

The new (x/y) import page name progress status will now update reliably on any change!

Advanced users can now test specific urls for url class recognition in the network->manage url classes dialog!

full list

- the file import status list now has 'open selected import files in a new page', which should show up where it is possible. this is a bit prototype and ugly–it'll show _all_ files, including in-trash and permanently deleted (which will show up with the hydrus thumbnail)

- the file import status list now prefixes the already in db/deleted notes with 'url' or the hash type that lead to the recognition

- these redundant/deleted notes now also propagate up from 'during import' recognition phase as well

- the 'delete seeds of type x' entries on the file import status button's right-click menu are now split into three smaller individual tyes and are more explicit about exactly which status types they will remove

- like import folders, subscriptions can now optionally publish their files to pages as well as popup buttons. also, subscriptions can optionally publish their files separately for each query instead of all merged together

- sped up multiple tag queries significantly

- sped up simple (file size, mime, etc…) system predicate queries that also include a tag/namespace/wildcard predicate significantly

- added a pixiv parser that pulls the japanese tags to the defaults–users can switch to this if they prefer under network->manage url class links

- fixed the 4chan parser to get part of comment as backup subject/page title

- removed the 'newgrounds' entry from the normal gallery page creation ui, as the basic gallery parser no longer works due to a dynamic loading change on their end. I hope to have it back with the new gallery parsing system I will soon be writing

- the edit url classes panel now has a little text box to put in example urls and see which class, if any, that they match to

- improved layout of edit url class links panel

- all url types are now displayable in the media viewer–only post url classes are default on

- the new (x/y) import page page_name progress count is now updated on all alterations to this value (previously, this was not updating when a user interacted with the import queue, only when the natural downloader loop cycled)

- added 'can produce multiple files' option to post url url classes, which informs client url-checking logic whether the url can be relied upon for 'already in db/deleted' calculations

- the pixiv file page url class now has 'can produce multiple files' checked, meaning some bad pixiv url association logic due to other sites referencing it as a source url is now fixed

- added a 'twitter tweet' url class, which is also a 'can produce multiple files' post url

- added a 'sync known urls?' action choice to the duplicate merge options panel, which governs whether urls should be copied from worse to better or in both directions

- gave the edit duplicate merge options panel a layout pass

- the edit duplicate merge options panel will now disable pointless/over-complicated choices on non-custom actions, let me know if this is a pain for your workflow

- added a 'manual' web browser path override to the 'files and trash' options panel, which fixes the new share->open->in web browser option for Windows and also fixes some #anchor link propagation

- consolidated all URL/Path web browser launching code to one location

- 'open in web browser' is now available for non-advanced_mode users and the 'open' submenu of the share menu is available in the preview window and the media viewer

- fixed a bug that was causing import folders to publish incorrect file identifiers, which was poisoning popup buttons and import page destinations

- gui sessions that fail to load a page will recover and continue to attempt loading the rest of their pages. some popups detailing the page's serialised data and error will be presented

- gui sessions that fail to save a page will recover and continue to attempt saving the rest of their pages. some popups detailsing the page's rough info and error will be presented

- the core controller inside all media pages will now present itself in a more beautiful way when asked to dump itself to a log (which should beautify the above save error a bit)

- wrote a subsidiary database->check->just repo update files that tests integrity of only repository update files

- fixed an issue where default tag import options were sometimes not being saved from the new dialog in the networking menu

- wrote a couple of layers of bad tag protection to help the new downloader deal with some occasional bad output from the old downloader

- network jobs can now reattempt connection attempts up to three times on POST requests (if you ever got inexplicable immediate 'could not connect' errors on repository uploads, this should now be fixed)

- replaced some archaic misc old import code with the new system, cleaning up a bunch of stuff and making space for further refactoring along the way

- fixed tags blacklist not being inherited in the old (through options dialog) system

- improved some invalid domain error handling

- fixed an animation update issue that would pause naturally updating controls on non-main-gui frames when there were no regular media pages open on the main gui

- added a BUGFIX option to 'files and trash' option page to override the default temp path for almost all client temp path requests

- the minimum value for the 'vacuum period' in maintenance and processing options is now 28 days. the control also has a little explanatory tooltip

- the 'try to auto-link url classes and parsers' function now always preferences parsers alphabetically

- fixed a typo in the string transformations prettyfication code that incorrectly summarised 'take the last x characters' as the opposite

- misc fix to file hash generation and status checking code

- the 'export tags to .txt files' checkbox on the export files panel will no longer bother you with a dialog as you uncheck it

- wrote some code to make it easier and more fool-proof to update the domain manager with new url classes and parsers on my end

- improved some popup message manager ok-to-alter-ui logic when the main ui is minimised and so on

- fixed some potential crash conditions (affecting linux mostly, seems like) in the service credential testing and access key fetching ui code

- fixed a bug when 'stopping' a gallery parse during a long error pause (like when it holds on '404')

- sped up some old set intersection code

- some import file presentation refactoring

- some url content application pipeline cleanup

- misc cleanup

next week

I got splatted with hydrus reports and work this week and didn't have time to knuckle down on a multi-thread watcher, so I'd like to get back to that.

In all the rush, I also fell way behind on replying to messages this week. If I can, I would like to catch up to everything on Saturday. If you are waiting on a reply, please hang in there.

____________________________

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/10/18 (Thu) 02:57:46 36ee3e No.8812

nice one :)

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/11/18 (Fri) 21:28:40 e89e7d No.8815

>>8808

Minor issue with pic related, file count is hidden by progress bar. Closing the dead subscription notice restored things to normal.

Also got another error during pixiv subscription download:

Exception
The subscription Pixiv encountered several errors when downloading files, so it abandoned its sync.
Traceback (most recent call last):
  File "include\ClientImporting.py", line 5276, in Sync
    self._WorkOnFiles( job_key )
  File "include\ClientImporting.py", line 4694, in _WorkOnFiles
    raise Exception( 'The subscription ' + self._name + ' encountered several errors when downloading files, so it abandoned its sync.' )
Exception: The subscription Pixiv encountered several errors when downloading files, so it abandoned its sync.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/11/18 (Fri) 22:20:34 c01b93 No.8816

on OS X getting problems with adding parent / sibling tag and being taken by default to remote tags. Same happens intermittently with pressing F3 on images.

Are there any plans to add .txt / .rtf support?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/12/18 (Sat) 00:00:10 e89e7d No.8817

File: 34df3a728dd9476⋯.jpg (24.51 KB, 432x167, 432:167, Untitled.jpg)

>>8815

Forgot image

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/12/18 (Sat) 07:18:09 5e7393 No.8818

Question: When you set duplicate relationships manually through the right-click menu, does that perform the predefined tag copying that happens on proper duplicate pages?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/12/18 (Sat) 08:32:07 ab167e No.8819

>>8816

I'm getting problems adding sibling too on windows. The old tagged images just won't display even though the tag shows the right number of images with that tag. I suck as explaining but for siblings I have it something like this

>creator:artist (15)

>artist (10) (will display as creator:artist)

I do a search for creator:artist and get creator:artist (25) in the tag tray

>only 15 images display

If that makes sense.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 05/12/18 (Sat) 19:44:08 ae28c2 No.8820

File: d00b0b267d1aba6⋯.jpg (48.02 KB, 700x482, 350:241, d00b0b267d1aba67400dc53a88….jpg)

>>8817

>>8815

Thanks, yeah, I noticed that bad layout as well. Popup manager isn't re-laying-out right on control unhide. I will have it fixed for next week!

If you go into your pixiv sub's error-hit query and look at its file import status window, can you find one of those errors and right-click->'copy notes' and then email me/post that here? If there error is about 'can't find file url' or 'couldn't parse anything', I think you got hit by this new dynamic javascript page presentation. I was talking with someone the other day and discovered this 'create a new throwaway account' is not 100% reliable. It may be Pixiv is assigning the new layout randomly on a per-login basis, not per-user account.

>>8816

Thanks, I got another report about this on OS X this week. I will check this out this week.

I'd like to add text file support at some point, but it'll need some new human-guided import ui as I can't reliably detect text file mime programmatically.

>>8818

Yeah, it'll do whatever is set in the 'default' merge options, which you should be able to double-check from the same thumbnail menu. The same default options are used for both that and the bigger duplicate filter.

>>8819

Thank you for this report. I understand the problem and will check it this week. Just to double-check, the files with just 'artist' are distinct from the 'creator:artist', right? Or could it be that ten of the fifteen files have both 'artist' and 'creator:artist' if you check in the manage tags dialog? Sometimes these counts get 'collapsed'–do you see (25) as the count, or (15-25)?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/12/18 (Sat) 20:52:13 e89e7d No.8821

>>8820

'NoneType' object has no attribute '__getitem__'… (Copy note to see full error)
Traceback (most recent call last):
  File "include\ClientImporting.py", line 4605, in _WorkOnFiles
    downloaded_tags = gallery.GetFileAndTags( temp_path, url )
  File "include\ClientDownloading.py", line 1644, in GetFileAndTags
    ( image_url, tags ) = self._GetFileURLAndTags( url )
  File "include\ClientDownloading.py", line 1632, in _GetFileURLAndTags
    return self._ParseImagePage( html, page_url )
  File "include\ClientDownloading.py", line 1597, in _ParseImagePage
    image_url = original_image[ 'data-src' ] # http://i3.pixiv.net/img-original/img/2014/01/25/19/21/56/41171994_p0.jpg
TypeError: 'NoneType' object has no attribute '__getitem__'

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 05/12/18 (Sat) 21:26:13 ae28c2 No.8822

>>8821

Aha, it looks like your client is falling back to using the old parser somehow. Please go to network->manage url class links and double-check your 'pixiv file page' url class has a pixiv parser connected to it. Clicking the 'try to fill in gaps…' button will try to do it automatically.

Once that is set up, you'll probably want to find these broken sources in the sub again and right-click->try again.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/13/18 (Sun) 02:50:10 59c774 No.8825

Hydrus crashes everytime I try to export a picture.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/13/18 (Sun) 09:18:28 ab167e No.8828

>>8820

>Just to double-check, the files with just 'artist' are distinct from the 'creator:artist', right? Or could it be that ten of the fifteen files have both 'artist' and 'creator:artist' if you check in the manage tags dialog?

I had to undo one of my siblings just to double check since I can't see the 'artist' tagged images if I did a search and there was something I fail to notice in my last post. Basically I get 2 tags showing when I type the name in the auto search tray.

(Still using the example btw)

>artist (10-25)

and

>creator:artist (15)

I open 2 tabs with both of those tags. The 'creator:artist' tab shows the number of images (15) while the 'artist' tag shows all 25 images(or a bit less since some of the images share both tags).

I know both these siblings were paired together before I updated Hydrus but when I tried to pair them again the other day from my last post, thinking I had forgot to pair them, the siblings list never showed them as being paired before as if I lost the pairings or something. So when I tried to pair them I just got 'creator:artist (15-25)' with only 15 images to display.

>do you see (25) as the count, or (15-25)?

Yeah that was another thing I faild to mention in my last post. It was something like 'creator:artist (15-25)'. So it seems like Hydrus knows they are paired and yet it doesn't for some reason.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/13/18 (Sun) 22:00:52 e89e7d No.8830

File: f4bd302253efa7c⋯.jpg (249.49 KB, 1440x762, 240:127, Untitled.jpg)

>>8822

"add defaults" just makes an exact duplicate of the existing pixiv listings

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 05/13/18 (Sun) 22:53:18 ae28c2 No.8831

File: 76a71ccca882424⋯.png (39.05 KB, 695x512, 695:512, manage url class links.png)

>>8825

I am sorry to hear you are having trouble. What OS are you on, and which release are you using (or running from source?)? Do you have a Nvidia GPU? What happens if you import a non-visual file format like a PDF or mp3?

>>8830

There's another dialog with a similar name, manage url class links (image related). This will actually connect (link) those url class definitions to the parsers. I think yours got disconnected somehow from the pixiv parser.

>>8828

Thanks, I will check this out.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/13/18 (Sun) 23:19:35 e89e7d No.8832

>>8831

>>8830

Got it. I updated it to the japanese tags version like in your pic, but it seems I'm still getting those errors. Some did download, but I don't know if those were from the same files that failed previously, or if artists had added new files since then.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/14/18 (Mon) 06:11:40 59c774 No.8833

>>8831

>I am sorry to hear you are having trouble. What OS are you on, and which release are you using (or running from source?)? Do you have a Nvidia GPU? What happens if you import a non-visual file format like a PDF or mp3?

Arch Linux.

Didn't compiled it from the source code, I'm using the Linux executable.

Looks like it only crashes with a very specific picture, every time I tried to export it hydrus just crashes.

Intel Graphics GPU.

Mesa drivers.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/14/18 (Mon) 19:33:52 59c774 No.8835

>>8833

Update.

Hydrus has been very unstable recently, but with this version it has been crashing for almost no reason, I'm trying to put some tags, it crashes, scrolling some pictures, it crashes, now it get stuck trying to load all my pictures.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

Anonymous 05/15/18 (Tue) 00:12:32 0e85ce No.8837

>>8835

works perfectly on my machine

windows pleb here.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

hydrus_dev ## Board Owner 05/19/18 (Sat) 19:27:47 ae28c2 No.8868

>>8835

>>8833

Thanks. I am sorry you are having problems. I am still working on this Linux instability, so I hope this situation improves in the future.

I have specifically tried to reproduce the 'crash on export files dialog' issue several times in Linux but never been able to do it. I recently changed it a bit and cleaned up its code, but if you are still getting crashes, I'll give it another look.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.