[ / / / / / / / / / / / / / ] [ dir / 4chon / cafechan / htg / hypno / leftpol / newbrit / strek / tijuana ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 1,181 items

Current big job: finishing login and domain managers and overhauling the downloader

YouTube embed. Click thumbnail to play.

bb1dc0 No.7247

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v282/Hydrus.Network.282.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v282/Hydrus.Network.282.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v282/Hydrus.Network.282.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v282/Hydrus.Network.282.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v282/Hydrus.Network.282.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v282.tar.gz

I had a great week. I fixed some things and added regex tag parsing to import folders.

flash fixed and debug exes

I accidentally broke flash embed windows for Windows users last week. It seems the new version of the program I use to make the Windows build was not linking something correctly, so many users were unable to play flash without a stream of errorspam, and some users could not even boot.

I have rolled back my PyInstaller to the old version and everything flash seems to be working again. I will be testing flash embeds every week from now on to make sure this doesn't slip through again.

Also: I think I have figured a way to include the debug executables in the regular builds. They are just called client_debug and server_debug. If you cannot boot the client in future, please try running the debug exe and screenshotting the additional error information it provides. (I'll write a .txt file in the base directory next week or so to explain this further)

faster thumbnails

This is just a little thing really, but I 'smoothed' out the thumbnail rendering pipeline a bit. The client should generate some thumbnails much faster, particularly when they are first viewed.

pixiv broken

It seems that Pixiv recentishly changed how their gallery pages work (they now load in a stream as you scroll down, via some javascript or json thing). Unfortunately, the existing hydrus pixiv parser cannot deal with this and was as a result not parsing these results (I think it was just getting a handful in the 'popular' row and then quitting).

I have hidden the pixiv downloader links in the client and paused any existing pixiv subscriptions. You'll also get a little popup about it on update.

I would like to say that the new hydrus downloading engine will be able to handle the new system, but I have not looked into it deep enough to be certain. I hope it can, and will look into it more when it finally comes time for us to rebuild the existing parsers in the new engine.

Pixiv do have an official json API, but it is an OAuth thing, which is an entirely different kettle of fish for our purposes.

EDIT: Someone just told me that artist pages are still using the old system, so I will revisit this next week to make sure and reactivate them if they are all ok.

filename tagging options

As has been long-planned, I have extracted the regex and other 'filename' tagging options you can set after the 'import files' dialog into their own object and applied them to import folders!

If you are into import folders or the regex parsing, please check it out under the manage import folders dialog–a new simple listctrl replaces the old '.txt parsing' button. You can't set 'tags for some files' or the '#'-based tags (because those require manual input), but you should be able to set anything else on a per-service basis.

I have tested this and it all seems to work correctly, but it was a complicated and semi-big rewrite, so please let me know if you run into trouble. Maybe try it out on a small example before you go full on!

string converter

This is an advanced thing for people interested in making scripts and downloaders with the parsing engine.

I realised this week that the login manager and downloader overhaul actually have a lot of overlap in the work that needs to be done, particluarly in the parsing engine. Rather than finish the login and domain managers first and then move on to the downloader overhaul, I have decided to work on all three at the same time, making small improvements as the new parsing engine can do more things.

So, this week, I finished a new 'StringConverter' class, that does simple string transformations–adding or removing text, encoding to hex, reversing, that sort of thing. I added it to the html formulas in the existing scripting system and expect it to put it in a couple of other places for when you are in the situation of, 'I can parse this data, but I don't need the first six characters.' I don't expect the existing file lookup scripts need this too much, but feel free to play around with it. I think I'll add a 'regex' transformation type in future.

I also added it to the 'file identifier' section of the initial file lookup request. Now, rather than the file's hash bytes being encoded to hex or base64 from the dropdown, it occurs through a string converter. There has been a persistent issue of wanting to generate an 'md5:[md5_hash]' argument, and I believe this will now fix it. I do not know the exact specifics here so I won't write the script myself, but I'm happy to work with people on the discord or email or whatever to figure out a solution here.

full list

- rolled back to an older version of pyinstaller that does not seem to have the embedded flash window problems windows users experienced

- added an error handler to wx.lib.flashwin import–if it fails to import, the client will print the error to the log and thereafter show 'open externally' buttons instead of the embedded flash window

- created a 'filename tagging options' object to handle the instance-non-specific auto-tagging parts of the filename tagging dialog

- moved all the appropriate tag generation code to this new object

- extracted the simple and advanced panels in the filename tagging dialog to a separate panel

- wrote a new wrapper panel to edit filename tagging options objects directly

- cleaned up some tag generation code–it'll now all be siblinged/parented/censored the same way

- import folders now use the filename tagging options object, harmonising this tag parsing code

- edit import folder gui now has an add/edit/delete listctrl to manage the new tag_service->filename_tagging_object relationship (this replaces the old .txt management button and summary text)

- finished the StringConverter class in the new parsing engine. it applies a series of transformations to text

- wrote a panel to edit string converters

- wrote a panel to edit string converters' individual transformations

- updated html formulas to use string converters instead of the old cull_and_add system

- html formula edit panels can now edit their new string converters

- file lookup scripts now 'stringconvert' their file identifier strings–this should allow the 'md5:md5_hash' fix we've been talking about (i.e. by first encoding to hex and then prepending 'md5:')

- the help->about window now shows the client's default temp directory

- thumbnail regeneration–particularly full-size regen and resized png gen–should be much faster in the client

- the debug exes are now included in the windows build

- the debug exes are now included in the non-windows builds

- the test exe is no longer included in the windows install (can't remember it ever being useful, and it is like 10MB now)

- unfortunately, a recentish change to how pixiv serves gallery page results has broken the hydrus pixiv downloader. this will have to wait for the downloader overhaul, and even then, it might be _slightly_ tricky. pixiv downloader entries are now hidden, and existing pixiv subscriptions will be paused on update

- the thumbnail select submenu now clarifies 'all (all in x)' if all are in the inbox or archive

- you can now archive/delete filter from the thumbnail menu on a single file

- lowered the http session inactivity timeout even more, to 45 minutes

- fixed a couple of instances of the new subscription dialogs being unable to mass-retry failures

- ffmpeg parsing errors now print more info about broken file path

- some daemons will be snappier about shutting down on application shutdown

- took out the sometimes invalid 'booting' phrase in the during disk cache population status report

- the client will now warn you about page instability at 165 pages after one user reported instability at 175. 200 is still the strict limit.

- downloader pages and subscriptions will fail more gracefully if their downloaders cannot be loaded (e.g. they used a since-deleted booru)

- fixed listctrl panel button enabled/disabled status when the child listctrl starts empty

- the new listctrl can now select data in a cleverer way

- misc fixes

- misc refactoring

next week

I want to get the 'StringMatch' object finished, and maybe the accompanying 'URLMatch' done as well. These will verify that strings and urls match certain rules, and will help with login and domain and downloader verification and veto states.

8844ed No.7248

>filename tagging options for import folders

thank you for this! i had to always import manually before because i used folder names for tags.


ba75a9 No.7251

I made an alternate Linux build for people having trouble with the official build. A few things are still weird, but overall it works better for me than the official. Run from source works slightly better, but pip is a headache. Built on Linux Mint 18, so it should also work on Ubuntu 16.04.

If you want open externally to work, you will have to manually set the open program for each mimetype in options->files and trash. For some reason, when Hydrus invokes xdg-open it opens a program in WINE.

https://www.mediafire.com/file/djtw4ftcu661m17/hydrus-282.tar.gz

I'll post one of these every week until the official Linux build works on my machine. If anything is broken, tell hydrus_dev (and thank him while you're at it). He was kind enough to give me his build command, but I don't know Python. If something is broken, I'm probably just as clueless as you are.


7930dc No.7252

wish i'd remembered to ask this last week…

can we get a "check all import folders" option? having to manually pick from the list when it's many folders is infuriating


11ae93 No.7255

>>7247

Three questions boss,

1. Is there any way to alter the vacuum behavior? I don't run it manually very often anymore (I'm the guy from >>6226 >>6274 >>6279) and it likes to run in the middle of an import or during subs checking. If it ran immediately after then that's cool, I just don't like the sudden freeze up in the middle of stuff while it runs. Particularly when I start typing a tag.

2. For the subscription merge, is there a way to keep the sub names? This is fine for tags on boorus but for pixiv you just have numbers that are sort of meaningless, and I'm expecting those to be the new name of the pop up (and therefore, tab) when it tells me "x files downloaded" from the sub. Not a big deal but it'd be helpful; I usually read the sub name so I know what artist to tag since artists don't tag their own name on pixiv all that often. Or at least if they do, it's not in English.

3. For the "dead" subs that haven't downloaded a file in 150+ days, is there any way to set some sort of manual refresh on that? As in, a way to recheck at 200 days, 300 days, etc. just to see if the artist came back to life? If that's a lot of effort to set up for low payoff, I can always check manually.


7930dc No.7256

bug: once we've created a new import folder tagging service and then fully left that set of menus, no further editing can be done. when i try to edit an import folder tagging service, i get "TypeError - unhashable type 'list'" and no edit dialog pops up


7930dc No.7258

i'm getting one cpu core maxed out for 5-10 minutes every time i run my import folders now - the whole program locks up for that time, cpu maxes on one core, and then after 5-10 mins it resumes working, showing a successful import. this is happening on a single folder with 8 jpg files in it, and is new behavior.


d6b9f7 No.7260

File: 5c61b2b7059a71d⋯.png (787.27 KB, 1183x665, 169:95, works on my machine™.png)

>pixiv broken

I just tried a pixiv download (artist id: 1116691) on v280 and didn't seem to have any problems.


11ae93 No.7262

>>7260

Same for 281 actually; I checked them before updating to 282 since they would be disabled. The only one that I think didn't work was a search


7c2306 No.7264

Haveing an issue with imports not happening, at least ones local ones, page of images has so issue with adding more images.


c1ae23 No.7265

>>7247

>- added an error handler to wx.lib.flashwin import–if it fails to import, the client will print the error to the log and thereafter show 'open externally' buttons instead of the embedded flash window

Traceback (most recent call last):
  File "client.pyw", line 20, in <module>
    from include import ClientController
  File "C:\Coding\python\hydrus\include\ClientController.py", line 23, in <module>
    import ClientGUI
  File "C:\Coding\python\hydrus\include\ClientGUI.py", line 11, in <module>
    import ClientGUIManagement
  File "C:\Coding\python\hydrus\include\ClientGUIManagement.py", line 14, in <module>
    import ClientGUICanvas
  File "C:\Coding\python\hydrus\include\ClientGUICanvas.py", line 15, in <module>
    import ClientGUIShortcuts
  File "C:\Coding\python\hydrus\include\ClientGUIShortcuts.py", line 8, in <module>
    import wx.lib.flashwin
  File "C:\Python27\Lib\site-packages\wx-3.0-msw\wx\lib\flashwin.py", line 20, in <module>
    cc.GetModule( ('{D27CDB6B-AE6D-11CF-96B8-444553540000}', 1, 0) )
  File "C:\Coding\python\hydrus\hydrus_env\lib\site-packages\comtypes\client\_generate.py", line 101, in GetModule
    tlib = comtypes.typeinfo.LoadRegTypeLib(comtypes.GUID(tlib[0]), *tlib[1:])
  File "C:\Coding\python\hydrus\hydrus_env\lib\site-packages\comtypes\typeinfo.py", line 478, in LoadRegTypeLib
    _oleaut32.LoadRegTypeLib(byref(GUID(guid)), wMajorVerNum, wMinorVerNum, lcid, byref(tlib))
  File "_ctypes/callproc.c", line 950, in GetResult
WindowsError: [Error -2147319779] Library not registered

Seems like a check was missing. I just duplicated the other check but I'm not 100% sure on what that specific function is used for so it might need different error handling.

--- a/include/ClientGUIShortcuts.py
+++ b/include/ClientGUIShortcuts.py
@@ -1,15 +1,27 @@
 import ClientGUICommon
 import HydrusConstants as HC
+import HydrusData
 import HydrusGlobals as HG
 import wx
 
 if HC.PLATFORM_WINDOWS:
     
-    import wx.lib.flashwin
-    
+    try:
+        
+        import wx.lib.flashwin
+        
+        FLASHWIN_OK = True
+        
+    except Exception as e:
+        
+        HydrusData.Print( 'Flashwin did not load OK:' )
+        
+        HydrusData.PrintException( e )
+        
+
 def IShouldCatchCharHook( evt_handler ):
     
-    if HC.PLATFORM_WINDOWS:
+    if HC.PLATFORM_WINDOWS and FLASHWIN_OK:
         
         window = wx.FindWindowAtPointer()


bb1dc0 No.7271

File: 3eb14f4a1d91329⋯.mp4 (4.61 MB, 1920x1080, 16:9, 3eb14f4a1d91329a7171df156c….mp4)

>>7252

Sure, no prob.

>>7255

1: What are your idle settings like in options->maintenance and processing? I would not typically expect a vacuum to kick in while you were using the program. I think it is only set up to happen after like an hour of idleness. You can turn vacuums off completely on the same panel, btw.

2: This is a good point. I will think about this. Adding another edit control to queries may be a bit finicky, so it might be more 'optional name' or 'display name' than I have elsewhere in the program. It might be good for over-the-shoulder privacy as well, to be able to 'rename' the raw query.

3: I recommend you increase the timing values under 'edit check timings' under the subscriptions dialog if you want to do this sort of thing. If you don't think a subscription is dead until it hasn't produced a file in 300 days, set it to '1 file per 300 days' and it'll essentially do what you are thinking of here. It does keep checking for new files, even if it hasn't downloaded in a while.

I'll be bumping the max time up to ten years btw, just to be on the safe side for super slow queries.

>>7256

Fugg, thank you for this report. I will make sure to fix this for next week.

>>7258

I am sorry you are seeing this. I'm afraid I do not get the same in my test system or my real one.

Are there any other files in the folder? Like ignored mimes or broken files?

Is the folder in an unusual location, like over a network?

Is your client very large, with very many tags and on a slow drive like a very full/fragged HDD?

Do you have some tags set up for these import folders? What sorts, if so? How many tags per file, typically?

What happens if you turn on db profile mode and then trigger an import folder? You can see about generating and reporting profiles at the bottom here:

http://hydrusnetwork.github.io/hydrus/help/reducing_lag.html

Do manual hard drive imports work ok?

>>7260

>>7262

Thanks, yeah, it looks like I messed up in only checking tags. I'll double-check this and then turn it back on for v283.

>>7264

I am sorry you are having a problem. What happens when you try to import local files? Do you get an error, or does it hang, or does it just do nothing? At which step in the process are you having the problem? If you check your client.log file in your file->open->database directory, are there any errors at the bottom corresponding with the approximate times you had the problems?

>>7265

Thank you. Can you at least boot ok now? It looks like the D27CDB6B-AE6D-11CF-96B8-444553540000 dll, which seems by a bit of searching to indeed be the 'flash dll', whatever that is called, is not actually registered in your registry. If you think you already have it installed, please try uninstalling and reinstalling it.

https://get.adobe.com/flashplayer/

Unless you have a problem installing it, in which case I'll try to figure out a better compromise here.

EDIT: After a bit of searching, it might be:

C:\Windows\System32\Macromed\Flash\Flash.ocx


c1ae23 No.7272

>>7271

>flash stuff

I just don't have flash installed (running win7x64). The error only occurs when running from source as running the windows version you provide doesn't crash.

I thought I'd just mention the other place having no flash can cause a crash.


11ae93 No.7274

File: f797120d604a29c⋯.png (70.01 KB, 845x843, 845:843, Untitled.png)

>>7271

>What are your idle settings like in options->maintenance and processing?

well this explains a lot. Not sure how those both got ticked but I'm guessing that should solve it. Changed them to 10 and 5 so I'll see how that works out.


be2191 No.7279

First of all, thanks for your work! I am on 282 now and made a fairly big jump coming from 269. The upgrade went fine and I like the improved UI responsiveness during local searches.

A problem that followed me with the upgrade is related to large subscriptions:

In particular I have one subscription that I find myself unable to continue, because it always fails during the URL fetching phase. The next time it retries the sub, it will restart from the beginning only to fail again at another point. Thus never making any progress.

As mentioned, it's a big subscription, so it's several hundred requests just to fetch URLs. I suspect the errors to be intermittent and would love to see the fetcher not bail out like it does now.

Maybe it could sleep for a few seconds and retry? Or show a prompt with the error message and choices to retry, abort, skip to the next page (in the unlikely case that a page is reproducibly broken)?

Maybe you have another idea or workaround.

Cheers!


debd11 No.7282

File: e70f3ea18be2b88⋯.jpg (136.65 KB, 1194x861, 398:287, e70f3ea18be2b888c3b4da77b1….jpg)

IO error when import folder tried importing during a first time connect with the public repo:

traceback:


IOError
[Errno 22] invalid mode ('wb') or filename: u'C:\\Exol\\import folder\\human
C:\\Exol\\import folder\\human
C:\\Exol\\import folder\\human
C:\\Exol\\import folder\\human\\tumblr_ohsi7jAouI1v69xp9o1_1280.jpg'
Traceback (most recent call last):
  File "include\HydrusPaths.py", line 475, in MergeFile
    shutil.move( source, dest )
  File "shutil.py", line 302, in move
  File "shutil.py", line 130, in copy2
  File "shutil.py", line 83, in copyfile
IOError: [Errno 22] invalid mode ('wb') or filename: u'C:\\Exol\\import folder\\humanC:\\Exol\\import folder\\humanC:\\Exol\\import folder\\humanC:\\Exol\\import folder\\human\\tumblr_ohsi7jAouI1v69xp9o1_1280.jpg'

attached is the file that failed. I must add that on fail I have it set up to move the file to the import folder again as there is no try again option.


debd11 No.7284

>>7282

I just saw that the file did not fail on import, although the file did not merge with the lower pixelsize version I have. That might happen on duplicate searching though. The client is now importing the public tag repo, so I'll search duplicates later. (>>7283)


bb1dc0 No.7287

File: 88fa4933bc008d4⋯.jpg (4.33 MB, 5009x6000, 5009:6000, 88fa4933bc008d4091fc8df979….jpg)

>>7272

Thanks, I missed this the first time around! I didn't realise I had it in a second location.

>>7284

>>7282

It looks like this failed in the step after the import, where the import folder deletes or moves files to the locations you set. That error suggests the path it has set up is:

C:\\Exol\\import folder\\human
C:\\Exol\\import folder\\human
C:\\Exol\\import folder\\human
C:\\Exol\\import folder\\human

(all one string)

It looks like somehow the good path has been duplicated several times with newlines. Can you please check your import folder and make sure the 'move to' location is good? Did you enter that path in the text box (maybe through several ctrl+v paste actions?) or did you enter it with the 'browse' button?

I do not recommend setting failed files to go back to the same location, as this will set up a loop of the file being retried and failing over and over, which just wastes resources. I recommend you either set them to be ignored or sent to a special folder where you can apply human brainpower to figure out what went wrong.

And yeah, files that are similar-looking but not byte-for-byte duplicates are dealt with in the duplicate system. The client isn't clever enough to know which ones you want to keep, so there's a whole filtering system to sort it all out. It is an advanced system, so if you are new to the program, you may want to put it off for a little while.


debd11 No.7288

File: f3c318cf80f2db4⋯.png (103.96 KB, 1918x1080, 959:540, qO82mJz[1].png)

>>7287

It looks like I indeed botched the logic there a bit. the folder name was in there more than once. I can confirm I copy pasted that, so I must have pressed V more than once.

>I do not recommend setting failed files to go back to the same location, (…)

I set this up, this sounds smarted indeed.

> And yeah, files that are similar-looking but not byte-for-byte duplicates are dealt with in the duplicate system. (…)

I'm familiar with the system, and it kinda did work, without needing to run the check myself I already found the two images, this and the lower quality one as a potential duplicate.

New problem popped up, I was fixing these things above and accidentally sorted the import folders by path, the client locked up until I only had the choice of opening my debugger in Visual Studio or closing the program. (pic related)

reloading the client via the close program option didn't show any apparent problems as all my tabs where still there, the things I was changing didn't save, but that wasn't a problem and was to be expected. Updating my two-day-old backup with 75% of the public repo in it now, as I really don't want to go through another download and install of all those rows. I hope you have enough info from this to fix the sorting by path crash.


bb1dc0 No.7330

>>7288

Great, I am glad this is working now. I'll add in a check-and-warning on dialog ok to make sure all the paths entered are valid.

I apologise for the sort-crash. The old listctrl object crashes on sort for some users. It isn't my code, and I don't know what is wrong with it. I wrote my own object and am slowly replacing all the listctrls across the program. Since you are affected by this, please refrain from sorting listctrls unless you know they work or you see in my changelog that I've updated them. The bigger ones, like manage subscriptions, are good, but there are a couple dozen or so smaller ones I still need to get to.



[Return][Go to top][Catalog][Nerve Center][Cancer][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / 4chon / cafechan / htg / hypno / leftpol / newbrit / strek / tijuana ]