[ / / / / / / / / / / / / / ] [ dir / hikki / leftpol / newbrit / strek / sw / tijuana / u / yg ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.


New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 1,181 items

Current big job: finishing login and domain managers and overhauling the downloader


YouTube embed. Click thumbnail to play.

221dc2 No.7167

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v281/Hydrus.Network.281.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v281/Hydrus.Network.281.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v281/Hydrus.Network.281.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v281/Hydrus.Network.281.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v281/Hydrus.Network.281.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v281.tar.gz

I had a great week. There is an important update to subscriptions this week.

subscriptions

Subscriptions now support multiple queries! This is an important and large change, so if you use subscriptions a lot, please check it out.

Essentially, you can now wrap all your different subscriptions for a particular site into one single object that has the same tag parsing options and so on. Adding a new query is as simple as just adding a new row, rather than setting up a whole new subscription every time, and editing tag options for many queues is now just a single action. The ui for manage subscriptions has changed as a result, but if you understand the basic new concept, I hope it is mostly obvious.

I also wrote tools to merge and separate existing subscriptions. They are buttons in the manage subscriptions dialog. I recommend all users with subscriptions try merging them on update to make it easier to keep track of things. Just select a bunch of subscriptions that use the same site and hit 'merge'. If it all goes wrong, hit cancel and let me know. Otherwise, please read the text notes as you go and double-check that the newly merged subscriptions have an appropriate new name and good tag settings and so on.

In switching over, I also changed it so subscriptions use the same new timing check system as thread watchers! Instead of a static 'check period', subscriptions will now check their individual queries more or less frequently depending on how fast they seem to be generating files. And if a query slows too much, it will be considered 'dead' and will no longer be checked. This information is listed in the ui for quick reference. On updating, I discovered I had about forty dead subs that hadn't produced anything for like ten months.

Your existing subscriptions will be updated to the new system automatically, with new check timing options populated based on the old period you had set. Please check your subscriptions to make sure you are ok with the new settings. I think the default is '5 files per check', '<1 file per 90 days', '1 day min', '90 day max', but feel free to change it however you like.

This rewrite was a lot of work, so I expect there are some rough edges here and there. The subscription popup message is a bit ugly and doesn't report the current query well, so I will work on that first, but if you notice any other issues, please report them. If anything goes really wrong, remember you can hit services->pause->subscriptions to hit the emergency brakes.

broken image support

Images that previous gave 'truncated image' errors should now import ok, albeit with similar rendering errors (typically white/grey bars or patches of low quality). They should also have decent thumbnails, but please send me any examples that come out either all black or with a colour-shifted palette.

In similar news, all thumbnails in the client are now generated with the faster image library. Everything seems great so far, but let me know otherwise (its default colourspace is BGR, not RGB, wew).

other highlights

Tumblr 'video' posts (where it is just the video, not a more complicated post with a video embed) are now supported! A user helpfully sent me a good example in the week and I figured out a way to parse it. It seems to work ok, but I am interested in examples where it doesn't.

You can now edit your favourite regexes under the options dialog.

The splash screen is now a little taller and wider, and it reports a few more things during boot and exit. If you get a hanging daemon on shutdown or a very slow boot, we should now have a bit more info to work with. On first glance, loading tag siblings and parents seems to be a bit of a CPU hog, so I'll looking into that.

The 'import files' dialog that spawns when you drop some files on the client now lets you interact with the client while it is open! You can leave it open doing a large parse job and still browse the main gui. You can even have several open at once!

full list

- subscriptions can now support multiple queries!

- subscriptions now use the new thread checker system to set their check periods!

- improved some check timing calculations to deal with subscription timestamps

- existing subscriptions will update

- subscriptions can now see any 'delay' reason on their edit page and can scrub it there as well

- subscriptions no longer track 'last error'. everything goes through the delay system and keeps track of reason

- added 'last new file time' to the subscription listctrls

- subscriptions now generate separate 'file button' popups for each query, and file button popups now include their name in the button label

- the subscription listctrl is now breddy wide. I will be adding some way of hiding columns (with persistent memory) on the new listctrl at some point in the future, so please bear with it for now

- manage subs now has a wrapper panel and buttons that disable when invalid

- tumblr simple 'video' posts should now be supported! turns out they are just mp4s

- in the client, static image thumbnails are now generated with opencv

- files that give truncated image errors should now import ok. they'll render with errors. if cv is up to date and working ok, they should have good thumbnails as well (otherwise they will have all-black thumbs)

- updated manage regex favourites dialog to the new panel system

- regex favourites are now editable under the new options->regex page

- ongoing import or export folders will now stop working if their respective manage dialogs are opened

- import folders can now stop working if the client shuts down during the initial file parse phase

- export folders will now stop working immediately if the client shuts down

- export folders can now stop working during the file copy stage

- the splash screen is now a little wider and taller

- the splash screen now has a subtext row for misc technical data

- the boot and exit splash screens show a bit more data about several things

- the repository sync daemons should be a bit snappier about responding to a client shutdown

- the import files dialog is now a non-modal frame! (you can now continue to use the client while this frame parses files) you can even open up several at once!

- the 'import file path tagging' dialog should now never be taller than its spawning screen and also slide up and left if it spawns with its bottom-right corner off-screen (this dialog is still on the old system, however, so the newer sizing stuff that does this stuff automatically is not yet available)

- sliding up-and-left should now also occur on spawn for frames on the new system

- the advanced 'process now' repository button will now disable itself if there is no work to do!

- if simple downloads like gallery pages are larger than 100MB, the client will throw an error (this is a stopgap for an ongoing issue)

- the 'maintenance and processing' page has a bit more warning text

- the new listctrl wrapper panel now supports menu buttons

- libpng's iccp terminal warnings from loading the flash/pdf/zip thumbnails should now be gone!

- misc refactoring

next week

Back to the login system.

8713f4 No.7168

I have noticed this recently.

Prior versions of the program would just lock the fuck up till everything was loaded, now once the pages are loaded, the program is pretty free to do its own thing

Is there a way to have a global 'loading initial files' gui element like there is for individual pages?

the program may be able to do shit, but it's a bit useless till everything is loaded up and it would be nice to see that instead of needing to find the biggest window.


f2dcb2 No.7169

Client doesn't boot at all after updating. No errors, no logs.


f2dcb2 No.7170

>>7169

Should add I'm on Win10


069dcf No.7171

>Not yet. Is some of R34H's content locked off if you are not logged in?

Yes,you need to be logged in to view 'rating=e' content.


c7f12e No.7172


0ceadc No.7173

YouTube embed. Click thumbnail to play.

I made an alternate Linux build for people having trouble with the official build. A few things are still weird, but overall it works better for me than the official. Run from source works slightly better, but pip is a headache. Built on Linux Mint 18, so it should also work on Ubuntu 16.04.

If you want open externally to work, you will have to manually set the open program for each mimetype in options->files and trash. For some reason, when Hydrus invokes xdg-open it opens a program in WINE.

http://www.mediafire.com/file/62vl7hahht3lazf/hydrus-281.tar.gz

I'll post one of these every week until the official Linux build works on my machine. If anything is broken, tell hydrus_dev (and thank him while you're at it). He was kind enough to give me his build command, but I don't know Python. If something is broken, I'm probably just as clueless as you are.


000000 No.7174

Hey Dev, just updated to the new version and so far everything was working until I tried to import files and it would be stuck on parsing. So I restarted the client and now am getting this error:


hydrus-client
/usr/lib/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py:16629: UserWarning: wxPython/wxWidgets release number mismatch
warnings.warn("wxPython/wxWidgets release number mismatch")
WARNING:root:pafy: youtube-dl not found; falling back to internal backend. This is not as well maintained as the youtube-dl backend. To hide this message, set the environmental variable PAFY_BACKEND to "internal".
2017/11/09 04:57:57: hydrus client started
Gtk-Message: (for origin information, set GTK_DEBUG): failed to retrieve property `gtk-primary-button-warps-slider' of type `gboolean' from rc file value "((GString*) 0x5578ce413c40)" of type `gboolean'
2017/11/09 04:57:57: booting controller...
2017/11/09 04:57:57: booting db...
2017/11/09 04:57:59: preparing db caches
2017/11/09 04:57:59: initialising managers
2017/11/09 04:58:01: CallBlockingToWx just caught this error:
2017/11/09 04:58:01: Traceback (most recent call last):
File "/opt/hydrus/include/ClientController.py", line 119, in wx_code
result = func( *args, **kwargs )
File "/opt/hydrus/include/ClientController.py", line 650, in wx_code
self._caches[ 'thumbnail' ] = ClientCaches.ThumbnailCache( self )
File "/opt/hydrus/include/ClientCaches.py", line 1723, in __init__
self.Clear()
File "/opt/hydrus/include/ClientCaches.py", line 1869, in Clear
thumbnail = HydrusFileHandling.GenerateThumbnailFromStaticImage( path, options[ 'thumbnail_dimensions' ] )
File "/opt/hydrus/include/ClientImageHandling.py", line 246, in GenerateThumbnailFromStaticImageCV
mime = HydrusFileHandling.GetMime( path )
File "/opt/hydrus/include/HydrusFileHandling.py", line 292, in GetMime
if HydrusVideoHandling.HasVideoStream( path ):
File "/opt/hydrus/include/HydrusVideoHandling.py", line 367, in HasVideoStream
lines = GetFFMPEGInfoLines( path )
File "/opt/hydrus/include/HydrusVideoHandling.py", line 173, in GetFFMPEGInfoLines
CheckFFMPEGError( lines )
File "/opt/hydrus/include/HydrusVideoHandling.py", line 34, in CheckFFMPEGError
raise IOError( "File not found!" )
IOError: File not found!

2017/11/09 04:58:01: A serious error occured while trying to start the program. Its traceback will be shown next. It should have also been written to client.log.
2017/11/09 04:58:01: Traceback (most recent call last):
File "/opt/hydrus/include/ClientController.py", line 1262, in THREADBootEverything
self.InitModel()
File "/opt/hydrus/include/ClientController.py", line 657, in InitModel
self.CallBlockingToWx( wx_code )
File "/opt/hydrus/include/ClientController.py", line 169, in CallBlockingToWx
raise error
IOError: File not found!

Segmentation fault (core dumped)

Output of log:


2017/11/09 04:53:06: hydrus client started
2017/11/09 04:53:06: booting controller…
2017/11/09 04:53:06: booting db…
2017/11/09 04:53:07: preparing db caches
2017/11/09 04:53:08: initialising managers
2017/11/09 04:53:10: CallBlockingToWx just caught this error:
2017/11/09 04:53:10: Traceback (most recent call last):
File "/opt/hydrus/include/ClientController.py", line 119, in wx_code
result = func( *args, **kwargs )
File "/opt/hydrus/include/ClientController.py", line 650, in wx_code
self._caches[ 'thumbnail' ] = ClientCaches.ThumbnailCache( self )
File "/opt/hydrus/include/ClientCaches.py", line 1723, in __init__
self.Clear()
File "/opt/hydrus/include/ClientCaches.py", line 1869, in Clear
thumbnail = HydrusFileHandling.GenerateThumbnailFromStaticImage( path, options[ 'thumbnail_dimensions' ] )
File "/opt/hydrus/include/ClientImageHandling.py", line 246, in GenerateThumbnailFromStaticImageCV
mime = HydrusFileHandling.GetMime( path )
File "/opt/hydrus/include/HydrusFileHandling.py", line 292, in GetMime
if HydrusVideoHandling.HasVideoStream( path ):
File "/opt/hydrus/include/HydrusVideoHandling.py", line 367, in HasVideoStream
lines = GetFFMPEGInfoLines( path )
File "/opt/hydrus/include/HydrusVideoHandling.py", line 173, in GetFFMPEGInfoLines
CheckFFMPEGError( lines )
File "/opt/hydrus/include/HydrusVideoHandling.py", line 34, in CheckFFMPEGError
raise IOError( "File not found!" )
IOError: File not found!

Running Antergoes (Arch), DB and files are on NFS


c02b86 No.7175

File: fbc2f76b94145cc⋯.png (1.11 MB, 1920x1040, 24:13, client_2017-11-09_17-09-31.png)

File: b53fa0b1eea1299⋯.png (121.44 KB, 1920x1040, 24:13, client_2017-11-09_17-11-56.png)

File: a46c790f80166be⋯.png (597.85 KB, 1921x880, 1921:880, 2017-11-09_17-13-36.png)

File: a5c30c82707b2dc⋯.png (977.55 KB, 1920x1040, 24:13, client_2017-11-09_17-17-05.png)

there seems to be a problem with some numbers, or I'm just being stupid.

it says 31 files are there, but the 32 files have been imported, am I just dumb?

after importing the whole folder again, the fourth pic applies:


221dc2 No.7176

File: a765b8fbb5e6c52⋯.webm (813.89 KB, 720x720, 1:1, a765b8fbb5e6c5231df8c291a….webm)

>>7168

I am sorry, but I do not completely understand. Would you like the client to delay the initial display while it loads? Say, keeping the splash screen up (with some 'loading initial session' text) until everything is ready to go? Or something else?

>>7169

>>7170

Please try this debug release:

https://www.mediafire.com/file/rbydsxdzsfygh0s/Hydrus%20Network%20281%20debug%20-%20Windows%20-%20Extract%20only.zip

Extract it somewhere, shift+right-click->open command window here, then type 'client' (or '.\client' if your Windows opens a powershell here) and hit enter. It will print a lot of load debug stuff and hopefully an error. Please screenshot that error and post it here or mail it to me!

>>7171

Thanks. The login manager isn't ready, but I should think it will fix this.

>>7172

Thank you very much. I will check these out when I have a bit of time.

>>7174

I am sorry you are having problems. I am not sure what is going on here. That error means that a png in the 'static' directory (like the default thumb that appears for pdf files) seems not to exist. The section of code is running because of an oversight in the new thumbnail rendering that I will fix for next week.

Please check your install_dir/static folder with a fresh extract. Are any 'flash.png'-like files missing?

If you try to run the fresh extract (creating a new database), does it run ok, or does it give the same error?

Is there anything unusual about your ffmpeg install, do you know? If you get one of the ffmpeg static builds here:

https://www.johnvansickle.com/ffmpeg/

And extract the straight ffmpeg executable into your install_dir/bin folder, do you get anything different?

>>7175

Thank you for this report. I suspect that 001_LeyP6Vc.jpg and 017_HQ4dvBk.jpg (the first of both rows in your third image) are duplicates. In hydrus, these two sources would have been merged to the same file/thumbnail/destination. If you just try importing just this pair again, I think it'll probably just give you one thumbnail with two 'already in db' results. Unfortunately, the cache where the 32 links/paths in the queue are stored in not very aware of itself, so it only currently understands 'already in db'. I may expand it to understand a different state like 'already imported in this queue' in the ongoing downloader/importer overhaul, although this may be more tricky than it is worth.

If I do put time into this, can you think of a simple way of informing the user that this event has occured? Is another file import state a good idea, or just extra clutter? Maybe only display it in the file list?


c02b86 No.7178

>>7176

>>7175

That was indeed the case importing them both at the same time did indeed show one image in the import with two images processed.

I feel like it would be nice to have a counter, of images that were already in the queue or in the database. Maybe in the 'file import status' window, like in image 3, could show this in the note column, and if not too much work show the other ignored source. but that might be more tricky than it's worth indeed.


e675ef No.7179

Gelbooru subscription unable to retry failed download.

AttributeError

'SubscriptionQuery' object has no attribute 'RetryFailures'

File "include\ClientGUICommon.py", line 280, in EventButton

self._func( *self._args, **self._kwargs )

File "include\ClientGUIScrolledPanelsEdit.py", line 2377, in _RetryFailed

query.RetryFailures()


8713f4 No.7180

>>7176

No, What I mean is in the I beleive lower left corner when the program is initially loading a session, there is a

XXXXXX/Somenumber

That shows how many images are on this page, and how many are currently loaded.

What im wondering is if there is a way to have a global version of this, lets use my retarded session as an example

Normally I load up the client and tab to the 40k image tab (holy shit is going through this a bitch) as this one will finish last, once this one is finished, the client works without issue, before then there is a good chance only the thumbnails will load and none of the full size images will load till everything is finished.

so I sit on my 46,470 tab 15,256/46,470 to tick over to 46,470/46,470

what may be a bit more helpful in this would be a global one so it shows

XXXXXX/419067

as I write that number it has occurred to me how absolutely fucked my session is and how massive the backlog has gotten, especially considering I have 2 other sessions that are 200k+ from when I was importing everything on one of my drives.

If you still don't understand I will restart my client so I can screenshot what i'm talking about.


ad283a No.7181

>>7176

>Is another file import state a good idea, or just extra clutter? Maybe only display it in the file list?

I think it's unnecessary. It''s pretty apparent when duplicates combine tags or namespaces and due to Hydrus' nature that's actually an intended feature: not having 2 copies of the same image that have the same hash


f2dcb2 No.7182


c146f5 No.7183

>>7176

>debug release

I had the same problem as f2dcb2 and I'm on Windows 7. Here's my error:


Traceback (most recent call last):
File "client.py", line 20, in <module>
from include import ClientController
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "include\ClientController.py", line 23, in <module>
import ClientGUI
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "include\ClientGUI.py", line 11, in <module>
import ClientGUIManagement
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "include\ClientGUIManagement.py", line 14, in <module>
import ClientGUICanvas
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "include\ClientGUICanvas.py", line 15, in <module>
import ClientGUIShortcuts
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "include\ClientGUIShortcuts.py", line 8, in <module>
import wx.lib.flashwin
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "site-packages\wx-3.0-msw\wx\lib\flashwin.py", line 15, in <module>
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "site-packages\wx-3.0-msw\wx\lib\activex.py", line 44, in <module>
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "site-packages\comtypes\gen\myole4ax.py", line 1, in <module>
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 161, in load_module
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "site-packages\comtypes\gen\_99AB80C4_5E19_4FD5_B3CA_5EF62FC3F765_0_1_0.p
y", line 7, in <module>
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "site-packages\comtypes\gen\_00020430_0000_0000_C000_000000000046_0_2_0.p
y", line 8, in <module>
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 161, in load_module
File "c:\python27\Lib\site-packages\PyInstaller\loader\pyimod03_importers.py",
line 396, in load_module
File "site-packages\comtypes\automation.py", line 141, in <module>
File "site-packages\comtypes\automation.py", line 142, in tagVARIANT
File "site-packages\comtypes\automation.py", line 143, in U_VARIANT1
File "site-packages\comtypes\automation.py", line 146, in __tagVARIANT
File "site-packages\comtypes\automation.py", line 147, in U_VARIANT2
File "site-packages\comtypes\automation.py", line 149, in _tagBRECORD
TypeError: _type_ must have storage info


8713f4 No.7184

Just got this error

IOError

cannot identify image file <open file u'C:\\Hydrus\\Thumbnails\\t72\\72460e12b26d8ac0982187fa643b53bdc6ccaa4976a46895e58f047b770622dc.thumbnail', mode 'rb' at 0x0000000E1B1015D0>

Traceback (most recent call last):

File "include\ClientCaches.py", line 1774, in _GetResizedHydrusBitmapFromHardDrive

hydrus_bitmap = ClientRendering.GenerateHydrusBitmap( path, mime )

File "include\ClientRendering.py", line 19, in GenerateHydrusBitmap

numpy_image = ClientImageHandling.GenerateNumpyImage( path, mime )

File "include\ClientImageHandling.py", line 63, in GenerateNumpyImage

pil_image = HydrusImageHandling.GeneratePILImage( path )

File "include\HydrusImageHandling.py", line 117, in GeneratePILImage

pil_image = PILImage.open( fp )

File "site-packages\PIL\Image.py", line 2572, in open

IOError: cannot identify image file <open file u'C:\\Hydrus\\Thumbnails\\t72\\72460e12b26d8ac0982187fa643b53bdc6ccaa4976a46895e58f047b770622dc.thumbnail', mode 'rb' at 0x0000000E1B1015D0>

The thumbnail is in the folder and works, not sure why the program threw an error


8713f4 No.7185

>>7184

found the thumbnail and the image in the program, nothing seems wrong with them.


000000 No.7186

>>7176

Found out my issue afterwards. I updated tesseract and it broke ffmpeg after attempting to play a video:


mpv
mpv: error while loading shared libraries: libtesseract.so.4: cannot open shared object file: No such file or directory

After recompiling/reinstalling, ffmpeg build works and hydrus runs.


c7f12e No.7187

>>7176

cute birb.


8713f4 No.7189

File: f29b9a33d03b83a⋯.png (166.96 KB, 1751x1170, 1751:1170, client_2017-11-10_21-17-55.png)

Ok, two things

1) had the crash again when I started to many thread watchers at once.

2)because I had to restart I took a screenshot of what I was talking about >>7168

see where we have the loading initial files

What i'm wondering is if next to it or inplace of it on initial loads, could there be a global initial file load as that seems to be more useful for telling us when the program is ready.


8713f4 No.7190

>>7189

Ok with the crash, I had around 6 files fuck up, and not have data for thumbs or images…

Now, Once multi thread watcher gets implemented this problem will likely go away entirely, but lets assume there is a time when the program crashes and fucks downloads, in something like a thread watcher or a page of pages search, would it be possible to have the program forget the thumb and file link and reacquire it from the new source?

I can't think of a lot of uses but I can at least see it's easier than me deleting the image from the archive, finding and downloading the individual image, then setting it to import the deleted image.


89ea57 No.7191

do you think we could get regex options for import folders? i might be the only one who wants it but it'd save me a lllllot of effort

i want to be able to drop a file named picture_thing_0005.jpg into my import folder, and have it import with the tag some_namespace:0005

at this point, if i want to do that kind of importing, it has to be manually entered - but i definitely have a lot of stuff i'm very regularly adding where i want to run basic regex on the filenames, which tend to stay the same format


221dc2 No.7193

File: 1e8b2bafcc2be10⋯.jpg (205.48 KB, 1280x1720, 32:43, 1e8b2bafcc2be10bdefeb632de….jpg)

>>7179

Thank you for this report. I apologise–I forgot to add the link to the retry failures action at that level. I have this fixed for next week, but if you would like to do it before then, going into the individual queries' edit pages and right-clicking the file import status button should allow you to retry failures.

>>7180

I see, thank you for the follow-up. Yeah, this sounds doable. There's a small problem in doing this stuff in that I want results to load as fast as possible, but if I push that as hard as possible, the gui then doesn't have any CPU time to update. It is common for my session to hang the gui on boot, but it takes about 10s so I don't mind. Now the client is more comfortable with 100+ nested tabs, I will put some time into improving the feedback. It'd be nice to have a cancel button as well.

Btw, if you haven't tried it yet, adding system:limit can be a great way to collapse those 40k 'sort through this when you have time' pages down to manageable chunks.

>>7182

>>7183

Thank you for these. This is breddy weird–it looks like flash isn't loading properly. I updated the module that puts together my builds last week, so I am roughly guessing that is what has done this. Maybe the flash activex dll is being searched for in a different way, and for your machines it is breaking. Since you haven't been able to boot, it should be safe for you both to rollback to v280 for now–just install/extract v280 and you should be fine to boot back into the old version.

I will write a better handler for a failed flash import on windows for v282. This should give us some breathing room to figure out what has gone wrong for you. Please let me know how it works for you once you have a chance to try it.

>>7184

>>7185

Thank you for this report. Can you post or email me the thumbnail and the original file? The .thumbnail is actually a .jpg or .png in secret, so you can just rename it with the .jpg extension and it should work to upload.

Keep a copy of the apparently broken one around, but you might like to try just deleting the thumbnail and then rebooting the client and loading that file again (system:hash can be helpful here). The client is supposed to recover from a missing full-size by generating a new one, so it would be useful to know if it is still generating bad new thumbs, or if this is a one-off.

>>7186

Thanks. I am glad this got sorted. In any case, I'll improve the error handling on my end.

>>7189

>>7190

Thanks. Although it is a pain in the ass to hear, a working solution for now for the 'it crashed because of too much shit' is to slow down or reduce the file count on some of your stuff. Having 100,000+ files open at once does add strain to the client's internals.

Unfortunately, there aren't usually easy ways to recover from a crash or detect what went wrong because by definition, I've lost control at that point. The client usually doesn't know a file is missing or broken until you try to load or click on it. I can always improve that error handling code, but we aren't at the level of 'ah yeah, it came from that downloader, so I'll reschedule it' yet.

>>7191

You aren't the only one. I'm absolutely keen to do it–I want it myself, even–but I am having trouble fitting it into a normal week's work.


8bf7c4 No.7194

File: 7ab422f45370826⋯.gif (187.06 KB, 827x1169, 827:1169, 3007344_p0.gif)

New user here. I've just set Hydrus 281 up on linux (Ubuntu 14.04) and there seems to be a bug with mime type detection.

It is identifying the attached gif as an mp3 file.

Is there any way to manually override this so I can view the file within Hydrus? Another thread mentioned something about a "system:mime type" tag.


8bf7c4 No.7195

>>7194

Disregard that. I'm an idiot. I had an xcf version of the same file which it was trying to import.


221dc2 No.7196

>>7195

np. The mp3 detector has always been a little overenthusiastic, but xcf is new to me. I'll see about moving it to ffmpeg or something that'll be more reliable.


8713f4 No.7198

>>7193

ill send the thumbnail at some point, got other shit to do first. may have it to you before next update may not.

as for the crash, the crash is simply me trying to add 5 thread watchers in short order before they are done, I have little over 400k images across all the tabs.

The issue is at least how I see it, I never find these on their own, it always happens due to a crash, and the only time I crash is on downloading things, so I load up the thread again, and here is a failed image.

In this scenario, would it be possible to add a delete+reacquire function, as the reason it was caught in the first place is the program is trying to download an image that the program already has, but when it tries to display it, its lacking thumb and file.

on a side note, are you able to limit a page of images when the page is already set? the 40k one I mention every now and then is a mass import from what was my desktops 'keep at the ready' folder. If I can limit what it loads up while still keeping all the images from this tab in memory that would be VERY helpful.


cb40c9 No.7199

>>7196

On linux it's a pretty safe assumption that the program "file" is available. If you call `file –mime-type -b /path/to/file.mp3` it will return a standard mime-type string such as "image/gif" or "image/x-xcf". It's probably more reliable than anything any of us could code by hand.

https://linux.die.net/man/1/file


cb40c9 No.7200

I've also noticed another problem on linux.

When I try to zoom in on an image, it can be extremely laggy. There is a large delay between issuing the command and seeing the zoom change on the screen, but once it has finished I can pan around smoothly. I suspect this is because you're rendering the entire image at a higher resolution rather than only the section which is visible on the screen, but I couldn't find the relevant part of the code to verify this hypothesis.

When I try to zoom out, everything works nicely except the screen doesn't update. I have to pan in order to force a redraw event. Again, I couldn't find the code responsible for this.

Also, once nice feature would be if zooming used the mouse location as the origin rather than the middle of the display area. It feels more intuitive to position the mouse over the part of the image I want to zoom in on rather than having to do a lot of panning around.


cb40c9 No.7201

>>7200

Actually, to expand on this, zooming out only fails to update the screen if the zoom factor is large. Testing suggests it only happens when the total zoomed-in image size is more than about 2500 px in one dimension.


cb40c9 No.7202

Oh, and if I zoom in too far on a large image, the whole program crashes with a memory error:

>The program 'client' received an X Window System error.

>This probably reflects a bug in the program.

>The error was 'BadAlloc (insufficient resources for operation)'.

>(Details: serial 666311 error_code 11 request_code 53 minor_code 0)

>(Note to programmers: normally, X errors are reported asynchronously;

>that is, you will receive the error a while after causing it.

>To debug your program, run it with the –sync command line

>option to change this behavior. You can then get a meaningful

>backtrace from your debugger if you break on the gdk_x_error() function.)

You should definitely make it only render the visible portion of the image at high zoom levels.


c02b86 No.7204

>>7181

I disagree, now Devanon spotted it, but imagine what will happen when I import larger and larger folders of my collection, I have folders with like 300 images should I personally check them all for being the same? that seems almost impossible


ad283a No.7205

>>7204

What the hell are you talking about? Unless Hydrus says the import failed then your file is in there. During initial import setup it even tells you if there are unsupported mimes and if you check the option to delete files once imported then your remaining folder would be either empty or contain the remaining files whose mimes could not be parsed, outright showing you that all files were successfully imported.

You don't need to check for anything you moron.


88dcfa No.7207

found a problem when merging subscription's any set explicit tags are broken.


6427dd No.7213

WindowsError

exception: access violation reading 0x00000000B354B020

File "site-packages\wx-3.0-msw\wx\lib\activex.py", line 151, in MSWTranslateMessage

is what I get trying to view shockwave/flash files.


c02b86 No.7214

>>7205

Yes, unsupported mimes are definitely not a problem, and when there is a problem, Hydrus tell's you, which is good.

I dutch we say: "Vertrouwen is goed maar controle is beter" meaning, trust is okay, but you're better off checking.

and this especially applied to the fact that there where two implied numbers of images on the same page. Which points, at least following my thoughts, to a problem somewhere.

>It's pretty apparent when duplicates combine tags or namespaces

That is indeed the fact, but let's say this: You are importing 1200 images that you collected over about a year. You have no idea how many duplicates are in there. The client reports no faulty mimes and says 1200/1200 have been imported. But there are only 1150 files in the import window, what happened? duplicates? other problems? a glitch in the system? nobody knows.

And since importing duplicates seems like a common thing, I feel like an import state is quite useful.


ad283a No.7215

>>7214

Nothing "happened." Your remaining 50 files are duplicates. If Hydrus glitches or if the file was unreadable then the file would fail to import and would be noted as such. It also would not be deleted with the deletion option checked, meaning you can check it out yourself or re-import.

The only possible ambiguous thing would be if the duplicates were already existing in the db or if they were from that exact import session, which can be determined with simple math along with manual date checking to determine the specific files in question. Which is completely pointless unless you're autistically curious about it.

t. someone who has done exactly what you are describing several dozen times with absolutely no deviation from what I have described above.


221dc2 No.7216

>>7183

>>7182

It looks like PyInstaller (the program that builds my python code into an exe) is the problem here. The new version breaks flashwin somehow. Some windows users who can boot (including me) get a raft of ugly popup error when we try to play flash files.

I have rolled back to the older version of pyinstaller for v282. I'll see if I can fit in the better import error recovery as well, but in any case, please let me know how it works for you.

I apologise for this. I will be testing flash embeds every week from now on.


8713f4 No.7217

Ran into instability on page counts now at 175 pages.

going to be rolling back pages till I hit something stable.


8713f4 No.7220

>>7217

Ok just got done crashing and reloading the loop, I had to exit 6 tabs to get a my files tab to open, but it imediately crashed

I had to close 7 to get the files tab open, and load the last 3 days of images, this crashed when I wanted to sort by time import

the program reloaded from this crash with no issues so far.

I would suggest making a limit/warning option at 160 tabs


147c7a No.7225

>>7215

I guess you feel quite strongly about this. I feel like I might be what you call 'autistically curious about it.' it's just how I learned to be, that saying, Vertrouwen is goed maar controle is beter, is really something that gets ingrained into you in some parts of the Netherlands. Or, like you suggest, I might be autistic, I really don't know.

I guess what I'd say is that when it's not too much work, add a new import state for duplicates within the import, and if dev thinks it's worth it I'd like to have, at least in the file list.

t.autistically curious


8713f4 No.7227

Ok, so here I am, I consolidated the tabs to get shit stable again, and I open a new one

the program is now in this constant 'this tab is importing' mode and it is completely killing performance, hanging the program.

I just opened all selected images in a new tab to get it out of the import one, and restarted it, let's hope this fixed it.


8713f4 No.7230

>>7227

Found out the fuck up, and its stupid

So, to consolidate tabs while i'm at the edge of stability, I decided not to import thread watchers and instead do page of images.

now, this with the help of 'copy all urls' extension makes it fairly easy to import all the 4chan tabs at once and not pay attention.

Apparently when I did this, steam was one of the tabs, either I didn't notice it or I thought fuck it, shit won't cause an issue.

so for the last 6 hours steam was causing constant lock ups of the program resulting in nothing.

So here is a suggestion.

is there a way to make a threshold for every single file is fucking up, let's hold up and inform the person using the program something is fucked up?

maybe 100-200 fuck ups in a row where every link fails, or possibly a failure threshold for x images,

Also, tab focusing is possible, because the 3 times I restarted the program, I thought it was the last tab because that's when the issues popped up or at least steamingly popped up. granted my ui setup (see >>7189) doesn't lend itself to showing much info that may have been helpful.

So yea, my thinking is a fuck up threshold that would cause the program to pause what it's doing, something in the range of 100-300 would likely be good enough

Tab focusing, because when I went to restart the program this would have been catchable if it would have focused on the tab.

Here is another one, You know how you have 404/dead for the thread watcher? What if you had a [in progress] [finished] [error] status for page of image downloader? This would likely work for every downloader other than thread watcher and could be used to convey info without tab focusing.


8713f4 No.7231

>>7230

oh, and just for reference, steam decided to make 7000 links to check up from… If I remember the chrome window right, 1 page of steam.


ad283a No.7232

>>7225

That's fine but that's not what you said originally. You're acting like you need to check every single file manually just to be sure it's imported into hydrus. That can already be done with things as they are now by you using a little mental math. Then you start inventing scenarios that don't happen. If you're this paranoid about Hydrus importing your files, then why stop there? How do you know that each file is a 1:1 copy of the fie you originally imported unless you run them all through a hash checker and make sure they match?


147c7a No.7234

>>7232

>it says 31 files are there, but the 32 files have been imported

That was what I meant, I don't check everything manually, when the numbers match up I trust it, but when the numbers seem to be confusing, or I don't understand them, I ask about them like I did in the first post I made regarding this issue.

I hope this cleared the confusion you had.


459b47 No.7236

where are the files i import into hydrus stored?

sorry for the noob question.


c02b86 No.7237

>>7236

The images are stored in

C:\Hydrus Network\db\client_files

for the standard install location Don't forget that these files should not really be removed or changed.


c02b86 No.7238

>>7237

to quickly add: you should be able to do RMB-> copy and then choose what you want to get a file or hash/your tags for sharing


459b47 No.7239

>>7237

>>7238

there's like 100 folders in db\client files and only some of them have images. Im confused here, what do these folders mean? why are there so many if i've only uploaded about 4 images?


c02b86 No.7240

>>7239

These are all the locations that hydrus will once use to house your images, in a machine friendly way, if you want to export your images to a new folder there is a function to do that under RMB->share->export.

Is there a specific reason you wanted to see the files?


459b47 No.7241

>>7240

not really, i just want to understand how this thing works, at least superficially. it just throws the images into random folders then?


c02b86 No.7242

>>7241

It creates hashes from the files that you import, I think. but am not sure, that the symbols in the folder names are the first parts of the hash, and that the filename is the rest of the hash. some same kind of structure is used for the thumbnails I think, when hydrusdev is back he might have some more info if he wants to


ad283a No.7243

>>7234

Yes, yes. But then you went on to create a scenario that doesn't work the way you're suggesting it does. I understand what you're trying to say but it's really not difficult to figure out. Another way to do this would be to keep the original file name when you import. That way you would see the name spaces stack for the exact duplicates.

>>7241

No, they aren't random. Hydrus deduces a file's hash and shoves in the appropriate folder that begins thar hash. All files whose hashes start with "10" are thrown into the f10 folder. All files whose hashes start with "A5" go into the fA5 folder. etc.


459b47 No.7244

So how would i go about tagging thousands of images? is it possible to sort tagged images or have them display in a specific order? (for example, the pages from a comic book)

>>7243

well that explains a little.


c02b86 No.7245

>>7243

>Another way to do this would be to keep the original file name when you import. That way you would see the name spaces stack for the exact duplicates.

that is a really good idea actually, I'll be sure to do that next time importing large amounts of images


8713f4 No.7249

>>7234

For this specific issue, at least on 4chan, there are usually 1 more link that is non image that the program sees as a potential spot to check. for the most part its nothing, looking through enough threads because of this and their logs made pretty much made me accept -1 as the a tolerance.

though to keep track of thread watchers, they have a tendency miss the last several images.


0ceadc No.7250

>>7199

There's also a program called "mimetype" that just returns the mimetype (warning: You have to supply mimetype with the -M switch, or else it will read the file extension and not the magic number). A quick search tells me that these two programs do not function quite the same way.

https://askubuntu.com/questions/427509/file-mime-type-vs-mimetype-for-php-files




[Return][Go to top][Catalog][Nerve Center][Cancer][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / hikki / leftpol / newbrit / strek / sw / tijuana / u / yg ]