[ / / / / / / / / / / / ] [ dir / 8teen / asmr / girltalk / maka / newbrit / s8s / strek / tijuana ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 12 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 1035 items

Current big job: downloader engine overhaul

YouTube embed. Click thumbnail to play.

8e8229 No.6445

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v267/Hydrus.Network.267.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v267/Hydrus.Network.267.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v267/Hydrus.Network.267.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v267/Hydrus.Network.267.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v267/Hydrus.Network.267.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v267.tar.gz

I had a great week. I fixed new and old bugs and polished a bunch of the new network stuff.

bug fixes

I was overambitious with my time last week, and it lead to some frustrating bugs. If you were affected, I apologise. Things should be running better now.

First of all, subscriptions with unicode characters in their names should save their bandwidth history without problems now, so please unpause them.

And the occasional flickering 'modal' popup messages should also be gone. They won't appear unless they have plenty of work to do, will behave better when the client is deminimising, and should exit cleanly (sometimes, they were exiting badly and locked the user out of the client). Let me know if you still get any problems here.

Also, I put some time into exploring the opening-dialog-when-busy crash properly and have it fixed with a new custom control. Anyone who previously had a crash opening a dialog or sorting a listctrl while they had 20+ pages open should now be ok to open dialogs again. I have turned off all auto-sorting on all the old controls. If this was an issue for you, please do not manually sort any listctrls (the grid/spreadsheet 'tables' of data that you can sort by clicking the column title) for now. The new control is only in 'review bandwidth' so far, so feel free to sort that all you like, but the others will take a few weeks to replace.

page improvements

Pages now have more power over naming themselves. File pages launched from a subscription popup will be named after the subscription, and several other instances will do similar.

By default, pages will now show their current number of files after their name. They will also clip to max 20 characters. If you do not like either of these, you can edit them in options->gui.

I have been talking with several users about better page information and layout, which these changes start. I would like to slip in double-notebook 'nested' layout in the coming weeks, perhaps even next week, to improve high-page-count workflow and make certain sessions a little easier to manage.

Also, pages will use less CPU in most situations, especially when they are not currently in view. 10,000+ file imports should run a lot quicker now!

import improvements

I did some background cleanup and polishing work on most of the importers–the hdd import page now uses the new file import status control, and galleries will sleep and wake more intelligently, for instance.

I also fixed some small bugs and a bigger issue with the gelbooru parser, which was only checking 'odd' page numbers and hence finding only half the results.

And now, drag-and-dropping a 4chan or 8chan thread url onto the client will spawn a thread watcher immediately! Just DnD from the address bar or the Reply link!

full list

- drag-and-dropping a 4chan or 8chan url onto the client will now automatically open a thread watcher for that url

- fixed an issue where web domain or subscription network contexts that included unicode characters in their context data were unable to serialise and save to the db, causing error spam when the bandwidth manager attempted to save itself

- subscriptions will now show a network job control in their popup as they do network work

- subscriptions will cancel more reliably during gallery parse

- subscriptions now have some 'delay' logic that will stop them sometimes restarting as soon as they are cancelled or otherwise have to stop mid-work

- subscriptions will now tolerate up to five 'already in cache' urls per page parse until it considers the page 'already seen'. this is to catch the odd additional late insert and avoid the problem of a page updating and shuffling everything up one as the subscription walks through pages

- maintenance modal popup messages should no longer appear if their jobs are very quick

- queued up modal popup messages will no longer flicker their dialogs on a de-minimise

- should have fixed an issue where modal popup messages could sometimes error out on a close attempt, locking the client's whole gui and requiring a force quit

- fixed an issue where the modal popup message dialog was OKing on a close attempt of a non-cancellable job, despite presenting user text indicating otherwise

- did a bunch of data work on pages

- page tab names will now be clipped to 20 characters by default

- pages now show (num_files) after their name by default, although you can set this for only import pages or turn it off completely

- options for these new behaviours are in options->gui–and page names will update immediately on dialog ok

- subscription and import folder 'show files' events (and a couple of other misc occasions) will now launch their page tab with the sub/folder name rather than a flat 'files'

- wrote a new listctrl class with custom sort code that does not suffer from the 'crash when sorted while many pages open' bug. it also handles data in a simpler way for hydrus

- review bandwidth listctrl now uses this new class and should now not crash your client. I will replace all the other listctrls with the new class over the coming weeks

- temporarily, no old listctrls will auto-sort themselves (as this causes a crash for many users)! if you do not suffer from the crash, please sort them yourselves for now

- reduced some large-scale gui import lag:

- reduced content processing CPU load on clients with many thumbnails open

- massively reduced content processing CPU load on non-current pages

- offloaded newly-imported file thumbnail detection and generation to a non-gui thread

- migrate database dialog now lets you move the whole database and all portable locations, which requires a client shutdown

- the raw url downloader will no longer have a problem with pasted url lists that include empty newlines

- all the downloaders/importers now sleep on cleverer event objects so they will burn less idle cpu and wake as soon as they have new work to do

- fixed and otherwise improved some gallery downloader timing logic

- hdd import pages now use the file import status control

- the thread watcher's controls now try to wrap in a single sizer. it uses less space, but might sperg out, let me know if it is a problem. I'll replace the whole watcher timing system in the new engine anyway

- fixed a gelbooru (and possibly others) booru parsing bug that meant half of the pages in the gallery walk were being skipped (e.g. 'mogudan' was producing ~350 files when there were actually 690 in the list)

- fixed some dialog panel layout scrollbar-cutoff in sevaral Linux places and perhaps elsewhere

- cleaned debug menu a little and added a save 'last session' entry

- fixed an issue where some kinds of media would error on notification of new url association

- clients will now save small transaction progress reliably within ten seconds no matter how idle they are

- deleted the old 'gui capitalisation' option–I never got around to expanding it beyond a handful of menu labels, it was always too much to work on for too little reward

- improved support for certain broken videos–these will import ok, but full rendering might be borked to different degrees, so let me know how it goes.

- cleared out some old content processing code

- cleared out some old unused db data

- removed all old 'waiting politely' download settings and gui code

- misc prep work for wx update

- misc gui code refactoring

- misc downloader cleanup and timing tweaks

- misc improvements

next week

I enjoyed working on smaller stuff and catching up on back-burner jobs and cleanup this week. I would like to do more of this stuff for a few weeks, but I'll also keep putting time into the network overhaul. Hydrus services and a couple of other things still use the old system, and there is plenty of more polish and test code to add before I can properly move on to the login and domain-handling engines.

I played with the new 'Phoenix' version of wx this week. After some work, I got the client to boot in Windows, but there will be a bunch to do to get it running without errorspam. I will try to replicate my experiment in Linux and OS X and then schedule a full week in the next few months just dedicated to changing over.

717c85 No.6447

8 tabs out of my 128 I have open load no images, it sees images were there, it has the records, but nothing is loading. I think this has been a bug for a while, but never really something that affected me much so a non priority, just a heads up that this happens.


8e8229 No.6448

>>6447

Thank you for this report. On the page, does it say like '23 images' on the status bar–is that how you know there are some files there? If you click the white of the thumbnail area and then hit ctrl+a to select everything, do the thumbnails load in then? Or if you just click where the thumbs should be, trying to force a change in thumbnail display state? If nothing loads, do the clicks seem to actually select something on the status bar or on the right-click menu, or are there no thumbs at all, not even invisible ones?

I think the thumbnail streamer is sometimes losing thumbs.


717c85 No.6449

File: f5479078e2aef4c⋯.png (151.72 KB, 1833x1170, 47:30, client_2017-08-03_08-33-40.png)

>>6448

Ok, one of them the download tab has 458 according to the tab number, it has 132 successful, 222 failed and 326 already in db, similar with thread watcher I can see the files this way, up till the recent update though I could not see one that was opened from files, such as my import folders,

the bar at the bottom is bare, ctrl A selects nothing, there is no scroll bar on the side,

If by force the thumbnail state, that is an initial loading issue where no thumbs will load till I resize the window once, but I am not sure if I still get that specific problem

The image is of a the 458 tab, i'm keeping it open because 222 of the images can still be gotten I just have to wait for the owner of the archive to do something.


e13ea8 No.6450

https://8ch.net/hydrus/res/6398.html#q6421

This is still a problem. I have noticed it on other artists as well, some images just get skipped for some reason. I have tested it on a brand new database and it still happens.


c63122 No.6454

File: 69b548a5405d8ef⋯.webm (1.58 MB, 464x222, 232:111, 123.webm)

Something weird going on with subscriptions. Now hydrus find more images with tag "mogudan", but dont download them. This happens with other tags too, "me!me!me!" for example. Here's the video of what happens after adding new or reset old subscription.


27cfd5 No.6456

>>6454

Check bandwidth limits?


c63122 No.6457

>>6456

Looks like that was the problem. Thanks.


ffc1fc No.6458

PyAssertionError

C++ assertion "win" failed at ../src/gtk/menu.cpp(84) in DoCommonMenuCallbackCode(): event for a menu without associated window?

File "/home/hydrus/Desktop/hydrus/build/client/out00-PYZ.pyz/include.ClientGUICommon", line 2567, in EventDirty

File "/home/hydrus/Desktop/hydrus/build/client/out00-PYZ.pyz/include.ClientGUI", line 3804, in RefreshMenu

File "/home/hydrus/Desktop/hydrus/build/client/out00-PYZ.pyz/wx._core", line 12282, in Replace


ffc1fc No.6459

>>6458

Using KDE Neon, originally expecting this is a permission bug, `chmod -R 777` everything but this error still stands and crash the program when using it with https://github.com/quartzic/Quicksync/


190782 No.6460

I am having an issue with one of my subscriptions on Gelbooru that may be related to the previous gelbooru parsing bug. my sub only finds 517 urls for the search term fate_(series) but there are around 72,000 files. It is set to download 200 a day but it is currently only downloading new image uploads and ignoring the other 71,500 files that should exist. if i make a new sub it finds them fine, but i want to know if there is a way to fix it so i don't have to redownload the 500 i already have. it is not related to the bandwidth manager as i have already increased the request and download size.


e834e1 No.6461

>>6460

>It is set to download 200 a day but it is currently only downloading new image uploads and ignoring the other 71,500 files that should exist.

Having the same problem with a tumblr sub. Check now only makes it look for new files and it doesn't bother picking up older stuff


190782 No.6463

>>6461

Well nice to know its not just me though i am sorry for you. Looks like my previous comment on it working if i remade the sub was wrong. after clearing the url cache and remaking it, it still does the same thing. Downloads the initial number i allow it (1,000 this time) and then stops picking up anything older than those so only new uploads are detected.


8e8229 No.6466

File: 62e38c2af6cc6d8⋯.jpg (2.57 MB, 2448x3264, 3:4, 62e38c2af6cc6d8e0e82b1db67….jpg)

>>6447

>>6448

>>6449

I have properly identified and fixed this problem for v268. It is the same problem that sometimes caused downloaders to initialise in a 1/1 state.

I believe these pages are completely safe internally, they are just not displaying correctly. Please keep them around for v268 and they should boot up ok. Let me know if you still have problems with this!

>>6450

Thank you for this report. I am sorry I missed it the first time. I will try to look at this problem tomorrow.

>>6454

>>6456

>>6457

Yeah, subs only get 256MB of files a day now, to split them up a bit. The ui feedback on this is rubbish–I hope in a week or two to have this displaying better text like 'no bandwidth left today, not downloading any more' kind of stuff. I apologise for the current inconvenience.

>>6458

>>6459

We talked about this in the discord. If anyone else has experienced this, please let me know.

>>6460

>>6461

>>6463

I think I will rename and better-tooltip some of these actions, as I have been changing a little of the way subs function in the past few weeks. I have a proper subs help page at http://hydrusnetwork.github.io/hydrus/help/getting_started_subscriptions.html , but it may be a little old. I should also point to it from within the program. I am sorry for the confusion here.

When subs do their sync check (which you trigger by hitting 'check now'), they run the search and get all 'new files', with new meaning the files that come earlier in the search than those it has previously seen.

If we label the results for blue_eyes on a booru 1-10000 and your sub has 200-500, on a check it will walk through the pages for files 1-199 and then stop. It always catches up on newer files. It won't ever look for 501+.

If you would like to go through everything, please 'reset' the subscription. This will clear its url cache and let it do its original sync again. Since in this case it has no urls to compare to, it will keep walking through the gallery results pages until it runs out of files. Or, in >>6463 's case, it only gets the first 1,000 and then stops.

I do not recommend running subs for very large numbers of files. Best practise is to keep total sub size to, say, less than 5,000 files. Artist-based subs are good for this. Or having ('fate', 'archer') type stuff. Very large subs can fail in very large ways, and they are generally unweildy in their initial sync and so on. If you want to do a large download, I recommend you do it through a regular download page for the booru where you have more control to pause and review results and so on.

Also, do not worry a lot about going over the same search twice. For the most part, hydrus does not have to re-download things it has seen before. It knows the urls its files came from, and even remembers those of deleted files, so it can skip redownloads if everything is working ok.

Let me know if I explained any of this badly. I will make sure to give my subs page a pass and add some diagrams or something.


50c178 No.6470

request: can you allow us to store PSD files?

i don't expect to have thumbnails or anything - but many of the images in my library have a corresponding PSD project, and i'd love to be able to find a given file's PSD by tags and vice versa etc.

thx


190782 No.6471

>>6466

Alright thanks for the explanation.

I hadn't reviewed the help page in months and going over the help page it does mention on there it won't go past any URL's it has seen before.

i misunderstood it to be a bug as i had always used get everything for the rest of my sub's and this was my first time using the limiter so i didn't even consider checking the subscription help page. In hindsight it does make sense that it works that way.


31e179 No.6474

>>6445

I'm probably retarded but I'm getting a "serious error" after I updated. Granted I haven't updated the client in like a few months- but did something happen to fuck over the database files? I'm hoping I didn't just lose everything right here.

Dumping the log.

2017/08/08 05:12:42: A serious error occured while trying to start the program. Its traceback will be shown next. It should have also been written to client.log.

2017/08/08 05:12:42: Traceback (most recent call last):

File "include\ClientController.py", line 1199, in THREADBootEverything

self.InitModel()

File "include\ClientController.py", line 557, in InitModel

HydrusController.HydrusController.InitModel( self )

File "include\HydrusController.py", line 234, in InitModel

self.db = self._InitDB()

File "include\ClientController.py", line 69, in _InitDB

return ClientDB.DB( self, self.db_dir, 'client', no_wal = self._no_wal )

File "include\ClientDB.py", line 152, in init

HydrusDB.HydrusDB.init( self, controller, db_dir, db_name, no_wal = no_wal )

File "include\HydrusDB.py", line 222, in init

raise e

Exception: Updating the client db to version 248 caused this error:

Traceback (most recent call last):

File "include\HydrusDB.py", line 201, in init

self._UpdateDB( version )

File "include\ClientDB.py", line 9294, in _UpdateDB

self._UpdateMappings( tag_service_id, mappings_ids = mappings_ids, deleted_mappings_ids = deleted_mappings_ids )

File "include\ClientDB.py", line 9786, in _UpdateMappings

self._CacheSpecificMappingsAddMappings( file_service_id, tag_service_id, mappings_ids )

File "include\ClientDB.py", line 2290, in _CacheSpecificMappingsAddMappings

self._c.executemany( 'DELETE FROM ' + cache_deleted_mappings_table_name + ' WHERE hash_id = ? AND tag_id = ?;', ( ( hash_id, tag_id ) for hash_id in hash_ids ) )

OperationalError: no such table: external_caches.specific_deleted_mappings_cache_1_7


8e8229 No.6479

File: e70d03f53d20062⋯.png (79.82 KB, 459x448, 459:448, e70d03f53d20062c14520d5502….png)

>>6470

I am not sure, but maybe! I use Pillow (aka PIL) as my image library. Its documentation says it has basic PSD support, so I can give it a go. Can upload and link me to an example PSD file of the sort you would like to import, so I can try it my end?

>>6474

I suspect this is a problem due to bit rot. Updating many versions in one step will sometimes not work because the update steps' underlying assumptions about the shape of the database will slowly change over time.

Please try updating in smaller steps, say, five or ten at a time. Your client failed to update to v248 by your error, so try that step first. If that works, try v258 or so, and then jump up to the latest release.

If you have a backup of your client before you tried this big update, I suggest rolling back to it before attempting these smaller update steps. It is possible the large attempt made a permanent change that will need additional work to fix.

If the smaller update steps do not work and you do not have a backup to roll back to, let me know any new information and errors you encountered, and I will try to figure out a manual fix for you to run.

I am confident you have not lost everything! Worst case, it will take a bit of back-and-forth to figure it out.


31e179 No.6480

>>6479

>Your client failed to update to v248 by your error, so try that step first.

Holy shit that actually worked. Thanks for the post.



[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / ] [ dir / 8teen / asmr / girltalk / maka / newbrit / s8s / strek / tijuana ]