[ / / / / / / / / / ] [ dir / cute / egy / fur / kind / kpop / miku / waifuist / wooo ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.

Catalog

Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 12 MB.
Max image dimensions are 10000 x 10000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 714 items

Current big job: finishing off duplicate search/filtering workflow

YouTube embed. Click thumbnail to play.

0e2e4c No.5287

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v246a/Hydrus.Network.246a.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v246a/Hydrus.Network.246a.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v246a/Hydrus.Network.246a.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v246a/Hydrus.Network.246a.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v246a/Hydrus.Network.246a.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v246a.tar.gz

I had a great week getting back to normal schedule. I've fixed a lot of outstanding bugs and moved other ongoing jobs forward.

a/c fixes

I have fixed a problem in how some autocomplete counts were being generated. You will be prompted to regenerate your autocomplete cache once you update. Please do so, and let me know if you discover any other miscounting in the future–I think there may be another issue, likely with 'all known files' counts (usually seen in the manage tags dialog).

Furthermore, autocomplete queries will now search for namespace, so typing 'char' will give you all the 'character:' tags. I find this very useful, but it is an experiment–if you find it way too laggy, let me know and I'll add an option to disable it.

I have also fixed multiple issues with wildcard searching (putting '*' in a tag autocomplete search to represent any number of characters). There are several changes and improvements to this code, so if you do a lot of these searches, please let me know how you now find it and whether you would like some options to customise its behaviour.

invalid tags

Last week, I made tags with spaces before the 'subtag' component invalid. An example would be 'series: kill la kill'–attempting to type that should collapse it down to 'series:kill la kill'. As some users prefer to have the leading space, my intention is to collapse everything to a common standard and then add gui options to customise display to whatever namespace separator people want, whether that is to stay as ':' or move to ': ' or ' - ' or whatever they like.

I converted and renamed these tags serverside last week, and now I have updated the clientside records. I apologise if you have had trouble editing tags in the meantime.

While I have tried to automatically swap invalid tags with good, unfortunately these systems are complicated and so I have sometimes had to replace invalid tag components with valid placeholders such as 'hydrus invalid tag:"old_invalid_tag_here"'. I have attempted to minimise this mess, but there is still a bit hanging around, particularly on tag repositories. Some clients will see more than others. If you see these tags, please do petition/delete/replace them as appropriate. I expect to write a bit more code to further clean these up in future, so please let me know if you are overwhelmed with them and what form they tend to be in.

other stuff

Review services will now show file and tag counts for services that have them!

The prototype duplicate search page now has a control to select the file domain, so I believe the issue of physically deleted files appearing in the current 'show some pairs' system should be fixed. Let me know if it isn't!

I fixed the issue of the new repository update files appearing in 'all local files'–they will now be hidden, and system file counts corrected.

full list

- fixed a critical bug in serverside content deserialisating that meant servers were not processing most client-submitted data properly

- fixed a critical bug in 'all known tags' autocomplete regeneration–please run database->regen->a/c cache when it is convenient and let me know if your numbers are still off

- fixed the pre-v238 update problem for good by abandoning update attempts and advising users to try v238 first

- clientside invalid tags will now be collapsed like with the server last week. if a tag is invalid (typically something with an extra space, like "series: blah"), the update code will attempt to replace existing mappings with the collapsed valid version. some unusual cases may remain–they will be replaced with 'invalid namespace "series "' and similar. Please remove and replace these when convenient and contact me if there are way too many for you to deal with

- duplicates pages now have a file domain for the filtering section, and they remember this domain through session loads

- this file domain is accurate–counting potential duplicates and fetching pairs for 'show some pairs' only from those domains. the issue of remote files appearing should be gone!

- there is now only one 'idle' entry in the duplicates page cog menu–it combines the three previous into one

- fixed numerous irregularities across the wildcard code. all search input now has an implicit '*' on the end unless you put a '*' anywhere else, in which case it acts exactly as you enter it, with a non-* beginning matching beginning of string, whitespace, or colon, and non-* end matching end of string or whitespace

- autocomplete now searches namespace, so entering 'char' will load up all the 'character:' tags along with 'series:di gi charat'. this can lag to hell and back, so it may either need some work or be optional in the future. feedback would be appreciated

- typing 'namespace:' will include all the series tags below the special optimised 'namespace:*anything*' tag

- autocomplete searches recognise an explicit '*' no matter where it is in the entry text. typing 'a*' will load up all the a tags and present a 'a*' wildcard option

- quickly entering a wildcard entry will now submit the correct wildcard predicate ('rather than a literal 'hel*' or whatever tag)

- review services panel now reports total mappings info on tag services

- review services panel now reports total files info on file services

- manage services's listctrl is now type | name | deletable and initially sorts by type. the strings used for hydrus service types are also improved

- manage serverside services (called by server admins to manage their services) have fixed setnondupe port and name on edit service events

- new popup messages will now also appear if there were previously no popup messages to display if the current focus is on a child on_top frame, such as review services (you'll now see the processing popup appear when you click 'process now' on review services)

- the popup message manager now initialises its display window with a single message that is quickly dismissed. this helps set up some variables in a safe environment so they don't have to be generated later when the gui might be minimised or otherwise unusual

- hid hydrus update files from 'all local files' searches

- added 'media_view' entries for hydrus update files, just in case they are still visible in some unusual contexts (and they may be again in a future update anyway)

- fixed 'recent tags' being returned from the database out of order

- by default, 'recent tags' is now on for new users

- 'get tags even if file already in db' now defaults to False

- file import status now allows a 'delete' action below the 'skip' action

- file import status right-click event processing is more sane

- fixed the new raw width/height sort choices, which were accidentally swapped

- cleaned the media sort code generally

- cleared out some redundant rows that are in some users' client_files_locations

- namespaced predicates are no longer count-merged with their namespaceless versions in 'write' autocomplete dropdowns

- 'unknown' accounts should now properly resync after clientside service change

- improved how registration keys are checked serverside when fetching access keys

- fixed a v244 update problem when unexected additional tag parent/sibling petitions rows exist

- improved my free space test code and applied it to the old v243->v244 free space test (it'll now test free space on your temporary path and report problems appropriately)

- to improve log privacy and cleanliness, and to make it easier to report profiles, db/pubsub profiles now write to a separate log file named appropriately and labelled with the process's start time

- profiles are more concise and formatted a little neater

- across the program, three-period … ellipses are now replaced with their single character unicode … counterpart (except to the console, where any instance of the unicode ellipsis will now be converted back to …)

- cleaned up some log printing code

- cleaned up some experimental static serialisation code, still thinking if I like it or not

- started on some proper unit tests for hydrus serialisable objects

- fixed and otherwise updated a heap of unit test code to account for the v245 changes

- cleaned up a bunch of old database table join code

- started some databse-query-tuple-stripping code cleaning

- deleted more old unused code

- misc timing improvements

- misc code cleanup experimentation

- misc cleanup

next week

More of this, I think. I have a ton of small stuff to catch up on.

39c572 No.5290

File: 0a1e3b8bdaba8e5⋯.jpg (131.31 KB, 700x1156, 175:289, 0a1e3b8bdaba8e5ca0a4a10314….jpg)

I get this error when updating from 245 to 256:

2017/03/09 01:32:18: hydrus client started
2017/03/09 01:32:18: booting controller…
2017/03/09 01:32:18: booting db…
2017/03/09 01:32:20: updating db to v246
2017/03/09 01:32:21: cleaning tags
2017/03/09 01:32:24: 76 dirty tag parts found, now cleaning
2017/03/09 01:32:24: A serious error occured while trying to start the program. Its traceback will be shown next. It should have also been written to client.log.
2017/03/09 01:32:24: Traceback (most recent call last):
  File "C:\Hydrus Network\include\ClientController.py", line 1106, in THREADBootEverything
    self.InitModel()
  File "C:\Hydrus Network\include\ClientController.py", line 527, in InitModel
    HydrusController.HydrusController.InitModel( self )
  File "C:\Hydrus Network\include\HydrusController.py", line 205, in InitModel
    self._db = self._InitDB()
  File "C:\Hydrus Network\include\ClientController.py", line 66, in _InitDB
    return ClientDB.DB( self, self._db_dir, 'client', no_wal = self._no_wal )
  File "C:\Hydrus Network\include\ClientDB.py", line 152, in __init__
    HydrusDB.HydrusDB.__init__( self, controller, db_dir, db_name, no_wal = no_wal )
  File "C:\Hydrus Network\include\HydrusDB.py", line 226, in __init__
    raise e
Exception: Updating the client db to version 246 caused this error:

Traceback (most recent call last):
  File "C:\Hydrus Network\include\HydrusDB.py", line 205, in __init__
    self._UpdateDB( version )
  File "C:\Hydrus Network\include\ClientDB.py", line 8204, in _UpdateDB
    self._c.execute( 'UPDATE subtags SET subtag = ? WHERE subtag_id = ?;', ( clean_subtag, dirty_subtag_id ) )
IntegrityError: UNIQUE constraint failed: subtags.subtag


2017/03/09 01:32:37: shutting down controller…
2017/03/09 01:32:37: hydrus client shut down

I assume what's happening is hydrus is finding of the one those "namespace: tag" tags and collapsing it to "namespace:tag" but there's a conflict as that tag already exists in my db?

Extra weird because I don't use that weird format of namespacing tags, it's probably residue from all the booru tags imported in my client.


865f2a No.5291

File: 5e6e187e97bbeee⋯.jpg (128.76 KB, 1000x1000, 1:1, 5e6e187e97bbeee208626fdaec….jpg)

>>5290

I did a pdb.set_trace() where the error occurs and dumped dirty_subtag_info and got this:

http://pastebin.com/k2AiMKjz

It was failing on this tag:

[
  84475,
  "-:",
  ":"
],

Hopefully that's a bit more helpful.


c6fb42 No.5295

Yay, it worked. Thanks for the hard work boss.


3ead8c No.5296

File: e206ccb7fb8d8d1⋯.jpg (199.39 KB, 492x514, 246:257, e206ccb7fb8d8d19fb97658cfb….jpg)

>>5290

>>5291

Hey, I'm sorry for this problem. There's a bunch of logic that is supposed to collapse these without dupes, but I guess that '-:' is an odd case. I wonder if it is conflicting with ' :' somewhere else, but I'll have to look at the code. Thank you for reporting this.

I will fix the update code for next week, but if you would like to attempt a manual fix in the meantime, you can try:

1) Make a backup of your client*.db files.

2) Run the sqlite3 executable in your db folder.

3) Copy paste this into the terminal that will appear:

.open client.master.db
UPDATE subtags SET subtag = "special invalid subtag" WHERE subtag = "-:";
.exit

And then try updating again. There might be some other cases that don't collapse properly, in which case you can catch and edit those manually as well.

Please let me know how you get on!


ba6999 No.5300

Is there an option to change get tags if file already in db to true by default?


ef1e5a No.5301

So…is there drag-select yet?


1caf26 No.5304

>>5301

No.

>>5300

The dev said he's gonna add it for next week's release.


67bd02 No.5305

File: 67ceaafd5dfc3bf⋯.png (1.84 MB, 1062x1577, 1062:1577, 67ceaafd5dfc3bfdbdfcddcd41….png)

>>5296

That worked, thanks.

I mucked with sqlite a bit to dig into what's up with the UNIQUE constraint conflict. It would appear that in both the original database and updated database I have a subtag that is just ":" no space included. Both of these subtags "-:" and ":" have only one normal tag entry each and reside in the "caption" namespace I have which at the moment is just preserved captions from pixiv and aren't very important it seems as they don't pop under the autocomplete which I assume means that I have deleted the files these tags were associated with.


71b185 No.5306

>>5287

Getting this after clicking commit when finished filtering:

ValueError

too many values to unpack

Traceback (most recent call last):

File "/home/hydrus/Desktop/hydrus/build/client/out00-PYZ.pyz/include.HydrusDB", line 478, in _ProcessJob

File "/home/hydrus/Desktop/hydrus/build/client/out00-PYZ.pyz/include.ClientDB", line 8896, in _Write

File "/home/hydrus/Desktop/hydrus/build/client/out00-PYZ.pyz/include.ClientDB", line 6310, in _ProcessContentUpdates

File "/home/hydrus/Desktop/hydrus/build/client/out00-PYZ.pyz/include.ClientDB", line 2281, in _DeleteFiles

ValueError: too many values to unpack


84d366 No.5307

>>5306

The result seems to be that files selected for arhival are archived correctly, files selected for deletion are not deleted, stay in inbox


228d2d No.5308

>>5307

Sorry for double posting, now I see that actually it happens for all mass deletion above a certain number of files not just when filtering. I can delete the files in smaller amounts


37129b No.5316

File: 6f678cf7af3f767⋯.jpg (80.18 KB, 858x741, 22:19, 6f678cf7af3f767b7aaf32f5d4….jpg)

>>5301

The thumbnail display code is a huge mess atm. Drag select will have to come in the next overhaul of that stuff, but I'm not sure when it'll happen.

>>5305

Thanks for this follow-up.

>>5306

>>5307

>>5308

Thank you, this is fixed for v247. The magic number is >256 files.



[Return][Go to top][Catalog][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / ] [ dir / cute / egy / fur / kind / kpop / miku / waifuist / wooo ]