[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
REC
STOP
Comment *
File
Password (Randomized for file and post deletion; you may also set your own.)
Archive
* = required field[▶Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options

Allowed file types:jpg, jpeg, gif, png, webp,webm, mp4, mov, swf, pdf
Max filesize is16 MB.
Max image dimensions are15000 x15000.
You may upload5 per post.


This board will be deleted next Wednesday. I am moving to a General on 8chan.moe /t/. This board is archived at 8chan.moe /hydrus/!

YouTube embed. Click thumbnail to play.

229563 No.11732

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.OS.X.-.App.dmg

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v341/Hydrus.Network.341.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v341.tar.gz

I had a great week after being ill at the end of the previous week. There is a bit more than usual in this release because of the extra time. The Client API has some new commands, and there are a variety of fixes and stability improvements.

client api

The Client API now does some fun stuff. If you would like to play with one user's browser extension, which recognises hydrus-compatible URLs–including whether you have the page's file already–and lets you send them straight to hydrus for downloading, please check it out here:

https://gitgud.io/prkc/hydrus-companion

And the Client API help itself is here:

https://hydrusnetwork.github.io/hydrus/help/client_api.html

There is a little info at the top telling you how to turn the API on.

This release brings some more API commands. You can now edit a file's tags, edit a file's URLs, and the 'add_url' command now lets you attach tags for any files coming from the given URL and also lets you determine which page (by page name) the URL will go to (for instance if you have multiple watcher pages working on different check timings for different boards).

the rest

I spent some time this week cleaning things up. Repository update processing uses less memory and now regularly saves its progress on big jobs, reducing transaction bloat, improving error recovery, and giving control back to you much faster when it finishes. Memory use all around should be reduced.

I also replaced several areas of messy UI code, which has resulted in improved stability, particularly for Linux. Some CRITICAL logspam is gone, and I believe I have finally fixed a lingering ui crash on the duplicates processing page (at least for search distance 0–I will still work on >0, which seems to be a db-level problem for linux). Several boot crashes due to incorrect boot-error handling (such as when catching that some client_files directories are missing) are also fixed. I would appreciate your continued feedback here.

Duplicate processing now treats ratings copy/move/merge actions a little different: Before, it would only make a change if the 'destination' file had no rating set–now it will do it if it has no rating or if the rating is smaller than what the 'source' has. For instance, merging two files that have rating 2/5 and 3/5 will result in both files having rating 3/5.

After talking with a user about different ways of storing and presenting thumbnails, I put some work into a thumbnail drawing experiment that only needs one set of thumbs to draw on screen (the current thumbnail system stores two copies for different purposes). If you would like to try it temporarily, please turn it on under help->debug->gui actions->thumbnail experiment mode. Instead of using your 'resized' thumbs, it will only use the 'master'. It makes no changes to your file structure, and you can turn it off at any time. Try loading up some random files in your client and let me know if you get any errors or if any thumbs take an extremely long time to load. I've wanted to drop the two-thumb storage system for some time but never had good numbers to feel good about it, but my tests here are promising–the experiment mode only takes a few hundred µs more per initial thumb load for me, and if I were to switch over to it permanently, I could reduce it even further. Assuming there are no big problems here, I expect to chip away at this problem over the coming weeks and hopefully make a significant simplification in hydrus file storage.

full list

- client api:

- added /add_tags/add_tags, which does several kinds of tag content updates

- added /add_tags/clean_tags, which shows how hydrus will handle potential tags

- added /add_urls/associate_url, which allows you to associate urls with files

- added 'destination_page_name' to /add_urls/add_url, which will choose which destination watcher/url importer to place the url (or create a new one with that name)

- updated client api version to 2

- updated client help and unit tests for the above

- added a linked contents to the client api help

- improved some server error handling, mostly moving 403s to more correct 400s

- improved how missing parameter 400 errors are reported from the server vs deeper keyerrors that should be 500

- .

- the rest:

- tag repository update processing now saves progress to disk every million rows or every minute, whichever comes first. this reduces journaling bloat, improves recovery when the process quits unexpectedly, and makes for significantly faster cancel when requested by the user

- when processing duplicates and copying/merging/moving ratings, the 'source' file will now also overwrite the 'destination' file's rating if that destination rating is lower (previously, the rating would only go over if the dest had no rating set)

- added a new 'thumbnail experiment mode' under help->debug->gui. this will load fullsize thumbs and resize them in memory, please see release post for more details

- reduced menubar replacement flicker while, I believe, keeping and strengthening recent menubar indexing stability improvements

- the tag autocomplete dropdown will now always embed (instead of floating) in non-Windows

- when data seems non-decodable, the fallback encoding format is now that given by chardet, rather than utf-8

- improved serialisability of some pending tag data

- watchers can now hold and pass on fixed pending tag data

- gallery log objects can now hold and pass on fixed pending tag data

- file import objects can now hold and action fixed pending tag data

- hard drive imports now store their paths-to-tags info in this new format, directly in the file import objects

- improved some url-import page drop-target-selection logic

- improved error reporting when dropping/api-adding urls

- adjusted some url import workflow so big 'already in db' download lists should work a bit faster

- attempting to start the program with some external database files but not the main 'client.db/server.db' file will now cause a boot-fail exception with an explanation before any stub db files can be made

- tightened up some hydrus service login-capability-testing code that was previously stopping certain error states from recovering promptly, even on a force account refresh, while the service was maxed on bandwidth

- fixed a source of linux CRITICAL logspam related to several common dialogs

- improved ui stability on boot when file folders are missing (particularly for linux)

- improved stability for the various async tasks on the duplicates processing page, particularly for linux. I am not sure I got everything here, but it is definitely better

- did some more misc stability improvements, particularly in various boot fail scenarios

- completely removed an ancient and janky focus catcher widget from the main gui frame

- now various db caching is improved on the python side, removed a sqlite instruction to force temp information to always stay in memory–hot data returns to staying mostly in memory to start and then spools to disk if the transaction gets too large

- fixed approx bitrate sorting for malformed video files with explicitly '0' duration

- daemon_profile_mode now spams some more info about export folders

- fixed an issue that meant client db maintenance was firing its jobs too aggressively, regardless of idle status

- updated windows build to cv 4.0

- misc refactoring and fixes

next week

We are in the home stretch of the Client API now. I just want to write some simple file searching and it'll be done. I do not expect to be done in one week, but I think I should have something working for 342.

It felt good this week to clear out some long-time bugs doing code cleaning, so I'd like to focus on that as well.

____________________________
Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

8a8e99 No.11734

>>11732

No immediate noticeable difference with the thumbnail mode on, if its pulling from my thumbnail library its pulling them off an nvme though so reguardless its going to be fast

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

9f0fef No.11737

hey dev i was just wondering wether or not the functionality to get/download the images by searching through them in a booru style fashion will be available through the api or if that will stay exclusive to the hydrus booru server

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

08f8cb No.11739

Hey dev, any chance of you adding in a way to exclude images with certain tags/ratings from searches unless they're explicitly searched for?

Sort of like the tag censoring, but with the images that fall under those tags being excluded.So, if I were to have a namespace entitled lewd: and set that to be excluded, no images whose tags included lewd: would show in searches unless I explicitly included the lewd: namespace into my current search

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

b84c1b No.11742

>>11739

This would be nice to have

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

389540 No.11743

Why does Hydrus need two sizes of thumbnails anyway? 🤔

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

3a6419 No.11744

Can you use the API to access hydrus features other than importing and exporting images/tags?

I was thinking of making a script to automate the IQDB lookup script for multiple files.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

1af804 No.11746

341 is broken for me, endless database errors

DBException

OperationalError: unable to open database file

Traceback (most recent call last):

File "include\HydrusThreading.py", line 342, in run

callable( *args, **kwargs )

File "include\ClientGUICanvas.py", line 4172, in process_in_thread

HG.client_controller.WriteSynchronous( 'content_updates', { service_key : [ content_update ] } )

File "include\HydrusController.py", line 790, in WriteSynchronous

return self._Write( action, True, *args, **kwargs )

File "include\HydrusController.py", line 228, in _Write

result = self.db.Write( action, synchronous, *args, **kwargs )

File "include\HydrusDB.py", line 976, in Write

if synchronous: return job.GetResult()

File "include\HydrusData.py", line 1519, in GetResult

raise e

include.HydrusExceptions.DBException: OperationalError: unable to open database file

Database Traceback (most recent call last):

File "include\HydrusDB.py", line 580, in _ProcessJob

result = self._Write( action, *args, **kwargs )

File "include\ClientDB.py", line 12306, in _Write

elif action == 'content_updates': self._ProcessContentUpdates( *args, **kwargs )

File "include\ClientDB.py", line 7990, in _ProcessContentUpdates

self._DeleteFiles( service_id, hash_ids )

File "include\ClientDB.py", line 3121, in _DeleteFiles

self._CacheSpecificMappingsDeleteFiles( service_id, tag_service_id, existing_hash_ids )

File "include\ClientDB.py", line 2353, in _CacheSpecificMappingsDeleteFiles

self._c.executemany( 'DELETE FROM ' + cache_current_mappings_table_name + ' WHERE tag_id = ? AND hash_id = ?;', ( ( tag_id, hash_id ) for hash_id in current_hash_ids ) )

sqlite3.OperationalError: unable to open database file

Database Traceback (most recent call last):

File "include\HydrusDB.py", line 580, in _ProcessJob

result = self._Write( action, *args, **kwargs )

File "include\ClientDB.py", line 12306, in _Write

elif action == 'content_updates': self._ProcessContentUpdates( *args, **kwargs )

File "include\ClientDB.py", line 7990, in _ProcessContentUpdates

self._DeleteFiles( service_id, hash_ids )

File "include\ClientDB.py", line 3121, in _DeleteFiles

self._CacheSpecificMappingsDeleteFiles( service_id, tag_service_id, existing_hash_ids )

File "include\ClientDB.py", line 2353, in _CacheSpecificMappingsDeleteFiles

self._c.executemany( 'DELETE FROM ' + cache_current_mappings_table_name + ' WHERE tag_id = ? AND hash_id = ?;', ( ( tag_id, hash_id ) for hash_id in current_hash_ids ) )

sqlite3.OperationalError: unable to open database file

putting back 340 I get no issues

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

2f5629 No.11748

>>11732

> help->debug->gui actions->thumbnail experiment mode

> Try loading up some random files in your client and let me know if you get any errors or if any thumbs take an extremely long time to load.

So far, this seems to be working just fine here. I don't notice any difference to the extent where it's pretty hard to tell if the option is doing anything at all, actually.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

bf140b No.11750

thumbnail experiment mode loads them nearly instantaneously for me. it'd be great to be able to free up that space up.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

355098 No.11751

File: 5351d354c75bf80⋯.jpg (170.99 KB,1440x862,720:431,Hydrus.jpg)

File: cdb04b8d1e561aa⋯.jpg (147.48 KB,1440x860,72:43,Hydrus2.jpg)

>>11732

Having issues with the new thumbnails mode on my 11 year old toaster of a system. They lag loading slightly while Hydrus is doing other stuff like subscription downloads. Also did this weird thing with my background color on the duplicates page…

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

44eedb No.11753

I'm getting a segfault in Python when clicking the duplicate '2 - discovery' play button


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `python -OO /opt/hydrus/client.pyw'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f1a954cb223 in sqlite3VdbeFindCompare () from /usr/lib/libsqlite3.so.0
[Current thread is 1 (Thread 0x7f1a6257a700 (LWP 9481))]
(gdb) bt
#0 0x00007f1a954cb223 in sqlite3VdbeFindCompare () at /usr/lib/libsqlite3.so.0
#1 0x00007f1a9542e157 in sqlite3BtreeMovetoUnpacked () at /usr/lib/libsqlite3.so.0
#2 0x00007f1a954c1b86 in sqlite3VdbeExec () at /usr/lib/libsqlite3.so.0
#3 0x00007f1a954c6920 in sqlite3_step () at /usr/lib/libsqlite3.so.0
#4 0x00007f1a955a10f0 in pysqlite_step ()
at /usr/lib/python3.7/lib-dynload/_sqlite3.cpython-37m-x86_64-linux-gnu.so
#5 0x00007f1a955a406b in _pysqlite_query_execute ()
at /usr/lib/python3.7/lib-dynload/_sqlite3.cpython-37m-x86_64-linux-gnu.so
#6 0x00007f1a95da7a78 in _PyMethodDef_RawFastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#7 0x00007f1a95dd30ff in _PyMethodDescr_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#8 0x00007f1a95e1e479 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#9 0x00007f1a95dd4149 in () at /usr/lib/libpython3.7m.so.1.0
#10 0x00007f1a95e1a271 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#11 0x00007f1a95da72eb in _PyFunction_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#12 0x00007f1a95e19c42 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#13 0x00007f1a95d60b99 in _PyEval_EvalCodeWithName () at /usr/lib/libpython3.7m.so.1.0
#14 0x00007f1a95d61dec in _PyFunction_FastCallDict () at /usr/lib/libpython3.7m.so.1.0
#15 0x00007f1a95d71908 in _PyObject_Call_Prepend () at /usr/lib/libpython3.7m.so.1.0
#16 0x00007f1a95d6233b in PyObject_Call () at /usr/lib/libpython3.7m.so.1.0
#17 0x00007f1a95e1b43c in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#18 0x00007f1a95d60b99 in _PyEval_EvalCodeWithName () at /usr/lib/libpython3.7m.so.1.0
#19 0x00007f1a95d61dec in _PyFunction_FastCallDict () at /usr/lib/libpython3.7m.so.1.0
#20 0x00007f1a95d71908 in _PyObject_Call_Prepend () at /usr/lib/libpython3.7m.so.1.0
#21 0x00007f1a95d6233b in PyObject_Call () at /usr/lib/libpython3.7m.so.1.0
#22 0x00007f1a95e1b43c in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#23 0x00007f1a95da72eb in _PyFunction_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#24 0x00007f1a95e19c42 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#25 0x00007f1a95d61c0b in _PyFunction_FastCallDict () at /usr/lib/libpython3.7m.so.1.0
#26 0x00007f1a95d71908 in _PyObject_Call_Prepend () at /usr/lib/libpython3.7m.so.1.0
#27 0x00007f1a95d6233b in PyObject_Call () at /usr/lib/libpython3.7m.so.1.0
#28 0x00007f1a95e1b43c in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#29 0x00007f1a95da72eb in _PyFunction_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#30 0x00007f1a95e19c42 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#31 0x00007f1a95da72eb in _PyFunction_FastCallKeywords () at /usr/lib/libpython3.7m.so.1.0
#32 0x00007f1a95e19c42 in _PyEval_EvalFrameDefault () at /usr/lib/libpython3.7m.so.1.0
#33 0x00007f1a95d61c0b in _PyFunction_FastCallDict () at /usr/lib/libpython3.7m.so.1.0
#34 0x00007f1a95d71908 in _PyObject_Call_Prepend () at /usr/lib/libpython3.7m.so.1.0
#35 0x00007f1a95d6233b in PyObject_Call () at /usr/lib/libpython3.7m.so.1.0
#36 0x00007f1a95e90aa7 in () at /usr/lib/libpython3.7m.so.1.0
#37 0x00007f1a95e38fa5 in () at /usr/lib/libpython3.7m.so.1.0
#38 0x00007f1a96196a9d in start_thread () at /usr/lib/libpthread.so.0
#39 0x00007f1a960c6b23 in clone () at /usr/lib/libc.so.6

Running the Arch Linux AUR package with Python 3.7.2

Problem occurred with Hydrus 340 as well.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

389540 No.11755

I think the gelbooru downloader grabbing the rating tag is broken because it seems to give everything the mature rating.

This for example got a mature rating but it's clearly marked as safe.

https://gelbooru.com/index.php?id=2475423&page=post&s=view

(also is mature the new explicit or what)

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

389540 No.11756

The new thumbnail system runs fine here but I have a pretty powerful system. A 9700k, 16gb ram and the thumbs are on an encrypted NVMe SSD with the originals on an encrypted 5400rpm SATA HDD.

However I did notice that while I had a page open that was constantly added to through subscription updates, some thumbnails didn't load and were black. Once I clicked on them they loaded, though.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

355098 No.11757

>>11756

> some thumbnails didn't load and were black. Once I clicked on them they loaded, though.

Yeah that's been an issue for me since before tumblr wiped all the porn. It's not really a big deal so I never mentioned it.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

3f70b4 No.11765

Thank you based dev.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

c7fd47 No.11767

I really want to try hydrus but you have 3 detections on virus total ;-;

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11768

>>11737

Yep, that should be what I am working on next. It will support simple tag searches to begin with and give basic file metadata and file and thumbnail fetches. If you want to talk to the API to do this stuff, then with a bit of wizardry on your end you could write up a simple booru much better than my old prototype, with proper tag search.

You'll be able to restrict API access a bit as well, btw, saying that it only works for searches that include certain tags (think maybe a tag something like "share this on my booru"), so you don't have to give full access to every booru 'window' that is looking at your client.

Let me know how this works for you. I haven't been able to find time to improve my old booru, so I'd love it I could offload that to other users who can do it justice. It is much easier for me to expand the API to now allow filesize searches or whatever than to keep up with a whole booru codebase.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11769

>>11739

>>11742

That's an interesting thought, and I hadn't considered approaching file censorship/hiding from that perspective before.

My original thought here has been to add multiple local file services, so you can stick all your nsfw in a completely different file domain. This way, when you type in 'fu' for a 'futurama' tag search with a family member looking over your shoulder, you won't get a variety of other 'fu-' beginning tags that you might not want appearing in the results.

I haven't made much progress on this because it will be a complicated job. It is still the best solution here–although I do like you idea of only allowing certain files if the search itself includes a magic word, and I'll think about it a bit more, I still think it will be a bit whack-a-mole trying to keep it secret/separated.

The old solution here for now has been to run two clients–one for safe stuff, and one for spicy stuff. It is clunky, but it works.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11770

>>11743

When I first wrote it, SSDs were still pretty rare, and I was working with the slower PIL library, so many thumbnail actions were still pretty slow. It was nice to have a fast backstop thumbnail at 200x200 to regen new 'resized' thumbnails rather than going back to the master file to regen. There's also some Hydrus Network legacy stuff to fit in, since a file repository always transfers the master to the client for resizing to whatever is desired locally.

Now the hardware and software has all moved on, fitting a more dynamic "Hey, that thumb we loaded is the wrong size, let's regen a new one from an appropriate source, save it back to disk, and deliver the correct one to the user all in one step" action is more feasible. We also have much bigger hydrus installs (I've got more than a million files on my IRL client, so I'm up to my ears in thumbs atm), so it is worth putting the effort in.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11771

>>11744

Please check the file search stuff I'll be rolling out in this final 1.0 phase. It might be able to do what you want–something like you tag a file in the client with 'do iqdb on this', and your script fetches those files and does its work, writing back what it finds.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11772

>>11746

Thank you for this report. I reversed an old memory-only db mode in 341. SQLite will now attempt to write a journalling file to your temp directory when it writes changes–is there anything about your temp dir (maybe a ramdisk?) that might cause permission problems here?

I assume you were able to boot the client and update to 341, right? So these ProcessContentUpdate calls were you trying to manage some tags once it had booted? It is odd that client.db presumably accepted the simple 'You are now 341' write but client.mappings.db had trouble. This is odd, but did your client.mappings.db somehow get set read-only? OperationalError is a pretty low-level error.

What happens if you run the sqlite3 executable in the install_dir/db directory and then type:

.open client.mappings.db
SELECT * FROM sqlite_master;
.exit

Is it ok reading that, or does that also give an error?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11773

>>11772

Thinking this through more, it doesn't make sense that the file would be read only and also 340 works ok for you. Please explore any permission errors in your temp dir, but I'll also do a bit of searching on my end.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11774

>>11751

Thank you. I should be able to clear up most of the lag here if I commit to this new system. The band of colour is odd–did you happen to turn on help->debug->debug modes->thumbnail debug mode? That shades the 'tiles' of the thumbnail grid differently so I can see where the borders are.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11775

>>11753

Thank you. This is the last crash in this system. Search distance 0 (exact match) works ok. I worked with another Linux user today and we believe we have this crash fixed for 342. It seemed to be unhappy with how I was doing some similar-hash comparison during the SQLite query iteration that was fetching them. Please let me know how that works for you.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11776

>>11755

Thank you for this report. The parser appears to be pulling rating info from the wrong location. I will fix this myself and update it for next week.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11777

File: ad6d3599a6c489a⋯.png (5.15 KB,200x200,1:1,hydrus.png)

>>11756

>>11757

Can you talk more about this black thumbnail issue? Could they have been dark with the hydrus image (pic related) transparently on top? This is the default image used when hydrus can't find a thumb on your disk, but it usually comes with an error popup. Normally, thumbs should be ready to display before a 'this file is imported, throw it on screen' call goes through, but I have been changing some of this recently.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11778

>>11767

Thank you for this report. I cannot promise with absolute certainty that my dev machine is clean, but every instance of this happening so far has been due to false positives. My code is unprofessional and unsigned, with unusual frozen python packaged into InnoSetup, and it has a bunch of downloading and uploading and UPnP components that I presume may appear to a scanner to be bad business. Every 4-6 months, a user reports that Kaspersky or Avast or whatever considers a particular dll (or often the upnpc exes in install_dir/bin) to be bad, and then it goes away in a bit when they update their definitions.

I recommend you wait a bit and scan the latest release in a few weeks. My guess is the report will be different. I would obviously be very interested in any evidence of a true-positive.

Out of interest, does the scan say which files are bad? If it is upnpc again, I can point you at the files I use on their original site, and you can even delete them and still run hydrus fine.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

3a6419 No.11779

>>11769

>I haven't made much progress on this because it will be a complicated job.

Couldn't you make so the user can blacklist/whitelist files based on tags and use those as search terms hidden from the user?

Like blacklisting rating:explicit and whitelisting "rating:safe" makes it so whever you search something, it automatically also searches for "rating:safe" and "-rating:explicit" as well, except it doesn't show it to the user?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

3a6419 No.11780

>>11769

>>11779

Also, I don't know how hydrus works, but if it just throws basic SQL around, how about having a whitelist/blacklist string and appending it to the end of searches?

Every time the user adds another whitelist/blacklist it automatically generates the string, then when you do searches you just use python concatenation to do "SQLshit + search_terms + whiteandblacklist".

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

5af3c2 No.11781

>>11773

I tested a thing, backup with FFS and then deleted the install folder. Install 341 again. Quick check that permissions look fine (removed the "read only" that windows loves SO MUCH) and tried to start the client:

A serious error occurred while trying to start the program. The error will be shown next in a window. More information may have been written to client.log.

2019/03/03 03:20:45: Traceback (most recent call last):

File "include\ClientController.py", line 1351, in THREADBootEverything

self.InitModel()

File "include\ClientController.py", line 605, in InitModel

HydrusController.HydrusController.InitModel( self )

File "include\HydrusController.py", line 501, in InitModel

self.db = self._InitDB()

File "include\ClientController.py", line 99, in _InitDB

return ClientDB.DB( self, self.db_dir, 'client', no_wal = self._no_wal )

File "include\ClientDB.py", line 191, in init

HydrusDB.HydrusDB.init( self, controller, db_dir, db_name, no_wal = no_wal )

File "include\HydrusDB.py", line 200, in init

self._InitDB()

File "include\HydrusDB.py", line 437, in _InitDB

self._CreateDB()

File "include\ClientDB.py", line 2987, in _CreateDB

self._AddService( service_key, service_type, name, dictionary )

File "include\ClientDB.py", line 335, in _AddService

self._CacheSpecificMappingsGenerate( file_service_id, service_id )

File "include\ClientDB.py", line 2452, in _CacheSpecificMappingsGenerate

self._CreateIndex( cache_pending_mappings_table_name, [ 'tag_id', 'hash_id' ], unique = True )

File "include\HydrusDB.py", line 375, in _CreateIndex

self._c.execute( statement )

sqlite3.OperationalError: unable to open database file

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

355098 No.11782

>>11774

>The band of colour is odd–did you happen to turn on help->debug->debug modes->thumbnail debug mode?

Ah, I did! I just saw thumbnails and thought that was it. Trying the correct option now, they seemingly load around the same speed either way.

>>11777

My background is set to black so it's possible. The title/artist bar and inbox icon still res'd in the grid like they should, just the thumbnail wasn't there. But clicking on the tile to highlight it instantly loaded the thumbnail. I noticed it when doing tumblr downloads and stuff so it was mostly on pages with a fair number of files ~100-500. Ironically now I can't replicate it after moving to 341. I was sitting on 339 and don't recall seeing it but I wasn't really using the client much either.

If I had to describe it, it was like when you normally load up a page of files and sometimes certain thumbs take longer than the rest of their row to load. So the slow loading thumb will be the last thumb to load despite it being in the middle of your scroll space. It was pretty much that except the thumb just never loaded. Actually, now that I think about it, something else seemed off too, like the border was darker or something. I'm probably just mis-remembering that.

I'll let you know if I see it again.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

c7fd47 No.11783

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

389540 No.11784

>>11777

My background is black so it is possible that image was there, but I don't know. There were no error popups though.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

8a8e99 No.11785

hey hdev, I thought I would bring this to your attention.

http://comicrack.cyolito.com/

So far, outside of acdsee, this is the best manga program I have found. as much as I like it, it seems abandoned. Ok, not horrible but not good either.

I think that when you do get rar/zip/manga version of the program going, you may want to look at this for inspiration, there are aspects I really like, like tabbed comics so you can have more than one book open at a time, personally this is VERY useful for when I have some shit open that i'm reading, I check the new stuff and I want to read that more, but I want to go back to the older book

How I am able to have multiple searches/lists open at once, this is something that may even be useful for images, but I think manga suits it more

especially for manga, the way it does tags is REALLY good, as there is a fairly limited range of tags you can do for it unless you want to really go into details, at least far more limited then with single images where people start to tag eye color and hairstyles. I think this program could very well be a blueprint for how to handle manga.

granted, this program will let you keep shit where it is, which if it was hydrus, that wouldn't fly, however, use parseable is a requirement I have for using hydrus for manga.

thought you may want to check it out.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

43a746 No.11789

>>11732

Problem with dark mode: the white boxes for tags and others don't change color, they stay white.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11792

>>11781

Right. If a fresh install cannot create a new db, and given the use of the OS's temp location for journalling files is I think the only thing I really changed, I suspect this is a temp directory problem.

Can you try opening the sqlite3 executable in install_dir/db and doing this:

.open test.db
CREATE TABLE t ( n INTEGER );
INSERT INTO t VALUES ( 13 );
SELECT * FROM t;
.exit

You should be able to paste that in line by line. If a test.db file is created, you can delete it afterwards.

If a very simple sqlite db on stock settings cannot be created here, please check that your temp location is 'correct'. I had presumed you were on Linux because this is where users have had temp directory problems before, but if you are on Windows, you are probably looking at C:\users\(You)\AppData\Local\Temp . Is there anything odd about that directory? Is C drive completely full, or could there be any user permissions problem related to writing to that folder? Has C drive (or wherever your temp folder is) had any problems recently, maybe that could have set a 'dirty bit' on the drive?

If 340 is still ok for you, you may get a hint at what SQLite is using for its temp location by looking at what hydrus has under help->about. It should be that …\local\temp location, but if you have an unusual environment, perhaps it has been relocated?

If the test.db goes fine, please check your temp location anyway and let me know and I'll see what else might be going on here.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

044f62 No.11793

341 and 340 (haven't tried earlier versions) are prone to locking up on Arch Linux when processing repository updates, forcing me to kill hydrus.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

44eedb No.11794

>>11775

You're a bloody legend mate.

>>11779

>>11780

I've observed that the blacklists on boorus just run a computationally cheap filter after the query. The query gets 30 posts per page normally, then the filter knocks out 9 on page 1, 13 on page 2, etc. Hydrus doesn't have to paginate the results so this is way less ugly than a web page.

>>11793

Repo updates are quite slow on my Arch system with /home on btrfs on a hard disk (I usually do them manually in services->review services where the progress bar locks up but eventually pulls through). It doesn't locked up more than usual with v340+ for me.

How many hours are you waiting for the 30 min shutdown process?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11796

Sorry lads, I have had a busy week and haven't found time to catch up with all my messages. I'll try to do a bit tomorrow morning before putting the release together, otherwise it may delay to Saturday.

>>11781

For tomorrow's 342, I have added a launch parameter to the program, '–no_db_temp_files', that will tell the database to return to 340's behaviour and operate entirely in memory. Please try this when it is convenient and let me know how it works.

If it works, that means this is a temp directory permissions/whatever problem. If at that point you are still interested in figuring out what's going on here, we can, or you can just stay in memory. If so, you might like to disable db vacuuming in the options.

If you are not sure how to launch the program with this parameter, try going to your install directory and creating a new shortcut to client.exe. Edit the shortcut's options and change the 'Target' line to:

"X:\path to hydrus\client.exe" --no_db_temp_files

If you already use the '-d="path"' parameter to relocate your db, it is fine to use both.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.
Post last edited at

fe42ee No.11810

>>11796

help>about has shown the issue… it used the install directory as temp. Don't ask how, this is as far as I know. In the settings I tried to put it in the correct folder (nope) and it worked in a new, different folder purpose made. 342 still spits errors with that, HOWEVER! –no_db_temp_files does work

The correct temp directory seems fine, several temporary looking files and there's plenty of free space so I really have no idea wtf is going on. Also I don't really have other issues (that I know of at least!) so uh… yay I guess?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11826

>>11780

>>11779

Thanks. Yeah, putting in an invisible 'explicitly required in search to appear' is a nice simple solution to the problem I had not thought of before. I have usually been thinking of 'sfw' modes and so on that would try to blacklist or whitelist according to complicated rules.

The complicated job is making multiple local file services. There would be a bunch of new verbs added to the program like moving files from one local domain to another, and formerly fairly simple ideas like deleting and undeleting would get an order more complicated.

The reason why I generally favour the latter over the former is all the edge cases. My running assumption in these cases is that you have a normie looking over your shoulder, so you don't want any nasty surprises. There are many places where whitelists and blacklists can go wrong, from false-positive tags to autocomplete suggestions (which are too complicated to filter), or even something like the 'recent tags' column in manage tags. If I have a strict database split between Content A and B, I can enforce a hard division on the caches that back up all these things, but if instead it is all mixed together, it is whack-a-mole.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11827

>>11783

Thanks. I see one of those definitions is unhappy with UPnP. Feel free to delete the three upnpc executables from install_dir/bin–they aren't needed to run the program, only if you want to explictly tell some hydrus service to keep a NAT port open for itself. They are just copies of the exes available here:

http://miniupnp.free.fr/

The others, I suspect, are as I said before. I haven't had a true positive here yet, but I'd appreciate any future information you have here.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11828

>>11785

Thank you, I really appreciate this recommendation and your thoughts about it. I will definitely revisit this when work on hydrus multi-page support starts.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11829

>>11789

Thank you for this report. My dark mode is a bit of a fudge–I only have control over the colour of my custom objects, whereas everything else (like window borders and backgrounds and button colours) is supposed to be inherited from your OS's current system colours. If you have a window manager that changes your OS colours, hydrus's dark mode switching can be helpful to make sure that my custom objects also change with it.

There's a very speculative long-term plan to move from wxPython to Qt. If this ever happens, theming support will get significantly better, but until then it is all duct-tape, I am afraid.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11830

>>11793

Thank you for this report. Are the updates processing during idle time or shutdown? If you have problems during idle time, can you change your settings under options->maintenance and processing to be shutdown-only, and, say, only 5 minutes? If that relatively small amount of work can go through, can a bit more, like 15 mins? Is there a significantly 'cleanup' lag, like 5 mins of work takes 3 mins to commit?

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.

229563 No.11831

>>11810

Thank you. That is very strange. Here's SQLite's and Python's info on this:

https://sqlite.org/tempfiles.html#temporary_file_storage_locations

https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir

So, for some reason, a different folder was not permitting a program launched from it to access any system temp location. Windows has simpler temp structure than Linux, so perhaps this really just meant stopping you from accessing the one under AppData/Local/Temp.

Could somehow that directory (or say the client.exe) have got a compatibility mode or some other protected state applied to it? I am afraid this is way beyond my expertise.

Disclaimer: this post and the subject matter and contents thereof - text, media, or otherwise - do not necessarily reflect the views of the 8kun administration.
Post last edited at



[Return][Go to top][Catalog][Nerve Center][Random][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / random / 93 / biohzrd / hkacade / hkpnd / tct / utd / uy / yebalnia ]