[ / / / / / / / / / / / / / ] [ dir / had / htg / leftpol / maka / polmeta / russian / tijuana / zoo ]

/hydrus/ - Hydrus Network

Bug reports, feature requests, and other discussion for the hydrus network.
Name
Email
Subject
Comment *
File
* = required field[▶ Show post options & limits]
Confused? See the FAQ.
Embed
(replaces files and can be used instead)
Options
Password (For file and post deletion.)

Allowed file types:jpg, jpeg, gif, png, webm, mp4, swf, pdf
Max filesize is 16 MB.
Max image dimensions are 15000 x 15000.
You may upload 5 per post.

New user? Start here ---> http://hydrusnetwork.github.io/hydrus/

Current to-do list has: 1,184 items

Current big job: finishing login and domain managers and overhauling the downloader

YouTube embed. Click thumbnail to play.

622ccf No.7296

windows

zip: https://github.com/hydrusnetwork/hydrus/releases/download/v283/Hydrus.Network.283.-.Windows.-.Extract.only.zip

exe: https://github.com/hydrusnetwork/hydrus/releases/download/v283/Hydrus.Network.283.-.Windows.-.Installer.exe

os x

app: https://github.com/hydrusnetwork/hydrus/releases/download/v283/Hydrus.Network.283.-.OS.X.-.App.dmg

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v283/Hydrus.Network.283.-.OS.X.-.Extract.only.tar.gz

linux

tar.gz: https://github.com/hydrusnetwork/hydrus/releases/download/v283/Hydrus.Network.283.-.Linux.-.Executable.tar.gz

source

tar.gz: https://github.com/hydrusnetwork/hydrus/archive/v283.tar.gz

I had a tough week, but it worked out ok in the end. There are a variety of different improvements this week.

highlights

Subscriptions now deal with and present their new multiple queries a little better. They will also notify you when a query dies. I improved the timing check calculations, but some queries are still dying after their initial check–if this happens to you, please tell them to 'check now' again, and they should revive. I am going to keep working on this.

The different time periods in subscription and thread watcher check options can now stretch to a maximum of ten years! If you want to quietly watch an artist or thread only once per year or more, just in case it is updated, please check this out.

The pixiv 'artist' downloader is turned back on! I made a mistake last week–it was only the tag gallery that has changed. If the artist gallery updates in the future, I will turn it off again, so I suggest you get your artist stuff now!

I added support for an unusual type of flash file. If you have a number of broken 'ZWS' .swf files in a folder somewhere, please try importing them again and let me know how you get on.

I fixed a bug with editing import folders that have existing 'quick namespaces' entries. Also, if you have multiple import folders, you can now manually check them all at once with a new menu entry.

The new StringMatch parsing object is done. I've added it to the 'html formula', so you can quietly filter parsed texts if they do not fit some common character limits or lengths or even a regex. It will appear in several other places in the future to do similar filtering and veto condition testing.

full list

- subscription popups show a bit more info about their individual queries

- subscription popups hide the sub-query info in text and file buttons if it is the same as the subscription name

- subscriptions now work through their queries' downloading phases in random order (so a heavy query named 'aardvark' won't repeatedly choke the sub's other queries if bandwidth is tight)

- if a subscription query dies, a notification popup will now be made

- changed a logical edge case that may have been killing some subscription queries after the first sync

- re-enabled pixiv artist search, as it still uses the old gallery style

- added tentative support for 'ZWS' flash files

- if there is more than one import folder, the 'check import folder now' submenu now has a 'check all' entry

- TimeDelta controls that have the 'days' entry (as in thread/subscription checker options panels) now have a max period of 10 years for those _really_ slow threads

- fixed an issue where the 'quick namespaces' section of the new filename tagging options panel was not initialising correctly when it loaded data from a previously serialised import folder (this was the "unhashable type: 'list'" bug in the edit import folder panel)

- the filename tagging dialog now works on the new panel system. it also has an entry under options->gui: 'local_import_filename_tagging'

- polished StringMatch object

- polished EditStringMatchPanel

- html formulas now have a string match object

- improved parsing error system

- miscellaneous improvements to the parsing system

- misc fixes to StringMatch and parsing system

- the main gui will no longer interpret text drops on child frames as url drops. this fix just quashes the drop–it doesn't allow the initially desired drop on the child frame to occur. I will keep thinking about this

- the dupe filter's 'no media to display' text now says 'Looking for more pairs–please wait'

- the 'remove' media action is now labelled 'remove from view' to reduce 'is it a delete?' confusion

- fixed an icon initialisation order problem that meant pre-boot errors were unable to display

- fixed a graceful-shutdown-on-boot-error bug

- the custom 'listboxtags' controls now post an event on their contents changing

- the tag censorship panel now updates its status text when its listboxes change due to a mouse double-click event

- fixed file import 'source time' column sometimes showing '3e-05 years' kind of garbage that was due to a missed float->int conversion

- cleaned up how some page names are thrown around, which should stop some stray [USER] prefixes slipping though

- the adminside petition management checklistbox will now escape mnemonic characters (typically '&') and hence display them correctly on sibling pairs and so on

- wrote a 'help my client will not boot.txt' that explains the new debug builds and some other common fixes to try

- 'all known files' file search domain is now hidden behind advanced mode

- fixed another instance of flashwin causing a failed boot

- cleaned up some old broken unit tests

- some misc help improvements

- updated Windows build to new version of sqlite

next week

I will now work on a 'URLMatch' object, which will let the client think about domains and URLs in a more clever way, which will be of use to the new downloader engine and login manager.

6227d7 No.7298

File: e3865d312a1d142⋯.png (219.78 KB, 1920x1040, 24:13, Untitled.png)

Just a quick question boss, how does hydrus currently deal with memory garbage collection?

Im asking this because ive noticed that hydrus memory usage tends to increase during heavy use but it just stays up there even when theres no need. pic related.


37386d No.7300

File: 6a8519b4de2a3af⋯.gif (1.13 MB, 255x255, 1:1, 6a8519b4de2a3af167bbea6162….gif)

>hydrus says it's done processing updates

>db read is still locked for minutes afterwards, making it impossible to import or exit

So is it done or not?


8efc43 No.7301

File: acebdc94efa82d2⋯.png (172.3 KB, 1920x1080, 16:9, Untitled.png)

Nice, I can know keep all my swf porn shit tagged on hydrus.

Thanks Devanon.

Shortcut support when


49548f No.7306

YouTube embed. Click thumbnail to play.

I made an alternate Linux build for people having trouble with the official build. A few things are still weird, but overall it works better for me than the official. Run from source works slightly better, but pip is a headache. Built on Linux Mint 18, so it should also work on Ubuntu 16.04.

If you want open externally to work, you will have to manually set the open program for each mimetype in options->files and trash. For some reason, when Hydrus invokes xdg-open it opens a program in WINE.

https://www.mediafire.com/file/e3k35dx1llgseez/hydrus-283.tar.gz

I'll post one of these every week until the official Linux build works on my machine. If anything is broken, tell hydrus_dev (and thank him while you're at it). He was kind enough to give me his build command, but I don't know Python. If something is broken, I'm probably just as clueless as you are.


8bd032 No.7322

Ok holidays and birthday shit are over, I can pay attention to the program for a few weeks now. I should have sent you the image and thumb from

post No.7184

and with the issue of imports not happening

page of pages, that works

drag and drop from disk, that works

it seems, if my import folder is correct, that importing new images ended on the 14th, on the 14th there seems to be a dump of errors that all look like

2017/11/14 20:05:48: FFMPEG couldn't figure out the mime for: (File in the temp folder that I redacted for post)

lookin in the temp folder there appears to be 62,000~ files with the same naming scheme, 33gb in total.

putting one of them through irfanview I got a gif, and importing that gif to the program seems to result in the image already being in there… so i'm honestly not sure what the fuck happened here.

Oh, and importing that file I got this message in the log

2017/11/25 02:20:16: 1 file was parsed successfully.

2017/11/25 02:20:30: Could not delete the temporary file (file again giving info I don't want on the forum)

I imported another file and also got the same, this time not changing the file type as per irfanview…

I think all these images may actually be in the archive already but for some reason they were tossed in the temp folder.

So, any ideas on getting the import folders working again?

Also, what the fuck are the 62k images in the temp folder?

they are all named something along the lines of hydrusmmeb1t


622ccf No.7329

File: 08eec7177933631⋯.jpg (449.19 KB, 1810x1716, 905:858, 08eec7177933631923165818d9….jpg)

>>7298

Sometimes it is my fault, sometimes it isn't. Since I write this in python, anything I've discarded is usually cleaned up super fast, but sometimes I don't discard things fast, or the way I load things is inefficient through redundancy. I use a lot of caches to speed things up, and since memory is usually the cheapest resource, I don't focus on keeping that side of things lean. The recent expansions that allow for more pages to be open and more thumbnails to draw and so on have aggravated the recent bloat.

There's also some stuff that may be memory-leaky, although I am not sure. That stuff mostly occurs in places out of my control, so I'll have to do more profiling to figure it out.

This is on my radar, and I have plans to reduce memory usage across the program. For now, a good way to recoup a decent amount of memory is to go undo->closed pages->clear all. Closed pages are really just hidden until they are deleted, so if they have ten thousand files all with complicated tags and file domains and so on, the memory does add up.

There's a small program-wide memory clearing job that occurs every 60s and a larger one that occurs every 300s. Sometimes just waiting five minutes will clear up a bunch of memory, particularly on machines with larger resolutions. I think 'closed' pages are automatically deleted every hour.

>>7300

The 'commit' stage of database transactions is unfortunately out of my hands. I basically say "Hey DB, here are a bunch of changes, go save them", and it then vanishes for a bit and only responds when it is finally done. For very large jobs that can be many minutes, particularly on HDDs (as opposed to SSDs). If you are on an HDD or another drive that takes a while to commit big jobs, please check your options->maintenance and processing settings. You might like to set processing to only occur on shutdown, so you can manually control when those jobs occur.

>>7301

Which sort of shortcuts? If you mean for playing the flash game, please put your mouse cursor inside the flash window and then try. I have a test set up based on mouse location so your arrow buttons either do or do not navigate the larger media viewer and so on. Let me know if that doesn't work for you!

>>7322

Thank you for this report. When hydrus imports a file, it temporarily copies it to your temp folder with the 'hydrusblah' name. It sounds like your client was importing files from there ok but was unable to clean up the temporary file afterwards, maybe due to a permissions issue.

If you import a file manually, say from your desktop, does the client still have a problem cleaning up the temp path afterwards? What is the temp path? Does it line up with what is in help->about? For Windows, it is typically, C:\users\(your username)\appdata\local\temp.

I've updated the error handling to print more error information on a temp file delete failure for next week. Please let me know what the longer error is when you update.

You might like to mass-import all those temp files just to make sure you haven't missed anything, but if they are all 'already in db', then you can safely delete them.


f009d2 No.7335

>>7329

>For now, a good way to recoup a decent amount of memory is to go undo->closed pages->clear all. Closed pages are really just hidden until they are deleted, so if they have ten thousand files all with complicated tags and file domains and so on, the memory does add up.

>There's a small program-wide memory clearing job that occurs every 60s and a larger one that occurs every 300s. Sometimes just waiting five minutes will clear up a bunch of memory, particularly on machines with larger resolutions. I think 'closed' pages are automatically deleted every hour.

Interesting. Thanks for the tip


6227d7 No.7336

>>7329

Thanks for the explanation, boss


8bd032 No.7342

>>7329

yea it lines up with the typical path

going to import them all a bit later to make sure

and ill keep an eye on it for tossing errors and not removing temp files

Most of the places I manually import from I handled the cleanup myself because there is a reason i'm importing them manually but just tested it, its able to clean the path if I tell it too.

anything on the first issue? The temp files were honestly something I stumbled across when trying to find out why the hell the program will no longer use my import folders with automatic import or manual importing.


622ccf No.7347

>>7342

Ah–sorry. This sounds stupid, but have you checked if import folders are paused under services->pause?

If that seems ok, I guess you don't have any obvious errors in your log? Are you running the client in 'no-daemons' mode from the command line?

I'll be adding a subscription-like 'working' popup for import folders soonish, btw, which may shed light on what is going on here if we can't figure it out now.


8bd032 No.7348

>>7347

Yea, pause imports was ticked… not sure how or why it was though. thanks.

on that note, would it be possible to add an override to manually importing the import folders when that is checked?


622ccf No.7352

>>7348

Great. There are debug reasons for the global pauses to be firm (the import folder objects are not even loaded in this case, which is a good stopgap for a 'shit's broken' situation), but I've set it so if you try to manually run an import folder while they are globally paused, it'll notify you in a popup.


8efc43 No.7359

File: 33ad75cb0b30fc9⋯.png (276.02 KB, 750x750, 1:1, 33ad75cb0b30fc942d66e52bca….png)

>>7329

Nah, I just meant generic shortcuts (to .exe files and whatnot), so you can have virtually any application in hydrus.



[Return][Go to top][Catalog][Nerve Center][Cancer][Post a Reply]
Delete Post [ ]
[]
[ / / / / / / / / / / / / / ] [ dir / had / htg / leftpol / maka / polmeta / russian / tijuana / zoo ]