/hydrus/ - Hydrus Network

Now on another topic, Hydrus loaded, and I tried to change times per watcher,

AttributeError

'ManagementPanelImporterMultipleWatcher' object has no attribute '_watchers'

File "include\ClientGUICommon.py", line 549, in EventButton

self._func( *self._args, **self._kwargs )

File "include\ClientGUIManagement.py", line 2207, in _SetOptionsToWatchers

for watcher in self._watchers:

AttributeError

'ManagementPanelImporterMultipleWatcher' object has no attribute '_watchers'

File "include\ClientGUICommon.py", line 549, in EventButton

self._func( *self._args, **self._kwargs )

File "include\ClientGUIManagement.py", line 2207, in _SetOptionsToWatchers

for watcher in self._watchers:

and

AttributeError

'ManagementPanelImporterMultipleWatcher' object has no attribute '_watchers'

File "include\ClientGUICommon.py", line 549, in EventButton

self._func( *self._args, **self._kwargs )

File "include\ClientGUIManagement.py", line 2207, in _SetOptionsToWatchers

for watcher in self._watchers:

it comes in when I set watcher options, it seems to be all the same for the 2 /b/ threads and the highlighted 60 threads on trash. I then tried to set a single one on trash and confirmed it was what was tossing errors.

just as a note, /b/ is set to a static 3 minutes, trash is set to a dynamic 3 min to 1 hour based on the oldest comment. the rest can keep the 1 day minimum check time

as for the ui change with minimums for thread watchers, honestly I think there should be an option to set the minimums and allow it to expand based on how much room I give it, also in regards to buttons, the timings and everything below it could be 3 buttons side by side, and especially with how wide I made the section, the thread watcher buttons could all be merged into a single line too, and without issue.

at 4k with no scroll bar showing, I need to expand my window so large it only allows 1110x606 for previews, for a normal images that is taller then wide, this will typically mean 403x604

previously that area would have been 1110x1460 and with a taller then wide ratio image, allowed for 976x1460

personal preference here, on import I don't give to much of a shit about tags, I personally assume everything I get is going to be untagged and will need me to tag it later on anyway so no real need to see them in a download window, I would say the ability to remove the tag section from area and a user defined minimum size for the thread area would get me pretty much where I want to be, at least as far as window size goes.

some creative rearrangements of ui would also free up around 120pixels of vertical space, its not allot but it adds up. this is something that has been on my mind since quick and dirty came around for duplicates, not something I gave a whole lot of thought due to my inability use use duplicate processing currently, but with the multi thread watcher having unnecessarily large buttons, and other buttons that could largely be put in one line, its on my mind again.

>>9231

>>9232

hey if hydev can fully dedicate to this thanks to generous people like you, so be it. i don't have the kind of spare money to give away, but im glad people like you do, it benefits all of us.

>>9227

>>9210

>>9229

Not this bad. I am not very wealthy and not very poor. My sources of income are typical to many other Anons :^). I spend about 25-30 hours on hydrus a week, all in, and I'd like to be able to do more. My money situation is ok enough that I can keep doing hydrus for the forseeable future. I am very appreciative of the support through Patreon–it absolutely makes a difference.

That said, Maruchan, pop tarts, and veg+pasta in a pressure cooker is the way to do programming life on the cheap. And having no gf/friends.

>>9211

Thanks. I think I'll be doing something like this, yeah–splitting up the page load so it happens slower but more in the background. Actually getting file hashes out of the database happens at like 3,000 files a second, but getting all the metadata afterwards in the actual page load is what kills a new db's responsivity.

>>9214

Fuck, sorry about this bug! This was something I did just after E3, and I can see in my mind now how I forgot to test this properly. It'll have to wait another week until it is ready to go. Thank you for the report.

For the listctrl column widths, my current plan is to convert the rest of the old ones over and then implement some kind of global history system for them, so the program will remember sort columns and column width (and maybe even some sort of column-selection) for every listctrl. I've got an identity system in place, but I want to be caught up with this new object before I start tacking on anything too clever.

And for the options buttons, the instances of those three, or just file/tag import options being stacked together so frequently makes me think I should maybe tuck it all into the same button and launch a dialog with notebook tabs, or something. This would save space and code across the program.

>>9215

Thanks m8, you too. Let me know if anything doesn't work.

>>9218

Can you give an example URL that gives this? Do any URLs work ok for you? Everything works fine over here. If you go to network->manage url class links, is the inkbunny page url class linked up with the correct parser and everything?

>>9220

Thank you, this is very generous. I'll be doing some checkbox-options work on tag import options this week, so I will try to fit it in. Let me know how it works for you when I roll it out.

The money is great, but please don't feel obligated to offer if there is something you would like.

>>9232

>>9235

>>9236

Yeah, all I can offer to this is that remember businesses have a lot of, ha ha, capital. Salary costs are extensive, but there are many other places money needs to go to keep the lights on. Line employees need to generate a lot more wealth than their salary in order to pay for benefits and overhead and building maintenance and new investments and bizdev and struggling departments and bond payments and totally-legit-not-to-be-questioned-executive-compensation and dividends and everything else. Contractors can often charge astronomical rates because they offer a business the same productivity in a more flexible way without the many of the same (particularly HR-related) burdens. They usually have more experience and willingness to work under worse conditions. If you are a 40-year expert in aerospace or some specific kind of legacy software or database, you can sell your talents to fix a broken project for megabucks. Your company doesn't pay you what you are 'worth', whatever that really means, but the minimum of the range you are willing to take.

Here's some ideas on how much it costs just to hire a top-line tech guy:

https://www.quora.com/What-is-the-average-cost-of-recruiting-an-engineer-in-Silicon-Valley

$30k seems a decent number. And firing someone, particularly if they tick any diversity checkboxes, can be much more expensive. So bringing on a contractor for four weeks can be cheaper, if kept strictly to the short term (pro tip: this never happens). Not every hire is a balls-to-the-wall professional of course, but when you think of the dullards you have met who pull $55k from a cushy job doing very little (or those 'coordinators' in government pulling $230k!), and consider how much capital really has to go into maintaining them in that position, it is a miracle that anything adds up to black at all.

>>9239

If you open up network->manage parsers and double-click any of them–we don't want to edit anything here, I just want access to the test/debug ui–and then click the 'link' icon on the right and put in this:

https://www.pixiv.net/member_illust.php?id=10886733&type=illust&p=1

It should then fetch the contents of that page–as your client sees–and dump it in the box below. You can click the 'copy' icon button to put it in your clipboard and then paste it somewhere better. Does it have anything like 'Error: login problem' or anything? If you can't see anything obvious, can you either email it to me or put it in pastebin and post that link?

>>9240

What would you like for the page numbers? zero-indexed? Try drag-and-dropping this png onto network->manage parsers and then go network->manage url class links and linking 'pixiv manga_big page' class with this new one. Let me know if it doesn't work.

>>9248

I'm >>9239.

Followed your instruction and I got these garbled characters.

I get sane strings when I copy them by pressing "Copy the current example data to the clipboard" button and paste it on pastebin.(at least on my environment.)

pastebin here.

https://pastebin.com/iKRM54x6

From the facts that another anon can use pixiv downloader and I'm using Japanese system locale so the cause might be my system locale setting.

>>9248

If you are going to put them in a dialouge window, it would have to be one where you can use the background parts of the program too, thinking about it a bit more, here is a program you may want to look into

acdsee 8 pro, this version of it specifically

note the area on the left above the thumbnail, you could have in the case of threadwatchers

the thread window with its basic functions,

a settings tab

and a tag tab

pretty much all giving you the same amount of information, just at different times.

that said thinking of it more,

clear highlight and highlight on all tabs would be very good, especially because of tabs like this one, 96000 with 14000 already in db, I would be able to unhighlight it, set the options to only show new ones, and then rehilight it, it seems quite a few issues I have would be solved with this being a part of every downloader.

On the note of download tabs, do you have any interest in expanding the multi watcher to a general multi downloader and multi gallery downloader?

The tab that has 96000 images is one from gelburro I believe, just looking at what was downloaded I think I put in 20 or so different artists in general, and then also added some tags like short stack or some character names, needless to say the things a gigantic clusterfuck of images.

however a multi gallery I could have each search handled on it own viewable selection,

and a multidownloader, well here, the 15000 image page is from a /aco/ download of old 404 threads.

Till you got the multithreadwatcher in place, what I did was I saved tabs, and I waited for them to either die or 404 then download them en mass, I said the first day multithreadwatcher came in that I had something along the lines of 800-900 threads set to watch, there was no way I could honestly use the thread watcher in its former state for my uses, but with thread watcher, I can. even now, every now and then I need to pull from the archives, and then I end up falling down a rabbit hole that ends up with me having a 26k image tab, and considering 4chan has at most a 251 image limit on threads, an issue should be visible.

>>9255

That's some interesting and weird stuff! I wonder if Jap lang pixiv has utf-16 or something and it is being decoded weird in that panel? I'm not sure why hydrus would be getting jp-lang back anyway (it doesn't send any system lang info in the http headers, afaik), unless that is tied to some cookie related to a previous successful login. I don't think your lang will affect the hydrus-side parsing. Anyway–

I tried rendering that page in my browser and got pic related, which looks like a moonrune login page, so my guess is that your hydrus got logged out somehow. Can you please check out what you have for pixiv.net under network->review session cookies? I have 9, expiring mostly 2-5 years from now, except for PHPSESSID, which expires in 29 days. Don't post your values here–they are private–but do you have a PHPSESSID, and what is its expiry?

Select and delete all the cookies for pixiv.net and try doing another pixiv query. This will cause hydrus to try to login again and hopefully fix your problem. Let me know if that does the trick.

>>9249

>>9250

>>9251

Hmm, this is odd. The lack of -p# shouldn't matter to the url class. Please hit network->manage url classes and paste an example inkbunny url you want to get in the textbox at the top–it should say "Matches inkbunny file page", but if it somehow matches some other class, that might suggest why a parser is being found but not then finding anything.

I'll also ask you to do what I suggested to the pixiv guy earlier:

If you open up network->manage parsers and double-click 'inkbunny file page parser', and then click the 'link' icon on the right and put in this:

(enter your inkbunny url that is not working right here)

It should then fetch the contents of that page–as your client sees–and dump it in the box below. You can click the 'copy' icon button to put it in your clipboard and then paste it somewhere better. Can you either email it to me or put it in pastebin and post that link?

You might want to click the 'test parse' button as well here, to see if this different area can parse anything or maybe give a better error message.

>>9257

I still have to pin down how to do the new gallery ui. Given how well the multi-watcher has worked, that might be exactly the way to go.

A big change in this new downloader will be that pages are not tied to parsers. There will be a dropdown on new download pages that has (hentai foundry, gelbooru, pixiv) and all the other 'searcher' objects listed, so you'll be able to fairly quickly spawn the same artist search on multiple sites on the same page. Since I also want to add some kind of 'query history/logging/inspection', I will need some sort of list to show everything that got entered and how it did. A listctrl like the multi-watcher's would be ideal for this. This will also allow parallel gallery parsing and file import queue processing on the same page.

I'm thinking about having a simpler page, a 'single downloader', for new users, but maybe not. I am trying not to get too much into adding thing after thing, when in this first iteration I really just want shit to work.

>>9260

IB guy here, pics related.

>Please hit network->manage url classes and paste an example inkbunny url you want to get in the textbox at the top–it should say "Matches inkbunny file page"

done, that part works perfectly

>If you open up network->manage parsers and double-click 'inkbunny file page parser', and then click the 'link' icon on the right……

also done, and pic 2 is the result.

i'll proceed to copy the full text below here:

—————————————

PREVIEW:

<!doctype html>

<html style='background: #333'><meta charset="utf-8">

<meta name="viewport" content="width=980">

<base href="https://inkbunny.net" />

<link rel="canonical" href="https://inkbunny.net/s/1639758-p1-Pokegirl-picture-pinup-toriel-" />

<link rel="preconnect" href="https://br.ib.metapix.net">

<title>Members Only &lt; Submission | Inkbunny, the Furry Art Community</title>

<meta name="og:title" content="Members Only &lt; Submission | Inkbunny, the Furry Art Community">

<link href="https://br.ib.metapix.net/images78/favicon.ico" rel="shortcut icon" type="image/x-icon">

<link href="https://inkbunny.net/css78/global.rev30.min.css" rel="stylesheet"><script src="https://inkbunny.net/js78/prototype.min.js"></script>

<script src="https://inkbunny.net/js78/scriptaculous/effects.min.js"></script>

<script src="https://inkbunny.net/js78/global.rev11.min.js"></script>

<script>

var skoo = 0;

var daytime_start = 0;

var dusk_start = 0;

var night_start = 0;

var dawn_start = 0;

var nolocation = tru

—————————————

i think the problem is in the ratings. I just realized that this issue doesn't just happen to me in single-files, it happens on any submission, single or multi page, whose rating isn't allowed to guests by default.

The problem seems to happen only when the rating is anything above from general/safe as IB has every other rating off by default when you first access the site and aren't logged in. The only way to view questionable/violent/NSFW/Adult situations as a site Guest is to click on pic 4 and manually check the boxes, which probably modifies the sites Cookie in the browser.

The last time i tried to download a multi page submission, i just realized, it worked because it was rated General and Not restricted to guests. My mistake was thinking that the problem was on single files oblivious to the fact that IB also has barriers such as RATING and NO GUESTS ALLOWED.

Speaking of, before this problem arises (because it inevitably will if you so decide to keep working on IB which im really grateful for btw :^)), IB also lets users restrict their content to guests, meaning only people who log in can see them. See pic 5, and here's an example URL of such a case which is both NSFW and doesn't allow guests

https://inkbunny.net/s/1639758 —– (single picture, Adult rating, restricted to guests, same picture i used in the example above)

https://inkbunny.net/s/1640941 —– (multi picture, Adult rating, not restricted to guests)

Neither of these will work in the URL downloader. and i'm guessing its because Hydrus connects to the site as an unregistered guest with default content settings. Hope this helps ya hydev.

>>9260

I'm stupid.

I forgot to have changed my pixiv password.

I logined with new password and everything works.

I'm so sorry.

Saying my excuse,I entered old password and test by pressing "test" button and it says OK.

I checked if made up email address and password pass this test and I got OK sign.

But it's all my fault for make you busy,sorry.

>>9261

Ah, thank you. I have saved this info. Yeah, until I have the login manager done, hydrus is almost always a 'guest' for all sites it visits. I would like this situation fixed neatly by the time I am done with this overhaul, but if you are feeling brave, you can try copying your inkbunny.net non-session cookies from your browser to the (ugly and debug-tier atm) ui under network->review session cookies. You can do a 'manual' login this way for now, basically telling your client to pretend to be your browser. It works in the short term, but needs refreshing whenever the critical 'login' cookie expires.

>>9262

>>9263

Yeah, I'll be moving towards 'url class' classification, so you'll be able to drop those gallery urls on the client like you can do the post urls, and it'll be able to spawn/reuse a downloader and link up the parser and all that without you having to do anything.

Better and more flexible ui layout is a problem. wx is not so great at some of this stuff, but I am basically stuck with it–it would take way too long to move to a new library. Still, I am happier with what I have now than I did a year ago, and I can keep improving. I wouldn't be surprised if new versions of wx are better as well.

For that 'failed' count issue, I'd like to do something like '122, 3 failed' in the status column, when failed/ignored exist. I think the subscription panel does something like that as well. This will highlight which threads have problems, and when I have more inspection ui available, it'll let you more easily find the ones you need to go into and 'try again' or whatever.

>>9266

Don't worry about it–thank you for letting me know, and I am glad you are now working. I hope to improve the general ui feedback here when I work on the login manager and present better error states. The Pixiv situation is particularly difficult, iirc, because the cookie I test for login success is basically the same one you get for browsing as a guest. I wouldn't be surprised if that dialog gives an OK for 'valid username, invalid pass' just because the test is subtle. I will check this out and improve the test, thank you for the report.

>>9208

Regarding the new pixiv manga downloader, I noticed that the file count doesn't increment properly. The total number never increases. Instead, the current file number decreases so it can fit the manga file count in it. ex. Say we have a sub where we've already got 475 files out of 480, and the remaining 5 are pixiv mangas. If the mangas have more than 2(?) files to them, the counter will decrement out of 475 so it can fit the total number of files and end up back at 475/480. 480 will never increase. Also, it doesn't catch files on import if you've already got them in db, i.e. it will use up bandwith redownloading the whole file before deciding you already have it in db. Might not be anything you can do about that tho.

Also, had a related issue. After running all my pixiv subs, I noticed I still had a mismatched file count (ex. 3449/3467), so I looked at my subs and saw one marked dead that had those remaining 18 urls, which was pic related 1. They had no status, so I set them to "try again" but it would never actually go through, and unpausing it just gave the notice that it was dead. So I ran those through the url downloader and it turns out I already had them all, and they were all tagged, but I somehow got "36 files imported, 32 already in db" and the page only shows 32 files. I think the only way that can work is if some of them were the exact same file with a different url, but oddly, none of the urls I entered got applied to the files.

Anyway, my actual question there was how to get those ones to properly "try again." Not sure if the other weirdness is a bug or me being retarded.

Did siblings get broken at some point? The manage tags list sometimes has double entries like that, but the one on the left never does.