/hydrus/ - Hydrus Network

any benefit to going to 3?

or is it just because 3 is newer?

On a side note, one of my sub I have check every 6 hours keeps having the 6 hours changed to 6 minutes. ill probably have a bit to say later on when I get more time with it, but this is an immediate thing I notice. it has happened before, but I assumed it was my own error that time.

>>10273

Be a chad and make this as your code of conduct

https://sqlite.org/codeofconduct.html

I decided to finally try running from source with this version, however I seem to be missing something and I can't figure it out. I'm on Ubuntu 16.04 and carefully followed the instructions for running from source.

Traceback (most recent call last):

File "client.pyw", line 13, in <module>

from include import HydrusData

File "[redacted]/hydrus/include/HydrusData.py", line 223

def ConvertResolutionToPrettyString( ( width, height ) ):

^

SyntaxError: invalid syntax

This is not correctly shown due to the proportional font, but it indicates the second bracket. Seems weird that it complains about a tuple, so I'm clueless about what the problem could be. Any help would be appreciated.

>>10278

What century are the sqlite devs from? This one seems more reasonable as a modern code of conduct: http://www.un.org/en/universal-declaration-human-rights/index.html

>>10278

>Do not return ~evil~justice for evil.

>Do no wrong to anyone, and bear patiently wrongs done to yourself.

>Love your enemies.

>Do not curse those who curse you, but rather bless them.

Hahaha nope

I just had Hydrus run maintenance (looking for duplicates) while a gallery downloader was busy downloading stuff. Should this be happening? I thought maintenance was supposed to run while the client was idle.

>>10282

>>10278

>>10283

>>10284

Thank you for your interest here. This is obviously in the news right now. I am no fan of codes of conduct in general, as I usually end up on the wrong side of them. Rest assured that I am not suddenly about to take up the Contributor Covenant.

I am most comfortable with imageboard culture, which I hope comes through in hydrus's design in various ways. I suspect the core of our culture is fundamentally incompatible with written social rules, and obviously with any attempt to reduce colon crucifixion.

As it is, the whole thing is moot as >>10278 says, as I am the only hydrus 'team member', and I don't see that changing as I am a complete sperg who can't work with others.

>>10274

I think it is 'about time', overall. I have been thinking about it for a couple of years now. I believe python themselves have finally stopped working on 2, and the big libraries are starting to drop support.

It has a multiple nice features (particularly unicode handling, including at the OS level, and improved C++ interactions, so simple things like thread event waits are more CPU efficient, and I'd like to play with the async stuff), and I understand they have now ironed out the cons, which was mostly slower execution. I expect to jump right up to 3.7, but I'll obviously read up more and play around a bit first.

>>10292

Sorry, as _ >>10284 _ says.

>>10276

>>10274

Thanks for this. I will check the 6 hour-6 min issue–can you take a screen of your checker options panel, just so I can see what death file velocity and all that you have set up?

I am hesitant to have the sub checker search multiple 'already seen' pages deep every check because A) It will make the existing 'one page' memory of the checker more complicated and B) It will waste bandwidth and CPU time for all involved for 99% of cases. From my perspective, there is a non-zero 'response is shit' rate due to server problems and Cloudflare changing hashes and resizes and stupid gif conversions and watermarks and bad uploads and all sorts, so I am not super interested in spending 70% more effort to finish the last 0.2% of results, particularly when most of us are downloading far faster than we process anyway.

In changing the check system this week, I realised that doing five consecutive 0s would be a better check than five 0s total (and stop there), and then I figured since I had fetched a page's worth anyway, there was no point stopping as soon as I hit the first five–the most effective place to check for the five consecutive 0s would be the final files in the page. I am overall happy with this change as it improves the number of True Positive 'need to fetch more pages' while only increasing False Positives in the rare case that the page border is the exact 'true' sync point, all without increasing bandwidth or CPU significantly.

My current plan is to verify this new logic is not hitting false positive too much and then hide the periodic check notifications, and then see how it works overall. I think I would be willing to add an option to change the x=5, maybe for advanced users, in which case you could boost your x up to 15 or whatever and get a deeper, more 'guaranteed' search that way. But I'd probably have to think about safeguards to stop runaway problems. I don't want, through my fault or user fault, to accidentally hit some site for 5,000 CPU-heavy pages every six minutes.

>>10281

I believe tuple parameter unpacking–the cheeky (width, height) param–is stopped in python 3, so my guess is you are running the code as 3? Hydrus is 2.7 for now (but not for long!), so if your machine is defaulting to 3, see if you can force it with python2 client.py or something? I thought my scripts' hashbang was supposed to force it, but I don't really know anything about that stuff.

If you just type just whatever python command you are using into a terminal, it'll give you the version it things it is just above its >>>.

>>10285

Thank you for this report. Unfortunately, ugoira is not yet supported, although it is a long-term project (likely conversion to video/apng). You'll see the 'notes' column there says 'ugoira veto', which is the new downloader noticing and skipping those URLs.

If you did not know, ugoira is an old Nip attempt to make Web 1.0 webm-like animation by tying blessed jpeg print-outs to a shrine tree dedicated to the Trickster God of Javascript. It may seemingly render correctly in your browser, but right-click->save as is disabled due to a curse placed on the West after the Nagasaki bombing.

>>10287

It depends–what are your idle settings under options->maintenance and processing? If the 'go idle if mouse stays still' setting is only set to like 2mins, or disabled completely, I could see it kicking in a bit too early.

That said, non-user actions like download queues don't count as foreground work, so if you weren't using the computer otherwise to browse or move your mouse, it is normal for maintenance to go. If you'd prefer it work a bit less, have a play with the settings on that options page. Only doing work on shutdown works for a lot of users, even if it is only one big job a week.

>>10297

Ugoira is actually the brain child of Hector Martin, better known as Marcan, one of the guys who made the Wii homebrew channel and now works for Pixiv.

That trivia aside, I would like to vote against any kind of format conversion happening in Hydrus, or to make it optional. Format conversion to webm/mp4 would be lossy, while format conversion to apng would be like those bloated png screenshots of jpgs.

Therefore I think ugoira should by default be preserved in it's original format.

The ugoira format is ridiculously easy to parse.

The container is a zip file, which according to specification must always have compression disabled, so you wouldn't even need a compression library to read the images inside.

The images are a sequence of either jpg or png files, sequentially numbered.

Last it contains a json file called animation.json, which contains the timing information. Instead of a fixed frame rate, each frame has a delay in milliseconds, like gifs.

There are slides of an old presentation by Marcan here, explaining some of the philosophy: https://marcan.st/talks/2014_pixiv_ugoku_player/

Note that some of the information is outdated. E.g. timing was originally not stored inside the zip file, but embedded in the html code of the site.

>>10293

Huh. I thought you had upgraded python a bunch already. Was that 2.XX versions or were the wx updates something else entirely?

>>10301

>>10302

What kind of "bloat" are we talking with apngs?

>>10302

To me the main problems with conversion are quality loss and inconsistent hashes.

Ugoira uses a single contained file, so it's not uglier than any other container format I think.

Of course proper video formats are better to handle. Webm is far superior in quality and file size, compared to the ugoira zip files, but I think it's better to keep the original data than making even more conversions, which would introduce more quality loss and more hashes to keep track of.

With common webm files, quality wise there would be chroma loss, since they are only encoded with 4:2:0 chroma. Encoders usually use the Rec. 709 standard, which reduces the 8 bit integer range to 16-235. We would need to encode to 4:4:4 chroma with full integer range, which not all players support yet. Even ignoring the lossy video compression, we would already lose so much image quality.

For the jpgs in ugoira files, 4:4:4 chroma with full integer range is a universally supported standard.

Then there is the problem of hashes.

Ugoira uploaded to boorus were already converted to webm or gif. Now we would make our own conversions from the ugoira, which would have different hashes than the ones in boorus. Even different Hydrus users would have different hashes, because only one component like the encoder or muxer need to be updated to a newer version to result in a different hash. The same encoder might even produce files with different hashes on different PCs. This would make the tag repository useless.

My one wish is to keep these files native. I think an optional encoder is a good idea for exporting/sharing files, but making it standard just sounds like it wouldn't be worth the little storage space it would safe.

>>10303

WIth "bloat" I meant the bloated file sizes when converting jpgs to png. Even for still pictures it's already annoying to have png conversions of jpg files. It doesn't make sense for them to exist. My guess is that people take screenshots of jpg pictures or something. Some people do that on their phones, because the screenshot shortcut is easier to use than opening a context menu to save the original file.

>>10302

It still considers the zip the downloadable URL according to a parser test.

Maybe have an option to have ffmpeg convert the contents of the zip into a hydrus video player friendly format and cache this file in the db somewhere, so that when the hash for an ugoira is called to the image viewer, it instead grabs the cached video linked to it?

>>10297

Oh, its a file format that is pretty much specifically pixiv…

welp, now I know, thought it was something else that was a weird error.

>>10304

>making it standard just sounds like it wouldn't be worth the little storage space it would safe

Disagree, I definitely would want it. I don't care about elegant retro solutions or what is simpler from a conceptual perspective, I want something that works well with both clients and servers and will actually run with modern players and embeds if exported from hydrus, and I want to avoid stutter, timing inconsistencies between hardware, memspikes as much as possible. So for someone like me, I would absolutely want a utility to convert any given memeformat into something more practical. I don't care if I can open it as a zip and extract individual frames, because I can just take a screenshot if I want a frame.

So for me, I would have no use for the import/handling unless it also included an option to convert to something more standard.

In fact, while I'm not holding my breath, I would love to eventually be able to convert all of my files freely and losslessly between common filetypes within Hydrus, and ideally endgame into mostly just FLIF/FLAC/whatever non-Google thing kills WebM/the PDF killer/etc

>>10309

>memeformat

I see that the two of us use Hydrus to achieve very different goals. You're making a good point for people who use it as a meme archive. Which is a legit use too. Conversion would definitely be better for people like you, for ease of sharing, but catastrophic for people like me, who want to use it for long-term archiving.

So I guess it makes sense to have a conversion option for some people. However I still think the original unchanged files should always be the default, especially since memes usually already come in easily shareable formats from image boards rather than Pixiv.

I would like to use an example to illustrate the problems of format conversions:

In 2003, somebody ripped a tv series from a DVD. It used MPEG-2, a horribly inefficient lossy video codec, that was already outdated back then. So to not waste space, he converted the DVD's video to use the latest state of the art codec he could find: DivX

You should already be cringing at this point, because DivX is a shitty old codec now.

In 2018, you would definitely try to find a better version of that tv series, than that shitty old DivX rip. You would most likely find a batch of mkvs using h.264, that look much better.

These better looking mkv/h.264 files exist because somebody kept the DVD's original MPEG-2 video and re-converted it to a modern format.

They will still need to keep the MPEG-2 videos archived though, because h.264 is the shitty DivX of tomorrow, and they will be re-released in h.265, VP9, or one of their successors.

The ugoira files with their shitty jpg compression are our equivalent to the DVDs from that story. Sure, they are using outdated compression, but it's the best version we have. Any secondary conversion would degrade them, just like that DivX rip did to the MPEG-2 video. Ideally we would want artists to release in a lossless format, or a more modern format like webm, but they didn't, so the ugoira version is the one we need to keep, from an archivist's point of view.

Using conversion is great for sharing, but please let people archive the originals. Any conversion happening in Hydrus should be optional, and it should automatically add the necessary meta data to identify the original source, because people suck at indicating sources.

>>10310

>>10311

That's a good example that all scanlation groups should read. Ideally you would need to re-scan the original, since, as you mentioned, the resolution should be higher for 1 bit color. The half-tone pattern shows some aliasing, though that might be an artifact of the jpg compression. 1 bit color depth with oversampling during scanning is the way to go for manga. But, your conversion only turned out good because you understand the medium and made an informed decision to chose the format. It would be catastrophic if it was done automatically, just because some algorithm detected that something might be a manga.

>>10313

>You're making a good point for people who use it as a meme archive.

Don't trip their senpai, I'm saying that Ugoira is only used by a few websites and is not a standard universal format (like DVDs are). A long-term archival format should be something with wide compatibility since it's pointless to archive something which can only ever be played on its host device once it has become broken and ancient.

>Any secondary conversion would degrade them

Any LOSSY secondary conversion.

At very worst case, they could still be converted to FFVI and it would be a more standard option.

>Any conversion happening in Hydrus should be optional

Yeah, we agree.

>>10304

>>10312

No shit. I meant how much bloat? Do you really have some many ugoiras as to get an extra GB worth in bloat?

Still, you have your point about keeping the ugoiras. If there was a way to read them natively inside Hydrus hten that would be best, with an option to convert to webm or whatever for sharing. Still, that itself would increase hashes, and apng might be the better solution for quality-minded individuals.

Regardless, this all depends on what magic Hydrus Dev can do. While I'd love to have my ugoiras work in Hydrus, if it's going to be a time consuming endeavor then there are other things I'd rather be worked on.

>>10318

Regex.

>>10320

Wait, I fucked up.

>>10316

>it's pointless to archive something which can only ever be played on its host device once it has become broken and ancient.

But jpg and zip have extremely wide compatibility. Much wider compatibility than webm and apng (which despite increasing in popularity still is not recognized as a valid standard by the png group afaik.)

As long as we have jpg decoders, we can play those files.

Even if no ugoira player existed anymore, you could just unzip an ugoira to a new folder, open the first frame in you OS's standard image viewer and keep the right cursor key pressed to play the animation like a flip book.

This example is just to illustrate how ridiculously low-tech this format is.

It shouldn't take a programmer more than a few hours to make a simple player, even if he never heard of ugoira before and needs to "reverse engineer" it. And my last claim was an insult to all the people who do real reverse engineering, because elementary school kids could figure this out.

>Any LOSSY secondary conversion.

Well, any lossless secondary conversion of jpgs would be a waste of storage space. It could be a good idea for png based ugoira, but then we still would generate new hashes with conversions.

>>10318

It's hard to tell how much bloat it would be for the ugoira I hoarded so far, since they are spread over multiple folders and mixed with regular pixiv image posts.

One of my old ugoira-only folders is 5 GB big and only contains jpg based ugoira.

If we check the jpg to png conversion of that Amy pic:

>>10284

>>10312

We see a 6.79x size increase, which means that folder might possibly become over 30 GB big, if converted to a lossless format.

>>10318

well this one, relatively small amy image has about 6.79 times the file size, lets say an animation was 10mb, that would bloat to 67.9mb

Its not about 1 file, its about potentially hundreds if not thousands.

>>10313

Honestly, every single manga can be put through a 16/10 color pass and come out the other end looking nearly the same if not the same, the reason is all of the smaller points that make up a gradient are more than capable of doing so when constricted in this way, and because manga is a bionary white and black, there is no smooth shading done digitally where a 256+ color gradient would come into play so the main reason to have full color range is gone, and because every area is broken up in light and dark, it's nearly impossible to sight read the differences, the long and short being you could robo every manga page to 16 color, 10 color, or even the majority to 8 color and you would be able to save fuckloads of space.

hell, If I put a bit of time into editing this and bringing the dark colors up and pushing the light back, I could probably make it work with the 4 color or maybe even 3 color but that would be pushing it due to the images being a jpeg at one point in time.

more or less, there are ways to automate image compression like this that are lossless, and if converting a jpeg to another lossy format, while it may lose something, if I believe I gain more then I lose i'm more then willing to take the hit.

I would fully support a compression part of hydrus just because it could save so much space it wouldn't even be funny, and with png, so many of them have wasted space that even lossless re encodes can end up shitting out good chunks of space.

now, for your dvd to mpeg2 to divx comparison analogy, sure that's bad, but I look at it more like a tiff to png, where both are lossless but one is far better, or a cd to flac, again both are lossless one is just losslessly compressed. I have no real desire to share my images most of the time, so just a huge archive is good enough for me, and if a conversion from jpeg to flif or whatever looks near identical but one is smaller by a good ratio, my image archive can take the hit.

I mean jpeg has been going on 26 years now, a new file format would let likely go on for another 20 odd years and by then there is a good chance i'll be dead so it's not like video formats where the compression vs quality ratio changed so drastically in my computer using life time that you could have completely obliterated a video by format changes.

Im not even say hydrus should do this as default, just I would probably use said feature if it was implemented especially for some images i know could easily be compressed.

>>10316

if ugoira is what I think it is, it's more like having the film reel the movie was shot on opposed to having a dvd of the final cut.

Just had an idea because of someone's fucking tumblr that decided to have 30k images on it, is it possible to import galleries file paused?

4chan or 8chan threads I want going almost immediately, however galleries dont have a shelf life, or at least normally have a shelf life, this way when fucking disasters like that tumblr happen, I don't have 3 or 4 galleries going that hang the program and make importing more a miserable experience.

Personally, files is where I want the program to pause, but finding the files I would like unpaused as it doesn't seem to really affect performance much at all.

Hey dev, I'm running into this issue with this version:

```WARNING:root:pafy: youtube-dl not found; falling back to internal backend. This is not as well maintained as the youtube-dl backend. To hide this message, set the environmental variable PAFY_BACKEND to "internal".

libdc1394 error: Failed to initialize libdc1394

2018/10/23 22:02:10: hydrus client started

2018/10/23 22:02:10: booting controller…

2018/10/23 22:02:10: booting db…

2018/10/23 22:02:10: preparing disk cache

2018/10/23 22:02:10: preparing db caches

2018/10/23 22:02:10: booting db…

2018/10/23 22:02:10: initialising managers

2018/10/23 22:02:10: If the db crashed, another error may be written just above ^.

2018/10/23 22:02:10: A serious error occurred while trying to start the program. The error will be shown next in a window. More information may have been written to client.log.

2018/10/23 22:02:10: Traceback (most recent call last):

File "/opt/hydrus/include/ClientController.py", line 1256, in THREADBootEverything

self.InitModel()

File "/opt/hydrus/include/ClientController.py", line 639, in InitModel

session_manager = self.Read( 'serialisable', HydrusSerialisable.SERIALISABLE_TYPE_NETWORK_SESSION_MANAGER )

File "/opt/hydrus/include/HydrusController.py", line 540, in Read

return self._Read( action, *args, **kwargs )

File "/opt/hydrus/include/HydrusController.py", line 180, in _Read

result = self.db.Read( action, HC.HIGH_PRIORITY, *args, **kwargs )

File "/opt/hydrus/include/HydrusDB.py", line 873, in Read

return job.GetResult()

File "/opt/hydrus/include/HydrusData.py", line 1498, in GetResult

raise e

DBException: ImportError: No module named ordered_dict

Database Traceback (most recent call last):

File "/opt/hydrus/include/HydrusDB.py", line 532, in _ProcessJob

result = self._Read( action, *args, **kwargs )

File "/opt/hydrus/include/ClientDB.py", line 8723, in _Read

elif action == 'serialisable': result = self._GetJSONDump( *args, **kwargs )

File "/opt/hydrus/include/ClientDB.py", line 5545, in _GetJSONDump

return HydrusSerialisable.CreateFromSerialisableTuple( ( dump_type, version, serialisable_info ) )

File "/opt/hydrus/include/HydrusSerialisable.py", line 136, in CreateFromSerialisableTuple

obj.InitialiseFromSerialisableInfo( version, serialisable_info )

File "/opt/hydrus/include/HydrusSerialisable.py", line 213, in InitialiseFromSerialisableInfo

self._InitialiseFromSerialisableInfo( serialisable_info )

File "/opt/hydrus/include/ClientNetworkingSessions.py", line 76, in _InitialiseFromSerialisableInfo

session = cPickle.loads( str( pickled_session ) )

ImportError: No module named ordered_dict

```

Something to note: my computer crashed shortly before this, so I'm unsure if it's related to that. However, it does not look like that, it looks like whatever pickled session object was created relied on ordered_dict which seems to have become unavailable for some reason in my Python 2 distribution.

>>10325

>Honestly, every single manga can be put through a 16/10 color pass and come out the other end looking nearly the same if not the same

I agree that for scanned manga it would most likely always work. Though I have some manga from a guy painting digitally, who shaded some areas in gray scale instead of using rasters. While the paper print would have had a half-tone pattern, the digital source I have has 8 bit gray scale in some places. So automatic conversion could result in loss of detail there. But yeah, I definitely support built-in conversion for Hydrus, as long as it's optional. Being able to define custom command lines that can then be called from some menu would be a nice idea. Not just for converting, but extending functionality in all kinds of ways. But I'm getting off topic.

>I look at it more like a tiff to png, where both are lossless but one is far better

That comparison is correct for png based ugoira, but most of them are jpg based, so already are lossy. Recompressing them will always result in more quality loss, or bigger file sizes. Even re-compressing to jpg would probably result in more loss, due to different quantization tables or rounding errors.

>I mean jpeg has been going on 26 years now, a new file format would let likely go on for another 20 odd years

Hm, it's really hard to say what's going to happen to web image formats.

For lossless formats, I guess flif might become a good replacement for png at some point.

For lossy formats it seems more complicated. Even though jpg is old, the world somehow decided that it prefers jpg's blocking artifacts over jpeg-2000's ringing artifacts, even though it's a more advanced format for many use cases. And I don't see people converting their image collections to google's shitty webp any time soon. It's entire specification is a collection of bad ideas.

>it's more like having the film reel the movie was shot on

That wasn't addressed to me, but that's the idea. Using old film as analogy, the archivist's goal is to preserve the version that's as close to the camera negative as possible.

>>10333

I managed to fix this by adding some exception-handling code that simply skips initializing the failed pickled sessions. Not sure if I broke something, but it seems to have fixed the issue at least for now.

>>10335

digital artists, who actually use digital techniques instead of trying to apply traditional to digital are fairly rare, as far as automatic goes, i'm against that just because it's so easy to make an oops, though some form of a png crush or something could likely be implemented as an every png as I believe the one i'm thinking of just gets rid of worthless data.

the tiff to png was more referring to a full conversion from png to another format if the format was handled better, lower file size while still having a lossless compressed image,

as for jpeg2000, I have to defer to this

http://www.stat.columbia.edu/~jakulin/jpeg/artifacts.htm

just looking at it compared to jpeg, its not a all round better format, while flif honestly seems to be a good contender for an all around better formant. I have issues with some of the formats that have little to no support, or demand to many resources when rendering, but it seems like flif can just out right halve lossless file sizes and even lossy, because of the way it loads, even a slow connection/slow server speed still gives you a reasonable file size. in my opinion, if I could convert my files to it, then on sharing them possibly convert them back, to a png (in lossless case) or just a fuck it give them a jpg, I could save somewhere around 500gb-1tb off my current image collection.

The main reason I don't do a full conversion of at least every lossless image to a better lossless format is simply compatibility, however since I moved 100% to hydrus for loose images, (and if they make a manga version, I will use that exclusively for magna too) Im more than willing to fully convert to formats like that.

>>10333

Same here, im on Arch and ever since i updated python, hydrus will not work anymore.

Oh fuck my life, just did a dup scan, got 53 thousand new hits.

>>10342

Or maybe wait until the dev addresses it, but he seems quite busy.

>>10337

>>10333

>>10340

>>10342

>>10343

Thanks. Someone emailed me with this earlier in the week. I did the same skip-fix as you here.

It looks like the new version of requests handles its sessions data with a different object. I never liked using cPickle here–I think I saw it working somewhere else. I will keep an eye on this, and likely will have to swallow resetting some sessions when I end up getting the new requests, or look for a better way of serialising this data.

>>10303

wxPython is the ui library I use. It is a python 'version' of wxWidgets. It draws all the windows and buttons and so on. When I updated wx, I went from v 3.x to 4.x, which was an important shift from one way of talking to the C++ layer to another. The API changed a bit, and bad code was punished more, so it needed a bunch of work.

This will be the actual language of python, which is what all of hydrus is written in. Python is doing a slow-motion titanic shift from some old ideas in 2 to some new ones in 3. I am on 2.7 now and will move to 3.6 or 3.7, depending on which works best for what we need. I don't know exactly how much work it will take, but it'll mostly be boring syntax changes and rebuilding libraries and the PyInstaller frozen exe stuff. It is possible to have a program work in 2 and 3 at the same time, but I don't feel I have the maintenance time or technical competence to try for that.

For ugoira, as with a lot of these file format issues, and much like mp3 vs flac, different people prioritise different things. I think both sides of 'just make it webm m8' and 'don't degrade any data' are reasonable, so I don't want to wade into this and force it either way. It'll be on the big things to work on next poll, and I expect to add auto-convert options for those who want it. I can't talk too intelligently as I still need to play with making variable frame rate videos with ffmpeg, whether you can deliver it a list of millisecond durations per frame or you have to figgledy-digglety-doo it a raw animation with dozens of duplicate frames and let it do vfr on its own.

>>10310

>>10311

This is interesting. My general personal feeling is that the value of a work is in how it makes your brain feel. A 0.2% difference in pixel colour vibrancy is unnoticeable in 99% of cases, so while it is desireable, the return on investment is a sinkhole. I don't really care about exact byte-for-byte preservation of content except when it helps us on the technical side of things like lining hashes up. And with Cloudflare et al increasingly fucking us anyway, trying to preserve 'True' originals may be a losing battle.

I feel that the coming 20 years will bring us many sorts of waifu2x that will be able to do denoising and other kinds of reverse-entropy better than the best manga cleaner alive today. Or even tools that let you say 'take this image, and put a artist-x-style penis on it', which if it works as a tech, opens up a whole can of worms of 'personal media', especially if it can be generated in anything close to real time. In any case, we'll want to update all our old 24bit 640x480 85% jpegs up to 48bit SuperK™ hyperimages, and all the bad decisions people are making now on jpg/png will be erased to a Good Enough standard by cleverer machines.

I think this will become more significant as we push into 4k and start getting areas of media that are higher quality than our dusty screens and smudgy glasses and imperfect eyes can see.

That feels like some rambly bullshit, but tl;dr: I think we'll fix all this shit later with better tools.

Watch this with the sound off–I thought the tech was great: https://www.youtube.com/watch?v=Mtz3KHWn1W8

>>10326

I may be misunderstanding your problem here–do you mean to start a new gallery query paused? You can't do this right now, but I've thought of it, maybe with a cog button on the downloader panel.

If you just want to pause your galleries from downloading files for a bit (but still let them do the gallery search, to build up their queue), select them in the downloader list right after you add them and click pause/play files. You can select and affect multiple queues at once by shift- and ctrl-clicking.

>>10334

Thanks. Yeah, I used to wangle a low-resolution URL for the NSFW DA images by picking some tumblr share link apart, but that got shut down and now you have to be logged in (or age-gate click-through, I think?) to load any NSFW image from DA. I think I maybe skip those urls automatically in the parser? I expect to roll out some login scripts for DA and the other sites like this in the near future to let you access this content (and fix the parser, if need be). Please give them a go and let me know how it works for you.

>>10341

Don't worry about it–that number is inflated because I am currently tracking groups with pairs. You get the number of combinations (as in combinations and permutations), which will reduce in size similarly faster than the number you process. There's a bunch of n(n-1) in there, essentially, when the true number is n.

At some point I expect to move the system to true group tracking, at which point many of the inflated numbers and unoptimised actions here will work a hell of a lot better.

>>10296

You were right about it defaulting to python 3. What actually happened was, I checked the python version and it displayed the 2.7 version. So my system defaults to 2.7. However as soon as I start venv, it defaults to 3.5.

I now installed the packages again using pip2, and tried running it with python2.

Now it shows the splash screen and starts initializing stuff, but then fails with this;

"ImportError: No module named ordered_dict"

I randomly tried installing packages that sounded like they might be related to ordered dicts using pip2:

ordereddict

persistent-ordered-dict

odict

But none of them solved the error. Do you have any idea what I'm missing?

>>10377

The new version of requests can't unpickle a session object from the old version of requests. Try v327, it should be fixed (by clearing your old sessions, argh).