Sadpanda Save/Export All Favorites

Load all favorites to the page and save it, export as JSON data or just copy the textbox contents. Works on exhentai and e-hentai.

< Feedback on Sadpanda Save/Export All Favorites

Review: Good - script works

§
Posted: 2022.04.20.

Very useful script! I made a fork to export data as json with tags if anyone is interested.

SaddestPandaAuthor
§
Posted: 2022.04.24.
Edited: 2022.04.24.

Sure. Can you link it?
I can link your fork in the description so people can see it.

§
Posted: 2022.11.03.
Edited: 2022.11.03.

Sorry mate I totally forgot I posted this :o
I'm currently working on an updated version for the new paging system but I'm dropping some features.
I'll link the script when I'm done (tomorrow or next week).

Honestly the new system is not too challenging to work with, you just have to get the next page url each time.
The issue of deleted galleries not being visible anymore make this export all the more valuable!

SaddestPandaAuthor
§
Posted: 2022.11.03.

No problem. I know that its easy to update so if you don't have the time let me do it.

§
Posted: 2022.11.05.

In the end I finished it tonight,

I'll put it here in case you want to pull some stuff to your main script, I didn't bother to update the metadata and such and removed a few features I didn't use.

The main differences are that it works only for the new exh update and that the export is in json format and exports some metadata from the galleries (favorite category, tags and notes). Because of this it only works in extended mode and produces a much bigger output so it may lag for big favorite lists.

// ==UserScript==
// @name        Sadpanda Save/Export All Favorites
// @namespace   SaddestPanda
// @description Load all favorites to the page and save it or just copy the textbox contents. Works on exhentai and e-hentai.
// @include     /^https?://e.hentai\.org/favorites.php.*/
// @homepage    https://sleazyfork.org/en/scripts/23406-sadpanda-save-export-all-favorites
// @supportURL  https://sleazyfork.org/en/scripts/23406-sadpanda-save-export-all-favorites/feedback
// @version     2.4.1
// @grant       GM.getValue
// @grant       GM.setValue
// @grant       GM.deleteValue
// @grant       GM_addValueChangeListener
// @grant       GM.addValueChangeListener
// @require     https://cdn.jsdelivr.net/npm/gm-webext-pref@0.4.0/dist/GM_webextPref.user.js
// @require     https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.0/jquery.min.js
// @require     https://cdnjs.cloudflare.com/ajax/libs/he/1.2.0/he.min.js
// ==/UserScript==
const PAGE_TIMER = 3000; //Waiting timer for each page in milliseconds. Default is 2500. Making it too short might prevent pages from loading.

//Play a silent audio file to prevent throttling (different files for firefox/chromium)
//Might require autoplay to be turned on for the website
let silentAudio, silentAudioFile;
const isFirefox = navigator.userAgent.search("Firefox");
if (isFirefox) {
    silentAudioFile = 'data:audio/ogg;base64,T2dnUwACAAAAAAAAAAAHA8coAAAAAIF47n4BE09wdXNIZWFkAQE0AEAfAAAAAABPZ2dTAAAAAAAAAAAAAAcDxygBAAAAUzJlNAGOT3B1c1RhZ3MNAAAATGF2ZjU1LjMzLjEwMAQAAAAVAAAAYXV0aG9yPUFkdmVudHVyZSBMYW5kFAAAAGFsYnVtPUFkdmVudHVyZSBMYW5kIwAAAHRpdGxlPUVtcHR5IExvb3AgRm9yIEpTIFBlcmZvcm1hbmNlFQAAAGVuY29kZXI9TGF2ZjU1LjMzLjEwME9nZ1MAAFi9AAAAAAAABwPHKAIAAAAyAzleMwoNCgsPCQkMDgwKDA0ODQ0NDAwMDAwMCw4MDQ4LDA0ODAwNDQwLCwoPDgwLCg0MDgsMDAgDQvtgoSdsKEAINjpmG6kyq3UB2fEgCDZ/rdQl7AWg4Ag2dYShfx2/4UHACDZ/ow5omm9mOzA0aJyXCDZ1OLVrIJjzCAVr7b2iKamgCDZ1SDnykImIDYOACDa3TfB0Gr+bWrtGb+AINnUv+ROFBR8zDoAINnVnnsPAbLa6CDZ6sGCGuOv3nlNwCDZ6I3ut9dAaHCqX3wg2XDBwFzhT1bJXG4SgCDZ1SFKCeQWBogMPggg2daDavCMN1fLmYOAINmjtoTkep+3zvsjPCDa5WDHLsGmySt1gCDZ6KI6PqMV1GKI4CDZ2BdGcG3TAu1uACDa9TAnYduSD+H3wCDZ1bhTJjA3kXrTACDU4w/g6h585SxZ0CDZ1oxPBDPDre7gINnVBqj1sCfuTiZIzsAg2dgRZAKiazM0ItQg2dgXiRKSpPCjR/SgINnXJOQWzDETgs/tm4Ag2O4gbz8WYlj2wCDWdjVsms44oglYaCDZ1S85AzQVEE/b83Ag2eW35JDbXlk8eCJfMCDcki/+9RVLZ3O4fCDVey7IEoi5wDZcVCDZ0LF6mtpuWlXG8YAg2V2QIXTjvaCEIOIwINnWjFnyIy+83bLgINnXL0yMgbqjfyAg2uVeivlfw1LfQCDZ1y8XIVqvYVQg2daJgNh1l6Ul2oTOYaAg2doUKwX0fVXOgLHrQCDZ1pGelDVg0sabgCDZC/Z1+dxdRPIwINnQw8kKLxYt5CDZ2BBYNLi6nUoeI4Ag2V5BoAlPMrQ57YAg2dW1ns/tgS0qGpA1ACDZX214nEzghqKQINacJO/rHa02gVEYINleU2/9UWr02NuBPZ2dTAAQwwwAAAAAAAAcDxygDAAAARL86cAEKCAVTo0DgY4cToA==';
    silentAudio = new Audio(silentAudioFile);
    silentAudio.volume = 0.01;
} else {
    silentAudioFile = 'data:audio/mp3;base64,SUQzBAAAAAAAI1RTU0UAAAAPAAADTGF2ZjU3LjcxLjEwMAAAAAAAAAAAAAAA/+M4wAAAAAAAAAAAAEluZm8AAAAPAAAAEAAABVgANTU1NTU1Q0NDQ0NDUFBQUFBQXl5eXl5ea2tra2tra3l5eXl5eYaGhoaGhpSUlJSUlKGhoaGhoaGvr6+vr6+8vLy8vLzKysrKysrX19fX19fX5eXl5eXl8vLy8vLy////////AAAAAExhdmM1Ny44OQAAAAAAAAAAAAAAACQCgAAAAAAAAAVY82AhbwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/+MYxAALACwAAP/AADwQKVE9YWDGPkQWpT66yk4+zIiYPoTUaT3tnU487uNhOvEmQDaCm1Yz1c6DPjbs6zdZVBk0pdGpMzxF/+MYxA8L0DU0AP+0ANkwmYaAMkOKDDjmYoMtwNMyDxMzDHE/MEsLow9AtDnBlQgDhTx+Eye0GgMHoCyDC8gUswJcMVMABBGj/+MYxBoK4DVpQP8iAtVmDk7LPgi8wvDzI4/MWAwK1T7rxOQwtsItMMQBazAowc4wZMC5MF4AeQAGDpruNuMEzyfjLBJhACU+/+MYxCkJ4DVcAP8MAO9J9THVg6oxRMGNMIqCCTAEwzwwBkINOPAs/iwjgBnMepYyId0PhWo+80PXMVsBFzD/AiwwfcKGMEJB/+MYxDwKKDVkAP8eAF8wMwIxMlpU/OaDPLpNKkEw4dRoBh6qP2FC8jCJQFcweQIPMHOBtTBoAVcwOoCNMYDI0u0Dd8ANTIsy/+MYxE4KUDVsAP8eAFBVpgVVPjdGeTEWQr0wdcDtMCeBgDBkgRgwFYB7Pv/zqx0yQQMCCgKNgonHKj6RRVkxM0GwML0AhDAN/+MYxF8KCDVwAP8MAIHZMDDA3DArAQo3K+TF5WOBDQw0lgcKQUJxhT5sxRcwQQI+EIPWMA7AVBoTABgTgzfBN+ajn3c0lZMe/+MYxHEJyDV0AP7MAA4eEwsqP/PDmzC/gNcwXUGaMBVBIwMEsmB6gaxhVuGkpoqMZMQjooTBwM0+S8FTMC0BcjBTgPwwOQDm/+MYxIQKKDV4AP8WADAzAKQwI4CGPhWOEwCFAiBAYQnQMT+uwXUeGzjBWQVkwTcENMBzA2zAGgFEJfSPkPSZzPXgqFy2h0xB/+MYxJYJCDV8AP7WAE0+7kK7MQrATDAvQRIwOADKMBuA9TAYQNM3AiOSPjGxowgHMKFGcBNMQU1FMy45OS41VVU/31eYM4sK/+MYxKwJaDV8AP7SAI4y1Yq0MmOIADGwBZwwlgIJMztCM0qU5TQPG/MSkn8yEROzCdAxECVMQU1FMy45OS41VTe7Ohk+Pqcx/+MYxMEJMDWAAP6MADVLDFUx+4J6Mq7NsjN2zXo8V5fjVJCXNOhwM0vTCDAxFpMYYQU+RlVMQU1FMy45OS41VVVVVVVVVVVV/+MYxNcJADWAAP7EAFVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV/+MYxOsJwDWEAP7SAFVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV/+MYxPMLoDV8AP+eAFVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV/+MYxPQL0DVcAP+0AFVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV';
    silentAudio = new Audio(silentAudioFile);
}
silentAudio.loop = true;

let currentPage = 0;
let totalPageCount = 0;
let totalFavoritesCount = 0;
let topMenuGenerated = false;

let exportResultTextField;
let notificationTextField;
let nextPageUrl;

const generateTopMenu = () => {
    const displayMode = $(".searchnav:first div:nth-child(6) option[selected='selected']").val();
    if (displayMode !== 'e') {
        alert("Use extended mode for favorites export")
        return;
    }
    if (!topMenuGenerated) {
        topMenuGenerated = true;
        const topMenu = $("#nb");

        topMenu.after('<br/>');
        topMenu.after($('<input/>').attr('type', 'button').attr('value', 'START').attr('id', 'start-button').attr('href', 'javascript:void(0)').attr('style', 'margin-left:5px;margin-top:5px;min-height:18px;'));
        $('#start-button').on('click', exportFavorites);
        topMenu.after('<br/>');
        topMenu.after('<textarea id="export-status" name="export-status" cols="100" rows="8" autocomplete="off" autocapitalize="off" spellcheck="false"/>');
        notificationTextField = $('#export-status');

        topMenu.after('<h2>Status</h2>');
        topMenu.after('<textarea id="export-fav-result" name="export-fav-result" cols="100" rows="8" />');
        exportResultTextField = $('#export-fav-result');
        topMenu.after('<h2>Text List of Favorites</h2>');
        topMenu.after('<br/><br/>');
        appendNotification("Chrome and some other browsers throttle background tabs so if you switch to another tab it might prevent you from loading all the pages.");
        appendNotification("To bypass that; turn this tab into its own window before you start and don't minimize it.");
        appendNotification("Press START below to start loading the pages.");
    }
};

const generateTagListByNameSpace = function (page, i) {
    const tags = {}
    $(page).find(".gl4e").eq(i).find("div .gt").each(function () {
        const content = $(this)[0].title.split(":");
        const namespace = content[0];
        const value = content[1];
        if (!tags[namespace]) {
            tags[namespace] = [];
        }
        tags[namespace].push(value)
    });
    return tags;
};

const exportPage = function (page) {
    appendNotification(`Loading Page ${++currentPage} of ${totalPageCount}`);
    const numberOfResults = $(page).find(".glink").length;
    for (let i = 0; i < numberOfResults; i++) {
        const tags = generateTagListByNameSpace(page, i);
        const exportData = {
            category: $(page).find(".itg .cn").next()[i].title,
            link: $(page).find(".glink")[i].parentNode.parentNode.href,
            title: $(page).find(".glink")[i].innerText.replaceAll("<",""), //Don't know why jquery fucks up writing this to textarea
            note: $(page).find(".itg .glfnote")[i].innerText.substr(6),
            tags
        }
        console.log(JSON.stringify(exportData));
        appendExportResult((currentPage === 1 && i === 0 ? '' : ',') + JSON.stringify(exportData))
    }
};

const endExport = () => {
    silentAudio.pause();
    appendNotification("Loading complete.")
    appendNotification("Copy above list, Ctrl+S to save the page or use an extension like SingleFile or Save Page WE.");
    appendExportResult(']}')
};

const computeTotalFavoriteCount = () => {
    //Get the current favorite category selected
    totalFavoritesCount = parseInt($(".nosel .fps").text());
    //If we are in show all favorites mode we have to sum all categories
    if (isNaN(totalFavoritesCount)) {
        totalFavoritesCount = parseInt($(".nosel .fp").toArray().reduce((acc, cur) => {
            const numberInCategory = parseInt(cur.textContent);
            if (!isNaN(numberInCategory)) {
                return acc + numberInCategory;
            }
            return acc;
        }, 0))
    }
};

const appendExportResult = (value) => {
    exportResultTextField.append(value + '\n');
};

const appendNotification = (value) => {
    notificationTextField.append(value + '\n');
    notificationTextField.scrollTop(notificationTextField[0].scrollHeight);
};

const exportFavorites = () => {
    computeTotalFavoriteCount();
    totalPageCount = Math.ceil(totalFavoritesCount / 50);
    nextPageUrl = $(".searchnav:first div:nth-child(4) a").attr('href');

    appendExportResult('{"favorites" : [');
    appendNotification(`Loading ${totalPageCount} page(s) for ${totalFavoritesCount} favorites..., estimated time is about ${Math.round((PAGE_TIMER / 1000) * totalPageCount)} seconds`);
    silentAudio.play();
    //Export first page
    exportPage(document)
    //If there is only one page we can return early without the need to make ajax calls
    if (!nextPageUrl) {
        endExport();
        return;
    }
    //Else we have to crawl next page for as long as possible
    const crawlInterval = setInterval(() => {
        $.ajax({
            url: nextPageUrl, type: 'GET', tryCount: 0, retryLimit: 5, success: result => {
                exportPage(result);
                nextPageUrl = $(result).find(".searchnav:first div:nth-child(4) a").attr('href');
                //If the requested page has no next page we can clear the interval
                if (!nextPageUrl) {
                    clearInterval(crawlInterval);
                    endExport();
                }
            }, error: () => {
                this.tryCount++;
                if (this.tryCount <= this.retryLimit) {
                    //try again
                    $.ajax(this);
                    return;
                }
                //Must have failed all retries for this page
                clearInterval(crawlInterval);
            }
        });
    }, PAGE_TIMER);
};

const addTopMenuEntry = () => {
    const topMenu = $("#nb");
    topMenu.css("max-width", "1400px");
    topMenu.append($('<div/>').attr('id', 'export-fav-div'));
    $('#export-fav-div').append($('<a/>').attr('href', 'javascript:void(0)').attr('id', 'export-favorites').text('Load All Favorites'));
    $('#export-favorites').on('click', generateTopMenu);
};

$(() => {
    addTopMenuEntry();
});

SaddestPandaAuthor
§
Posted: 2022.11.05.

Thanks for the huge contribution.
I also had some changes done on my side. I'll take these improvements from your script and combine them into an update.

SaddestPandaAuthor
§
Posted: 2022.11.06.

The new version is up. I changed the json format.
Your code had a small problem with the tags so you should either switch to the new version or backport my fix.

Post reply

Sign in to post a reply.