Global

Members

(constant) got

Author:
  • IITII
Source:

(constant) logger

logger
Source:

Methods

(async) cdn_preheating(token, preHeatArray, MAX_TRY, QUERY_BREAK, chunk_size, got_instance)

cdn preHeating task
Parameters:
Name Type Default Description
token String huaweicloud IAM TOKEN
preHeatArray Array pre-heating url array
MAX_TRY Maximum attempts
QUERY_BREAK Number setTimeout time unit
chunk_size 10
got_instance null Got instance, default null As we know, one we commit the preHeatingTask, more than one cdn node will get resource file from the source site via public Internet.
So, the source site's traffic will be very busy after we commit the task. If too many URLs are submitted in a short time, most tasks will fail.
huaweicloud will retry once if connection time is more than 30s.
Source:

(async) cdn_preheatingtasks(array, token, instance) → {JSON|Error}

huaweicloud cdn preheatingtasks
Parameters:
Name Type Default Description
array Array | String
token String huaweicloud IAM TOKEN
instance null Got instance, default null
Source:
See:
Returns:
result
Type
JSON | Error

(async) cdn_refreshtasks(array, token, types, instance) → {JSON|Error}

huaweicloud cdn refreshtasks
Parameters:
Name Type Default Description
array Array | String
token String huaweicloud IAM TOKEN
types 'file' | 'directory' file refresh types, default 'file'
instance null Got instance, default null
Source:
See:
Returns:
result
Type
JSON | Error

checkUrl(array) → {Array}

Give a string or array, get a compact array
Parameters:
Name Type Description
array String | Array Input array or string
Source:
See:
Returns:
Output array without null object
Type
Array

(async) get(key) → {Number}

Get value for special key
Parameters:
Name Type Description
key pid, task "https://www.pixiv.net/artworks/84521887" for instance, the key of this url is 84521887
Source:
Returns:
duplicate times
Type
Number

get_bytes(INTERFACE_NAME)

get previous break traffic state, only support linux
Parameters:
Name Type Default Description
INTERFACE_NAME eth0 network interface device name
Source:
Returns:
unit: byte, previous break traffic state

(async) getDailyRankUrl(limit)

Get Daily Rank Url
Parameters:
Name Type Default Description
limit 50 limit array size max: 50, default: 50
Source:
Returns:
Array A array for Daily Rank Url with limit

(async) getDom(url, proxy) → {Promise.<*>}

return dom base on url
Parameters:
Name Type Description
url Telegraph URL
proxy http proxy
Source:
Returns:
cheerio
Type
Promise.<*>

getRandomMin(low, high) → {number}

Get a random milliseconds from low to high
Parameters:
Name Type Description
low start (unit: minute)
high end (unit: minute)
Source:
Returns:
return a random milliseconds from low to high(unit: milliseconds) return 0 if high is lesser than low
Type
number

getRandomSec(low, high) → {number}

Get a random milliseconds from low to high
Parameters:
Name Type Description
low start (unit: second)
high end (unit: second)
Source:
Returns:
return a random milliseconds from low to high(unit: milliseconds) return 0 if high is lesser than low
Type
number

(async) getRealImgUrl(driver, imgUrl, js) → {Array}

Get img origin url
Parameters:
Name Type Description
driver selenium driver
imgUrl URL
js String
Source:
Returns:
Array array length maybe greater than 1
Type
Array

(async) getToken() → {JSON}

Get huaweicloud IAM TOKEN
Source:
See:
Returns:
huaweicloud IAM TOKEN
Type
JSON

got_instance(prefixUrl, token, json_body) → {Got}

Simple packaging got
Parameters:
Name Type Default Description
prefixUrl String | URL got prefixUrl
token String huaweicloud IAM Token
json_body null POST BODY, NULL for GET Method
Source:
See:
Returns:
Got instance
Type
Got

human_net_speed(bytes, pre_unit, bits) → {string}

Get human readable network speed
Parameters:
Name Type Default Description
bytes bytes
pre_unit number 1024 1024 or 1000, default 1024
bits 1 Keep decimal places, default 1
Source:
Returns:
human readable network speed
Type
string

(async) hwc_common(prefixUrl, api_url, body, token, instance) → {JSON|Error}

huaweicloud common api operation
Parameters:
Name Type Default Description
prefixUrl String | URL prefixUrl got prefixUrl
api_url API URL which shouldn't start with '/'
body request body
token String huaweicloud IAM TOKEN
instance null Got instance, default null
Source:
See:
Returns:
Type
JSON | Error

isNil(object) → {boolean}

Checks if value is null or undefined or ''.
Parameters:
Name Type Description
object object
Source:
Returns:
true for nil or ""
Type
boolean

(async) live_net_speed(INTERFACE_NAME, sampling_break, human)

Get live network speed
Parameters:
Name Type Default Description
INTERFACE_NAME eth0 Interface name
sampling_break 1000 Sampling break
human false Show human readable net speed
Source:

(async) login(driver, username, password)

Login to pixiv
Parameters:
Name Type Description
driver selenium driver
username pixiv username
password pixiv password
Source:

mkdir(dir, cb)

递归创建文件夹
Parameters:
Name Type Description
dir 文件夹目录
cb callback
Source:

mkdirSync(dir)

同步递归创建文件夹
Parameters:
Name Type Description
dir 文件夹目录
Source:

preCheck() → {Boolean}

pre-check for huaweicloud cdn
Source:
Returns:
true for passed, false for failed
Type
Boolean

(async) saveImg(data, IMG_TMP_DIR, useragent)

Download images
Parameters:
Name Type Description
data Array
IMG_TMP_DIR String
useragent String
Source:

(async) set(key, value) → {Number}

Set value for special key
Parameters:
Name Type Default Description
key pid, task "https://www.pixiv.net/artworks/84521887" for instance, the key of this url is 84521887
value 1 stepper
Source:
Returns:
duplicate times
Type
Number

(async) setRedis(array, value) → {Array}

Put data to redis
Parameters:
Name Type Default Description
array Array origin array
value Number 1 defa
Source:
Returns:
removed array
Type
Array

(async) showHistoryTaskDetails(history_tasks_id, token, instance) → {JSON|Error}

huaweicloud cdn refreshtasks
Parameters:
Name Type Description
history_tasks_id TaskID
token String huaweicloud IAM TOKEN
instance Got instance, default null
Source:
See:
Returns:
result
Type
JSON | Error

(async) sleep(ms)

sleep for a while
Parameters:
Name Type Description
ms ms
Source:

(async) sleep(ms)

sleep for a while
Parameters:
Name Type Description
ms ms
Source:

(async) spendTime(func, …args)

Calc how much time spent on run function.
Parameters:
Name Type Attributes Description
func Run function
args <repeatable>
 function's args
Source:

(async) unique(array) → {Array}

Remove duplicate url which was downloaded in the past
Parameters:
Name Type Description
array Array origin array
Source:
Returns:
removed array
Type
Array

(async) wait_for_low_traffic_usage(INTERFACE, break_time, TX_RX, MAX_RX_SPEED, MAX_TX_SPEED, usage,)

Wait for low traffic usage
Parameters:
Name Type Default Description
INTERFACE eth0 interface name, default eth0
break_time 5000 query break,unit: millisecond, default 5000
TX_RX TR 'TX' | 'RX' | 'TR', listening on TX or RX or TX & RX traffic, default TR
MAX_RX_SPEED 100 Max Receive speed,unit: Mbit/s, default 100
MAX_TX_SPEED 30 Max transmit speed,unit: Mbit/s, default 30
usage, network usage limit, default 0.6
Source:

(async) waitForRefreshTaskDone(token, refreshTaskId, MAX_TRY, query_break, instance) → {Array|Number|Error}

wait for refresh task done
Parameters:
Name Type Description
token String huaweicloud IAM TOKEN
refreshTaskId Number cdn refresh Task ID
MAX_TRY Maximum attempts
query_break Number setTimeout time unit
instance Got instance, default null
Source:
Returns:
Number for succeed task numbers
Array for failed task array
Error for other error
Type
Array | Number | Error

(async) zipDir(dirName, zipFileName)

docs is out-of-date Unsupported chinese folder name
Parameters:
Name Type Description
dirName compress dirname
zipFileName compressed filename
Source:
See: