The StarThinker project will no longer receive new solution contributions from the Google team.
Please read the full StarThinker Open Source Support Ends At Google article for more details.
Pull URL list from a table, fetch them, and write the results to another table.
| Specify a table with only two columns URL, URI (can be null). |
| Check bigquery destination for results of fetching each URL. |
| Open Source | YES |
| Age | March 10, 2021 (1 year, 9 months) |
| Authors | kenjora@google.com |
| Shedule Days | Configured by user. |
| Shedule Hours | Configured by user. |
[
{
"url": {
"auth": {
"field": {
"name": "auth",
"kind": "authentication",
"order": 1,
"default": "service",
"description": "Credentials used for rading and writing data."
}
},
"status": {
"field": {
"name": "status",
"kind": "boolean",
"order": 2,
"default": true,
"description": "Pull status of HTTP request."
}
},
"read": {
"field": {
"name": "read",
"kind": "boolean",
"order": 3,
"default": false,
"description": "Pull data from HTTP request."
}
},
"urls": {
"bigquery": {
"dataset": {
"field": {
"name": "dataset",
"kind": "string",
"order": 4,
"default": "",
"description": "Name of Google BigQuery dataset to write."
}
},
"query": {
"field": {
"name": "table",
"kind": "text",
"order": 5,
"default": "",
"description": "Query to run to pull URLs."
}
},
"legacy": false
}
},
"to": {
"bigquery": {
"dataset": {
"field": {
"name": "dataset",
"kind": "string",
"order": 6,
"default": "",
"description": "Name of Google BigQuery dataset to write."
}
},
"table": {
"field": {
"name": "table",
"kind": "string",
"order": 7,
"default": "",
"description": "Name of Google BigQuery table to write."
}
}
}
}
}
}
]Everything from a quick Google Cloud UI to reference developer code for your team in one GitHub repository.