The StarThinker project will no longer receive new solution contributions from the Google team.
Please read the full StarThinker Open Source Support Ends At Google article for more details.


build Setup Project

Almost every recipe in StarThinker uses a Google Cloud Project to store data. ALL DATA COSTS ARE BILLED TO THE GOOGLE CLOUD SERVICE CREDENTIALS SPECIFIED IN EACH RECIPE. To add a Google Cloud Project for billing and permissions, create a Service Credentials, download it, and enter the JSON into the StarThinker Project page.

Cloud service setup diagram.

Screenshot of StarThinker Edit Project page.
Getting Service Credentials

Each user should have their own Google Cloud Project to avoid accidental data sharing and allow easier auditing.

  1. Visit Google Cloud Project > APIs & Services > Credentials > Create Credentials > Service Account Key > New Service Account.
  2. Service Account Name is starthinker.
  3. Click Create.
  4. The most robust role is Project Editor, review your role choice with your security team.
  5. Click Continue.
  6. Click Create Key.
  7. Key Type is JSON.
  8. Click Create and open the downloaded JSON file.
  9. Copy the contents and paste into the StarThinker Project page Service field.
  10. Click save.
  11. The new project will become a dropdown option on StarThinker Recipe pages.

WARNING: If you share your service credentials, other users will be able to use them in their recipes but not download or view them. Sharing the service credentials grants other users all the permissions your service account has, use extreme caution.

view_list Run Recipe

All recipes run as the logged in user or selected service account. StarThinker DOES NOT grant any additional permissions or access to any products or endpoints used in the recipe.

Screenshot of StarThinker Edit Recipe page.
Creating A Recipe Workflow

Recipes can be manual, initiated by the user and run only once, or scheduled, running at set times on set days. Other than the schedule, all steps are the same.

  1. Visit the scheduled or manual recipe page.
  2. Name the recipe, the name is often used to create assets such as Sheets and Dataset by some recipes, choose the name to reflect an account or client. If changed, already created assets may become disconnected.
  3. If the recipe writes to Google Cloud, select the Google Cloud Project Service Account, set up earlier.
  4. For scheduled recipes, choose the timezone, days, and hours to run every task in the recipe. All task times and assets created by the recipe will use the timezone selected as a refrence.
  5. Click Add.
  6. Use the left side filters to search for the specific task to be added to the recipe. The filters scroll up and down. Most tasks provide a description, samples, and links to assets for review on this page. Click add_circle to add the task to the end of the recipe. A check_circle will appear with a count of this task in the recipe. Continue adding tasks, or close the task filter window.
  7. Each task has a card with its own instructions, inputs, and fields. Fields are marked optional or required. Some have default values. All values are derived from the API endpoints for that product.
  8. Click Save, the recipe is now ready to run, and will run on schedule.
  9. Click Run to execute the recipe immediately, if workers are busy there may be a delay. The Job Status and log update every 10 seconds.
  10. Click Log to view recipe task events including errors and completes.
  11. Click Download to share the recipe with a client.

network_check Job Status

Each task in a recipe workflow is executed in sequence one at a time. Some recipes have tasks executing at different times, and some tasks executing multiple times a day. Job status is updated automatically by workers as the job is processed. The UI updates the log and status every 10 seconds.

Status Definition Action
Queued The recipe is waiting for workers to pick up the next task. Wait for the next scheduled task to execute. If workers are busy, the next taks may be executed after its scheduled time.
Running A task in the recipe is currently executing. Wait for the task to execute or cancel a running recipe. Check the logs to see which task is running and see real time task updates.
Finished All tasks in the recipe completed without error. The task will resume again the next day as controlled by its timezone setting.
Paused No tasks in the recipe will be run. Unpause the recipe to resume execution.
Timeout The task exceeded its 8 hour limit for execution and was terminated. Check the logs, Edit the task to process less data and complete sooner. Some timeouts may be acceptable, for example a report was not ready, typically these tasks are scheduled multiple times in a workflow and eventually work.
Error The task failed to execute. The error will be printed in the log, usually near the bottom for that task. Correct the error, and click Run Now to re-execute the recipe. Alternatively some errors may be acceptable, for example, a task may run multiple times a day, return errors until eventually one execution works.

share Share Recipe

Why share instructions when you can share a working workflow. Every custom recipe created in the StarThinker UI can be shared with clients. Create a recipe then click the Download button. DO NOT INCLUDE PASSWORDS OR CONFIDENTIAL INFORMATION IN SHARED RECIPES. Shared recipes can be executed via Google Collaboratory, Airflow/Composer, Python, or UI.


Screenshot of StarThinker Download Recipe page.

Google Collaboratory

Ideal for non-technical clients who want to execute the recipe on demand.

  1. Use for demos and workshops.
  2. Allows customization.
  3. No coding necessary.
  4. Requires a free Google Cloud Project for credentials.
Airflow/Composer

Ideal for highly technical clients with engineering teams already using Airflow/Composer.

  1. Use for long running jobs using Airflow.
  2. Allows customization.
  3. Extensive coding required.
  4. Requires a paid Google Cloud Composer instance.
Python

Ideal for highly technical clients who want a minimal deployment footprint.

  1. Use for custom infrastructure integration.
  2. Allows minimal customization.
  3. Extensive coding required.
  4. Requires a free Google Cloud Project for credentials.
UI

Ideal for teams wanting to build recipes at scale for multiple accounts.

  1. Use one UI for entire team.
  2. Support no-coding UI recipe builder.
  3. Operations and support required.
  4. Requires a paid Google Cloud Project.

report_problem Warnings

StarThinker will do what you tell it to. IT RUNS AS YOU, if you have access it will do it, be careful about overwriting, creating, sharing, or deleting data. Double check everything before running tasks. Undo is not a feature.

  • DATASETS ARE NOT DELETED when a recipe is deleted.
    1. You will have to manually delete the dataset.
    2. You will have to verify dataset contains what you want before sharing.
  • Changing DBM / DCM report definition requires delete of report.
    1. You will have to go into DBM / DCM and delete the report.
    2. StarThinker will then create the new report definition.
  • BigQuery views are not modified by StarThinker if they exist.
    1. This is to protect any customizations you may have added.
    2. You will have to delete the view for StarThinker to recreate it.
  • All tasks in a recipe are executed, even if ones prior fail.
    1. This allows retries of the same task later in the day.
    2. For mission critical data flows, check your data before acting on it.
  • All errors and fields come from the API documentation.
    1. When looking for field values or formats consult the approriate API documentation.
    2. When errors are thrown, the log shows them, consult the API documents to resolve the error.
  • StarThinker does not gurantee exact task execution time.
    1. All tasks are executed in sequence.
    2. Tasks run at or after the scheduled hour, depending on worker load.
    3. If a task fails, remianing tasks are still executed.

help Debugging

Errors in StarThinker can range from a misconfigured field value to a bug in the code. To promptly resolve most issues please follow these debugging steps.

When working with StarThinker administrators, please include the recipe number, found in the recipe edit URL, in all communications.

  1. Consult the log and reference the error in the API documentation relevant to the product used in the task.
  2. Ensure the service account used in the recipe exists and is valid.
  3. Try logging out and logging back in to refresh user credentials and access to APIs.
  4. Contact the administrator of the StarThinker UI where the recipe was deployed.
  5. Contact the owner of the solution listed on the Solution Gallery Page.
  6. Contact starthinker-help@google.com.