For the data loaded from the raw and unstructured data sources (RESTful APIs, files, feeds) I'd recommend checking tablum.io. The data first gets into the staging SQL DB where you can apply transformations in SQL (ClickHouse dialect with hundreds of ready-to-use functions for aggregations, transformations, string and array processing, etc). After that, you can set up a pipeline to external DWH or DB. Or just... Source: about 1 year ago

Fairly easy to parse it with Python (or any other scripting languages that support regular expressions). Besides, there're no-code parsers that can do that. E.g. tablum.io, it has a regexp parser that turns poorly formatted text into an SQL table. All you need it is to provide the regular expression (regexp) to split the text (per-line basis or the entire text) into chunks that become SQL rows and columns, after... Source: about 1 year ago

You're right. There should be some way to safely pass the authentication token without storing it in the query. BTW, to keep all data within the company network perimeter and no share any keys with the cloud service, one can install tablum.io as a self-hosted version on a Linux server (it runs in Docker). Source: about 1 year ago

For a small database you don't need Snowflake. You need Postgres or MySQL. Power BI for visualizing data seems fine. For entering data you can use Airforms. Source: 11 months ago

PostgreSQL is an open-source relational database, used by many companies, and is very common among cloud applications, where companies prefer an open-source solution, supported by a strong community, as an alternative to commercial database engines. The simplest way to run the PostgreSQL engine in the cloud is to choose one of the managed database services, such as Amazon RDS for PostgreSQL or Google Cloud SQL... - Source: dev.to / about 1 year ago

For the database, I used Cloud SQL, which is a managed database service from Google Cloud Platform (GCP). This GCP product provides a cloud-based alternative to MySQL, PostgreSQL and SQL Server databases. The great advantage of Cloud SQL is that it is a managed service, that is, you do not have to worry about some tasks related to the infrastructure where the database will run, tasks such as backups, maintenance... - Source: dev.to / almost 2 years ago

Database: Managed Postgres -- "Cloud SQL". Source: over 2 years ago

Why would not just use https://cloud.google.com/sql/docs/postgres ? Source: about 3 years ago